Tables and DataFrames
Tables.jl provides an ecosystem-wide interface to tabular data in Julia, ensuring interoperability with DataFrames.jl, CSV.jl, and hundreds of other packages that implement the standard.
Dimensional data are tables
DimensionalData.jl implements the Tables.jl interface for AbstractDimArray and AbstractDimStack. DimStack layers are unrolled so they are all the same size, and dimensions loop to match the length of the largest layer.
Columns are given the name of the array or stack layer, and the result of DD.name(dimension) for Dimension columns.
Looping of dimensions and stack layers is done lazily, and does not allocate unless collected.
Materializing tables to DimArray or DimStack
DimArray and DimStack have fallback methods to materialize any Tables.jl-compatible table.
By default, it will treat columns such as X, Y, Z, and Band as dimensions, and other columns as data. Pass a name keyword argument to determine which column(s) are used.
You have full control over which columns are dimensions - and what those dimensions look like exactly. If you pass a Tuple of Symbol or dimension types (e.g. X) as the second argument, those columns are treated as dimensions. Passing a Tuple of dimensions preserves these dimensions - with values matched to the corresponding columns.
Materializing tables will worked even if the table is not ordered, and can handle missing values.
Example
using DimensionalData
using Dates
using DataFramesDefine some dimensions:
julia> x, y, c = X(1:10), Y(1:10), Dim{:category}('a':'z')(↓ X 1:10,
→ Y 1:10,
↗ category 'a':1:'z')julia> A = rand(x, y, c; name=:data)┌ 10×10×26 DimArray{Float64, 3} data ┐
├────────────────────────────────────┴────────────── dims ┐
↓ X Sampled{Int64} 1:10 ForwardOrdered Regular Points,
→ Y Sampled{Int64} 1:10 ForwardOrdered Regular Points,
↗ category Categorical{Char} 'a':1:'z' ForwardOrdered
└─────────────────────────────────────────────────────────┘
[:, :, 1]
↓ → 1 2 3 … 8 9 10
1 0.0779365 0.284432 0.828376 0.101372 0.16707 0.551195
2 0.219304 0.615316 0.449789 0.853508 0.429108 0.326674
3 0.808553 0.453097 0.434883 0.965292 0.337536 0.135547
⋮ ⋱ ⋮
7 0.985053 0.235447 0.518961 0.079027 0.901092 0.613534
8 0.837837 0.132421 0.665932 0.313258 0.906392 0.492469
9 0.900526 0.69021 0.284993 0.340798 0.665965 0.514539
10 0.336887 0.353286 0.647132 … 0.719959 0.688119 0.17463Converting to DataFrame
Arrays will have columns for each dimension, and only one data column
julia> DataFrame(A)2600×4 DataFrame
Row │ X Y category data
│ Int64 Int64 Char Float64
──────┼───────────────────────────────────
1 │ 1 1 a 0.0779365
2 │ 2 1 a 0.219304
3 │ 3 1 a 0.808553
4 │ 4 1 a 0.864836
5 │ 5 1 a 0.487313
6 │ 6 1 a 0.823322
7 │ 7 1 a 0.985053
8 │ 8 1 a 0.837837
⋮ │ ⋮ ⋮ ⋮ ⋮
2594 │ 4 10 z 0.958015
2595 │ 5 10 z 0.0903243
2596 │ 6 10 z 0.819821
2597 │ 7 10 z 0.980845
2598 │ 8 10 z 0.914611
2599 │ 9 10 z 0.129294
2600 │ 10 10 z 0.578021
2585 rows omittedConverting to CSV
We can also write arrays and stacks directly to CSV.jl, or any other data type supporting the Tables.jl interface.
using CSV
CSV.write("dimstack.csv", st)
readlines("dimstack.csv")2601-element Vector{String}:
"X,Y,category,data1,data2"
"1,1,a,0.009977352122249039,0.8228601434032986"
"2,1,a,0.1305251128647209,0.33772632881409725"
"3,1,a,0.6259155174888883,0.21993961918654126"
"4,1,a,0.06519322052560106,0.08263686242822754"
"5,1,a,0.41204657699507596,0.7107248802913473"
"6,1,a,0.2266498078450272,0.8487078638946295"
"7,1,a,0.5768042809962604,0.8566521484695245"
"8,1,a,0.46048147930842187,0.8838950820535972"
"9,1,a,0.03556049992826038,0.6889547118345359"
⋮
"2,10,z,0.23633602235389983,0.2577676702146605"
"3,10,z,0.9636812107748465,0.948386944896322"
"4,10,z,0.5296580222236127,0.7053177109576688"
"5,10,z,0.8628968265470025,0.8772934954059349"
"6,10,z,0.9987530174404816,0.16548974916294534"
"7,10,z,0.9241779906093128,0.06081752015564701"
"8,10,z,0.7182127191255562,0.5837897355818146"
"9,10,z,0.3444618364233747,0.5092491568326865"
"10,10,z,0.6093106260763147,0.018384253568273556"Converting a DataFrame to a DimArray or DimStack
The Dataframe we use will have 5 columns: X, Y, category, data1, and data2
julia> df = DataFrame(st)2600×5 DataFrame
Row │ X Y category data1 data2
│ Int64 Int64 Char Float64 Float64
──────┼───────────────────────────────────────────────
1 │ 1 1 a 0.00997735 0.82286
2 │ 2 1 a 0.130525 0.337726
3 │ 3 1 a 0.625916 0.21994
4 │ 4 1 a 0.0651932 0.0826369
5 │ 5 1 a 0.412047 0.710725
6 │ 6 1 a 0.22665 0.848708
7 │ 7 1 a 0.576804 0.856652
8 │ 8 1 a 0.460481 0.883895
⋮ │ ⋮ ⋮ ⋮ ⋮ ⋮
2594 │ 4 10 z 0.529658 0.705318
2595 │ 5 10 z 0.862897 0.877293
2596 │ 6 10 z 0.998753 0.16549
2597 │ 7 10 z 0.924178 0.0608175
2598 │ 8 10 z 0.718213 0.58379
2599 │ 9 10 z 0.344462 0.509249
2600 │ 10 10 z 0.609311 0.0183843
2585 rows omittedConverting this DataFrame to a DimArray without other arguments will read the category columns as data and ignore data1 and data2:
julia> DimArray(df)┌ 10×10 DimArray{Char, 2} category ┐
├──────────────────────────────────┴────────────────── dims ┐
↓ X Sampled{Int64} 1:1:10 ForwardOrdered Regular Points,
→ Y Sampled{Int64} 1:1:10 ForwardOrdered Regular Points
└───────────────────────────────────────────────────────────┘
↓ → 1 2 3 4 5 6 7 8 9 10
1 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z'
2 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z'
3 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z'
4 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z'
⋮ ⋮ ⋮
7 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z'
8 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z'
9 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z'
10 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z' 'z'Specify dimenion names to ensure these get treated as dimensions. Now data1 is read in instead.
julia> DimArray(df, (X,Y,:category))┌ 10×10×26 DimArray{Float64, 3} data1 ┐
├─────────────────────────────────────┴──────────────── dims ┐
↓ X Sampled{Int64} 1:1:10 ForwardOrdered Regular Points,
→ Y Sampled{Int64} 1:1:10 ForwardOrdered Regular Points,
↗ category Categorical{Char} ['a', …, 'z'] ForwardOrdered
└────────────────────────────────────────────────────────────┘
[:, :, 1]
↓ → 1 2 3 … 8 9 10
1 0.00997735 0.0341906 0.909241 0.220025 0.0145673 0.308713
2 0.130525 0.254528 0.283612 0.351579 0.559111 0.236336
3 0.625916 0.785481 0.971546 0.691567 0.99022 0.963681
⋮ ⋱ ⋮
7 0.576804 0.0274633 0.161403 0.364875 0.925617 0.924178
8 0.460481 0.0302125 0.120721 0.428295 0.938904 0.718213
9 0.0355605 0.498208 0.203494 0.798389 0.0182774 0.344462
10 0.740236 0.477097 0.935915 … 0.14575 0.0548911 0.609311You can also pass in the actual dimensions.
julia> DimArray(df, dims(st))┌ 10×10×26 DimArray{Float64, 3} data1 ┐
├─────────────────────────────────────┴───────────── dims ┐
↓ X Sampled{Int64} 1:10 ForwardOrdered Regular Points,
→ Y Sampled{Int64} 1:10 ForwardOrdered Regular Points,
↗ category Categorical{Char} 'a':1:'z' ForwardOrdered
└─────────────────────────────────────────────────────────┘
[:, :, 1]
↓ → 1 2 3 … 8 9 10
1 0.00997735 0.0341906 0.909241 0.220025 0.0145673 0.308713
2 0.130525 0.254528 0.283612 0.351579 0.559111 0.236336
3 0.625916 0.785481 0.971546 0.691567 0.99022 0.963681
⋮ ⋱ ⋮
7 0.576804 0.0274633 0.161403 0.364875 0.925617 0.924178
8 0.460481 0.0302125 0.120721 0.428295 0.938904 0.718213
9 0.0355605 0.498208 0.203494 0.798389 0.0182774 0.344462
10 0.740236 0.477097 0.935915 … 0.14575 0.0548911 0.609311Pass in a name argument to read in data2 instead.
julia> DimArray(df, dims(st); name = :data2)┌ 10×10×26 DimArray{Float64, 3} data2 ┐
├─────────────────────────────────────┴───────────── dims ┐
↓ X Sampled{Int64} 1:10 ForwardOrdered Regular Points,
→ Y Sampled{Int64} 1:10 ForwardOrdered Regular Points,
↗ category Categorical{Char} 'a':1:'z' ForwardOrdered
└─────────────────────────────────────────────────────────┘
[:, :, 1]
↓ → 1 2 3 … 8 9 10
1 0.82286 0.893471 0.0216358 0.4497 0.588269 0.338276
2 0.337726 0.669018 0.615299 0.88672 0.232312 0.432012
3 0.21994 0.942886 0.575545 0.688617 0.198964 0.657162
⋮ ⋱ ⋮
7 0.856652 0.402187 0.804752 0.292401 0.705191 0.921592
8 0.883895 0.10895 0.946389 0.818221 0.931104 0.546192
9 0.688955 0.695436 0.265118 0.461449 0.282442 0.815821
10 0.566576 0.271396 0.82125 … 0.0527801 0.522909 0.847799