API Reference
Arrays
AbstractBasicDimArray <: AbstractArray
The abstract supertype for all arrays with a dims
method that returns a Tuple
of Dimension
Only keyword rebuild
is guaranteed to work with AbstractBasicDimArray
.
AbstractDimArray <: AbstractBasicArray
Abstract supertype for all "dim" arrays.
These arrays return a Tuple
of Dimension
from a dims
method, and can be rebuilt using rebuild
.
parent
must return the source array.
They should have metadata
, name
and refdims
methods, although these are optional.
A rebuild
method for AbstractDimArray
must accept data
, dims
, refdims
, name
, metadata
arguments.
Indexing AbstractDimArray
with non-range AbstractArray
has undefined effects on the Dimension
index. Use forward-ordered arrays only"
DimArray <: AbstractDimArray
DimArray(data, dims, refdims, name, metadata)
DimArray(data, dims::Tuple; refdims=(), name=NoName(), metadata=NoMetadata())
The main concrete subtype of AbstractDimArray
.
DimArray
maintains and updates its Dimension
s through transformations and moves dimensions to reference dimension refdims
after reducing operations (like e.g. mean
).
Arguments
data
: AnAbstractArray
.dims
: ATuple
ofDimension
name
: A string name for the array. Shows in plots and tables.refdims
: refence dimensions. Usually set programmatically to track past slices and reductions of dimension for labelling and reconstruction.metadata
:Dict
orMetadata
object, orNoMetadata()
Indexing can be done with all regular indices, or with Dimension
s and/or Selector
s.
Indexing AbstractDimArray
with non-range AbstractArray
has undefined effects on the Dimension
index. Use forward-ordered arrays only"
Example:
julia> using Dates, DimensionalData
julia> ti = Ti(DateTime(2001):Month(1):DateTime(2001,12));
julia> x = X(10:10:100);
julia> A = DimArray(rand(12,10), (ti, x), name="example");
julia> A[X(Near([12, 35])), Ti(At(DateTime(2001,5)))]
╭───────────────────────────────────────╮
│ 2-element DimArray{Float64,1} example │
├───────────────────────────────────────┴─────────────── dims ┐
↓ X Sampled{Int64} [10, 40] ForwardOrdered Irregular Points
└─────────────────────────────────────────────────────────────┘
10 0.253849
40 0.637077
julia> A[Near(DateTime(2001, 5, 4)), Between(20, 50)]
╭───────────────────────────────────────╮
│ 4-element DimArray{Float64,1} example │
├───────────────────────────────────────┴───────────── dims ┐
↓ X Sampled{Int64} 20:10:50 ForwardOrdered Regular Points
└───────────────────────────────────────────────────────────┘
20 0.774092
30 0.823656
40 0.637077
50 0.692235
Shorthand AbstractDimArray
constructors:
Base.fill(x, dims::Dimension...; kw...) => DimArray
Base.fill(x, dims::Tuple{Vararg{Dimension}}; kw...) => DimArray
Create a DimArray
with a fill value of x
.
There are two kinds of Dimension
value acepted:
A
Dimension
holding anAbstractVector
will set the dimension index to thatAbstractVector
, and detect the dimension lookup.A
Dimension
holding anInteger
will set the length of the axis, and set the dimension lookup toNoLookup
.
Keywords are the same as for DimArray
.
Example
julia> using DimensionalData, Random; Random.seed!(123);
julia> rand(Bool, X(2), Y(4))
╭──────────────────────╮
│ 2×4 DimArray{Bool,2} │
├──────────────── dims ┤
↓ X, → Y
└──────────────────────┘
0 0 0 0
1 0 0 1
Base.rand(x, dims::Dimension...; kw...) => DimArray
Base.rand(x, dims::Tuple{Vararg{Dimension}}; kw...) => DimArray
Base.rand(r::AbstractRNG, x, dims::Tuple{Vararg{Dimension}}; kw...) => DimArray
Base.rand(r::AbstractRNG, x, dims::Dimension...; kw...) => DimArray
Create a DimArray
of random values.
There are two kinds of Dimension
value acepted:
A
Dimension
holding anAbstractVector
will set the dimension index to thatAbstractVector
, and detect the dimension lookup.A
Dimension
holding anInteger
will set the length of the axis, and set the dimension lookup toNoLookup
.
Keywords are the same as for DimArray
.
Example
julia> using DimensionalData
julia> rand(Bool, X(2), Y(4))
╭──────────────────────╮
│ 2×4 DimArray{Bool,2} │
├──────────────── dims ┤
↓ X, → Y
└──────────────────────┘
0 0 0 0
1 0 0 1
julia> rand(X([:a, :b, :c]), Y(100.0:50:200.0))
╭─────────────────────────╮
│ 3×3 DimArray{Float64,2} │
├─────────────────────────┴───────────────────────────────────── dims ┐
↓ X Categorical{Symbol} [:a, :b, :c] ForwardOrdered,
→ Y Sampled{Float64} 100.0:50.0:200.0 ForwardOrdered Regular Points
└─────────────────────────────────────────────────────────────────────┘
↓ → 100.0 150.0 200.0
:a 0.443494 0.253849 0.867547
:b 0.745673 0.334152 0.0802658
:c 0.512083 0.427328 0.311448
Base.zeros(x, dims::Dimension...; kw...) => DimArray
Base.zeros(x, dims::Tuple{Vararg{Dimension}}; kw...) => DimArray
Create a DimArray
of zeros.
There are two kinds of Dimension
value acepted:
A
Dimension
holding anAbstractVector
will set the dimension index to thatAbstractVector
, and detect the dimension lookup.A
Dimension
holding anInteger
will set the length of the axis, and set the dimension lookup toNoLookup
.
Keywords are the same as for DimArray
.
Example
julia> using DimensionalData
julia> zeros(Bool, X(2), Y(4))
╭──────────────────────╮
│ 2×4 DimArray{Bool,2} │
├──────────────── dims ┤
↓ X, → Y
└──────────────────────┘
0 0 0 0
0 0 0 0
julia> zeros(X([:a, :b, :c]), Y(100.0:50:200.0))
╭─────────────────────────╮
│ 3×3 DimArray{Float64,2} │
├─────────────────────────┴───────────────────────────────────── dims ┐
↓ X Categorical{Symbol} [:a, :b, :c] ForwardOrdered,
→ Y Sampled{Float64} 100.0:50.0:200.0 ForwardOrdered Regular Points
└─────────────────────────────────────────────────────────────────────┘
↓ → 100.0 150.0 200.0
:a 0.0 0.0 0.0
:b 0.0 0.0 0.0
:c 0.0 0.0 0.0
Base.ones(x, dims::Dimension...; kw...) => DimArray
Base.ones(x, dims::Tuple{Vararg{Dimension}}; kw...) => DimArray
Create a DimArray
of ones.
There are two kinds of Dimension
value acepted:
A
Dimension
holding anAbstractVector
will set the dimension index to thatAbstractVector
, and detect the dimension lookup.A
Dimension
holding anInteger
will set the length of the axis, and set the dimension lookup toNoLookup
.
Keywords are the same as for DimArray
.
Example
julia> using DimensionalData
julia> ones(Bool, X(2), Y(4))
╭──────────────────────╮
│ 2×4 DimArray{Bool,2} │
├──────────────── dims ┤
↓ X, → Y
└──────────────────────┘
1 1 1 1
1 1 1 1
julia> ones(X([:a, :b, :c]), Y(100.0:50:200.0))
╭─────────────────────────╮
│ 3×3 DimArray{Float64,2} │
├─────────────────────────┴───────────────────────────────────── dims ┐
↓ X Categorical{Symbol} [:a, :b, :c] ForwardOrdered,
→ Y Sampled{Float64} 100.0:50.0:200.0 ForwardOrdered Regular Points
└─────────────────────────────────────────────────────────────────────┘
↓ → 100.0 150.0 200.0
:a 1.0 1.0 1.0
:b 1.0 1.0 1.0
:c 1.0 1.0 1.0
Functions for getting information from objects:
dims(x, [dims::Tuple]) => Tuple{Vararg{Dimension}}
dims(x, dim) => Dimension
Return a tuple of Dimension
s for an object, in the order that matches the axes or columns of the underlying data.
dims
can be Dimension
, Dimension
types, or Symbols
for Dim{Symbol}
.
The default is to return nothing
.
dims(x, query) => Tuple{Vararg{Dimension}}
dims(x, query...) => Tuple{Vararg{Dimension}}
Get the dimension(s) matching the type(s) of the query dimension.
Lookup can be an Int or an Dimension, or a tuple containing any combination of either.
Arguments
x
: any object with adims
method, or aTuple
ofDimension
.query
: Tuple or a singleDimension
orDimension
Type
.
Example
julia> using DimensionalData
julia> A = DimArray(ones(2, 3, 2), (X, Y, Z))
╭───────────────────────────╮
│ 2×3×2 DimArray{Float64,3} │
├───────────────────── dims ┤
↓ X, → Y, ↗ Z
└───────────────────────────┘
[:, :, 1]
1.0 1.0 1.0
1.0 1.0 1.0
julia> dims(A, (X, Y))
↓ X, → Y
refdims(x, [dims::Tuple]) => Tuple{Vararg{Dimension}}
refdims(x, dim) => Dimension
Reference dimensions for an array that is a slice or view of another array with more dimensions.
slicedims(a, dims)
returns a tuple containing the current new dimensions and the new reference dimensions. Refdims can be stored in a field or discarded, as it is mostly to give context to plots. Ignoring refdims will simply leave some captions empty.
The default is to return an empty Tuple
()
.
metadata(x) => (object metadata)
metadata(x, dims::Tuple) => Tuple (Dimension metadata)
metadata(xs::Tuple) => Tuple
Returns the metadata for an object or for the specified dimension(s)
Second argument dims
can be Dimension
s, Dimension
types, or Symbols
for Dim{Symbol}
.
name(x) => Symbol
name(xs:Tuple) => NTuple{N,Symbol}
name(x, dims::Tuple) => NTuple{N,Symbol}
name(x, dim) => Symbol
Get the name of an array or Dimension, or a tuple of of either as a Symbol.
Second argument dims
can be Dimension
s, Dimension
types, or Symbols
for Dim{Symbol}
.
otherdims(x, query) => Tuple{Vararg{Dimension,N}}
Get the dimensions of an object not in query
.
Arguments
x
: any object with adims
method, aTuple
ofDimension
.query
: Tuple or singleDimension
or dimensionType
.f
:<:
by default, but can be>:
to match abstract types to concrete types.
A tuple holding the unmatched dimensions is always returned.
Example
julia> using DimensionalData, DimensionalData.Dimensions
julia> A = DimArray(ones(10, 10, 10), (X, Y, Z));
julia> otherdims(A, X)
↓ Y, → Z
julia> otherdims(A, (Y, Z))
↓ X
dimnum(x, query::Tuple) => NTuple{Int}
dimnum(x, query) => Int
Get the number(s) of Dimension
(s) as ordered in the dimensions of an object.
Arguments
x
: any object with adims
method, aTuple
ofDimension
or a singleDimension
.query
: Tuple, Array or singleDimension
or dimensionType
.
The return type will be a Tuple of Int
or a single Int
, depending on whether query
is a Tuple
or single Dimension
.
Example
julia> using DimensionalData
julia> A = DimArray(ones(10, 10, 10), (X, Y, Z));
julia> dimnum(A, (Z, X, Y))
(3, 1, 2)
julia> dimnum(A, Y)
2
hasdim([f], x, query::Tuple) => NTuple{Bool}
hasdim([f], x, query...) => NTuple{Bool}
hasdim([f], x, query) => Bool
Check if an object x
has dimensions that match or inherit from the query
dimensions.
Arguments
x
: any object with adims
method, aTuple
ofDimension
or a singleDimension
.query
: Tuple or singleDimension
or dimensionType
.f
:<:
by default, but can be>:
to match abstract types to concrete types.
Check if an object or tuple contains an Dimension
, or a tuple of dimensions.
Example
julia> using DimensionalData
julia> A = DimArray(ones(10, 10, 10), (X, Y, Z));
julia> hasdim(A, X)
true
julia> hasdim(A, (Z, X, Y))
(true, true, true)
julia> hasdim(A, Ti)
false
Multi-array datasets
AbstractDimStack
Abstract supertype for dimensional stacks.
These have multiple layers of data, but share dimensions.
Notably, their behaviour lies somewhere between a DimArray
and a NamedTuple
:
indexing with a
Symbol
as indimstack[:symbol]
returns aDimArray
layer.iteration and
map
apply over array layers, as indexed with aSymbol
.getindex
and many base methods are applied as forDimArray
- to avoid the need to always usemap
.
This design gives very succinct code when working with many-layered, mixed-dimension objects. But it may be jarring initially - the most surprising outcome is that dimstack[1]
will return a NamedTuple
of values for the first index in all layers, while first(dimstack)
will return the first value of the iterator - the DimArray
for the first layer.
See DimStack
for the concrete implementation. Most methods are defined on the abstract type.
To extend AbstractDimStack
, implement argument and keyword version of rebuild
and also rebuild_from_arrays
.
The constructor of an AbstractDimStack
must accept a NamedTuple
.
DimStack <: AbstractDimStack
DimStack(data::AbstractDimArray...; kw...)
DimStack(data::Tuple{Vararg{AbstractDimArray}}; kw...)
DimStack(data::NamedTuple{Keys,Vararg{AbstractDimArray}}; kw...)
DimStack(data::NamedTuple, dims::DimTuple; metadata=NoMetadata(); kw...)
DimStack holds multiple objects sharing some dimensions, in a NamedTuple
.
Notably, their behaviour lies somewhere between a DimArray
and a NamedTuple
:
indexing with a
Symbol
as indimstack[:symbol]
returns aDimArray
layer.iteration and
map
apply over array layers, as indexed with aSymbol
.getindex
orview
withInt
,Dimension
s orSelector
s that resolve toInt
will return aNamedTuple
of values from each layer in the stack. This has very good performance, and avoids the need to always usemap
.getindex
orview
with aVector
orColon
will return anotherDimStack
where all data layers have been sliced.setindex!
must pass aTuple
orNamedTuple
matching the layers.many base and
Statistics
methods (sum
,mean
etc) will work as for aDimArray
again removing the need to usemap
.
function DimStack(A::AbstractDimArray;
layersfrom=nothing, name=nothing, metadata=metadata(A), refdims=refdims(A), kw...
)
For example, here we take the mean over the time dimension for all layers:
mean(mydimstack; dims=Ti)
And this equivalent to:
map(A -> mean(A; dims=Ti), mydimstack)
This design gives succinct code when working with many-layered, mixed-dimension objects.
But it may be jarring initially - the most surprising outcome is that dimstack[1]
will return a NamedTuple
of values for the first index in all layers, while first(dimstack)
will return the first value of the iterator - the DimArray
for the first layer.
DimStack
can be constructed from multiple AbstractDimArray
or a NamedTuple
of AbstractArray
and a matching dims
tuple.
Most Base
and Statistics
methods that apply to AbstractArray
can be used on all layers of the stack simulataneously. The result is a DimStack
, or a NamedTuple
if methods like mean
are used without dims
arguments, and return a single non-array value.
Example
julia> using DimensionalData
julia> A = [1.0 2.0 3.0; 4.0 5.0 6.0];
julia> dimz = (X([:a, :b]), Y(10.0:10.0:30.0))
↓ X [:a, :b],
→ Y 10.0:10.0:30.0
julia> da1 = DimArray(1A, dimz; name=:one);
julia> da2 = DimArray(2A, dimz; name=:two);
julia> da3 = DimArray(3A, dimz; name=:three);
julia> s = DimStack(da1, da2, da3);
julia> s[At(:b), At(10.0)]
(one = 4.0, two = 8.0, three = 12.0)
julia> s[X(At(:a))] isa DimStack
true
Dimension generators
DimIndices <: AbstractArray
DimIndices(x)
DimIndices(dims::Tuple)
DimIndices(dims::Dimension)
Like CartesianIndices
, but for Dimension
s. Behaves as an Array
of Tuple
of Dimension(i)
for all combinations of the axis indices of dims
.
This can be used to view/index into arbitrary dimensions over an array, and is especially useful when combined with otherdims
, to iterate over the indices of unknown dimension.
DimIndices
can be used directly in getindex
like CartesianIndices
, and freely mixed with individual Dimension
s or tuples of Dimension
.
Example
Index a DimArray
with DimIndices
.
Notice that unlike CartesianIndices, it doesn't matter if the dimensions are not in the same order. Or even if they are not all contained in each.
julia> A = rand(Y(0.0:0.3:1.0), X('a':'f'))
╭─────────────────────────╮
│ 4×6 DimArray{Float64,2} │
├─────────────────────────┴───────────────────────────────── dims ┐
↓ Y Sampled{Float64} 0.0:0.3:0.9 ForwardOrdered Regular Points,
→ X Categorical{Char} 'a':1:'f' ForwardOrdered
└─────────────────────────────────────────────────────────────────┘
↓ → 'a' 'b' 'c' 'd' 'e' 'f'
0.0 0.9063 0.253849 0.0991336 0.0320967 0.774092 0.893537
0.3 0.443494 0.334152 0.125287 0.350546 0.183555 0.354868
0.6 0.745673 0.427328 0.692209 0.930332 0.297023 0.131798
0.9 0.512083 0.867547 0.136551 0.959434 0.150155 0.941133
julia> di = DimIndices((X(1:2:4), Y(1:2:4)))
╭─────────────────────────────────────────────╮
│ 2×2 DimIndices{Tuple{X{Int64}, Y{Int64}},2} │
├─────────────────────────────────────── dims ┤
↓ X 1:2:3,
→ Y 1:2:3
└─────────────────────────────────────────────┘
↓ → 1 3
1 ↓ X 1, → Y 1 ↓ X 1, → Y 3
3 ↓ X 3, → Y 1 ↓ X 3, → Y 3
julia> A[di] # Index A with these indices
╭─────────────────────────╮
│ 2×2 DimArray{Float64,2} │
├─────────────────────────┴───────────────────────────────── dims ┐
↓ Y Sampled{Float64} 0.0:0.6:0.6 ForwardOrdered Regular Points,
→ X Categorical{Char} 'a':2:'c' ForwardOrdered
└─────────────────────────────────────────────────────────────────┘
↓ → 'a' 'c'
0.0 0.9063 0.0991336
0.6 0.745673 0.692209
DimSelectors <: AbstractArray
DimSelectors(x; selectors, atol...)
DimSelectors(dims::Tuple; selectors, atol...)
DimSelectors(dims::Dimension; selectors, atol...)
Like DimIndices
, but returns Dimensions
holding the chosen Selector
s.
Indexing into another AbstractDimArray
with DimSelectors
is similar to doing an interpolation.
Keywords
selectors
:Near
,At
orContains
, or a mixed tuple of these.At
is the default, meaning only exact or withinatol
values are used.atol
: used forAt
selectors only, as theatol
value.
Example
Here we can interpolate a DimArray
to the lookups of another DimArray
using DimSelectors
with Near
. This is essentially equivalent to nearest neighbour interpolation.
julia> A = rand(X(1.0:3.0:30.0), Y(1.0:5.0:30.0), Ti(1:2));
julia> target = rand(X(1.0:10.0:30.0), Y(1.0:10.0:30.0));
julia> A[DimSelectors(target; selectors=Near), Ti=2]
╭─────────────────────────╮
│ 3×3 DimArray{Float64,2} │
├─────────────────────────┴───────────────────────────────────────── dims ┐
↓ X Sampled{Float64} [1.0, 10.0, 22.0] ForwardOrdered Irregular Points,
→ Y Sampled{Float64} [1.0, 11.0, 21.0] ForwardOrdered Irregular Points
└─────────────────────────────────────────────────────────────────────────┘
↓ → 1.0 11.0 21.0
1.0 0.691162 0.218579 0.539076
10.0 0.0303789 0.420756 0.485687
22.0 0.0967863 0.864856 0.870485
Using At
would make sure we only use exact interpolation, while Contains
with sampling of Intervals
would make sure that each values is taken only from an Interval that is present in the lookups.
DimPoints <: AbstractArray
DimPoints(x; order)
DimPoints(dims::Tuple; order)
DimPoints(dims::Dimension; order)
Like CartesianIndices
, but for the point values of the dimension index. Behaves as an Array
of Tuple
lookup values (whatever they are) for all combinations of the lookup values of dims
.
Either a Dimension
, a Tuple
of Dimension
or an object x
that defines a dims
method can be passed in.
Keywords
order
: determines the order of the points, the same as the order ofdims
by default.
Tables.jl/TableTraits.jl interface
AbstractDimTable <: Tables.AbstractColumns
Abstract supertype for dim tables
DimTable <: AbstractDimTable
DimTable(s::AbstractDimStack; mergedims=nothing)
DimTable(x::AbstractDimArray; layersfrom=nothing, mergedims=nothing)
DimTable(xs::Vararg{AbstractDimArray}; layernames=nothing, mergedims=nothing)
Construct a Tables.jl/TableTraits.jl compatible object out of an AbstractDimArray
or AbstractDimStack
.
This table will have columns for the array data and columns for each Dimension
index, as a [DimColumn
]. These are lazy, and generated as required.
Column names are converted from the dimension types using DimensionalData.name
. This means type Ti
becomes the column name :Ti
, and Dim{:custom}
becomes :custom
.
To get dimension columns, you can index with Dimension
(X()
) or Dimension
type (X
) as well as the regular Int
or Symbol
.
Keywords
mergedims
: Combine two or more dimensions into a new dimension.layersfrom
: Treat a dimension of anAbstractDimArray
as layers of anAbstractDimStack
.
Example
julia> using DimensionalData, Tables
julia> a = DimArray(ones(16, 16, 3), (X, Y, Dim{:band}))
╭─────────────────────────────╮
│ 16×16×3 DimArray{Float64,3} │
├─────────────────────── dims ┤
↓ X, → Y, ↗ band
└─────────────────────────────┘
[:, :, 1]
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 … 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 … 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 … 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 … 1.0 1.0 1.0 1.0 1.0 1.0 1.0
julia>
Group by methods
For transforming DimensionalData objects:
groupby(A::Union{AbstractDimArray,AbstractDimStack}, dims::Pair...)
groupby(A::Union{AbstractDimArray,AbstractDimStack}, dims::Dimension{<:Callable}...)
Group A
by grouping functions or Bins
over multiple dimensions.
Arguments
A
: anyAbstractDimArray
orAbstractDimStack
.dims
:Pair
s such asgroups = groupby(A, :dimname => groupingfunction)
or wrappedDimension
s likegroups = groupby(A, DimType(groupingfunction))
. Instead of a grouping functionBins
can be used to specify group bins.
Return value
A DimGroupByArray
is returned, which is basically a regular AbstractDimArray
but holding the grouped AbstractDimArray
or AbstractDimStack
. Its dims
hold the sorted values returned by the grouping function/s.
Base julia and package methods work on DimGroupByArray
as for any other AbstractArray
of AbstractArray
.
It is common to broadcast or map
a reducing function over groups, such as mean
or sum
, like mean.(groups)
or map(mean, groups)
. This will return a regular DimArray
, or DimGroupByArray
if dims
keyword is used in the reducing function or it otherwise returns an AbstractDimArray
or AbstractDimStack
.
Example
Group some data along the time dimension:
julia> using DimensionalData, Dates
julia> A = rand(X(1:0.1:20), Y(1:20), Ti(DateTime(2000):Day(3):DateTime(2003)));
julia> groups = groupby(A, Ti => month) # Group by month
╭───────────────────────────────────────────────────╮
│ 12-element DimGroupByArray{DimArray{Float64,2},1} │
├───────────────────────────────────────────────────┴───────────── dims ┐
↓ Ti Sampled{Int64} [1, 2, …, 11, 12] ForwardOrdered Irregular Points
├───────────────────────────────────────────────────────────── metadata ┤
Dict{Symbol, Any} with 1 entry:
:groupby => :Ti=>month
├─────────────────────────────────────────────────────────── group dims ┤
↓ X, → Y, ↗ Ti
└───────────────────────────────────────────────────────────────────────┘
1 191×20×32 DimArray
2 191×20×28 DimArray
3 191×20×31 DimArray
⋮
11 191×20×30 DimArray
12 191×20×31 DimArray
And take the mean:
julia> groupmeans = mean.(groups) # Take the monthly mean
╭────────────────────────────────╮
│ 12-element DimArray{Float64,1} │
├────────────────────────────────┴──────────────────────────────── dims ┐
↓ Ti Sampled{Int64} [1, 2, …, 11, 12] ForwardOrdered Irregular Points
├───────────────────────────────────────────────────────────── metadata ┤
Dict{Symbol, Any} with 1 entry:
:groupby => :Ti=>month
└───────────────────────────────────────────────────────────────────────┘
1 0.500064
2 0.499762
3 0.500083
4 0.499985
⋮
10 0.500874
11 0.498704
12 0.50047
Calculate daily anomalies from the monthly mean. Notice we map a broadcast .-
rather than -
. This is because the size of the arrays to not match after application of mean
.
julia> map(.-, groupby(A, Ti=>month), mean.(groupby(A, Ti=>month), dims=Ti));
Or do something else with Y:
julia> groupmeans = mean.(groupby(A, Ti=>month, Y=>isodd))
╭──────────────────────────╮
│ 12×2 DimArray{Float64,2} │
├──────────────────────────┴─────────────────────────────────────── dims ┐
↓ Ti Sampled{Int64} [1, 2, …, 11, 12] ForwardOrdered Irregular Points,
→ Y Sampled{Bool} [false, true] ForwardOrdered Irregular Points
├────────────────────────────────────────────────────────────── metadata ┤
Dict{Symbol, Any} with 1 entry:
:groupby => (:Ti=>month, :Y=>isodd)
└────────────────────────────────────────────────────────────────────────┘
↓ → false true
1 0.499594 0.500533
2 0.498145 0.501379
⋮
10 0.501105 0.500644
11 0.498606 0.498801
12 0.501643 0.499298
DimGroupByArray <: AbstractDimArray
DimGroupByArray
is essentially a DimArray
but holding the results of a groupby
operation.
Its dimensions are the sorted results of the grouping functions used in groupby
.
This wrapper allows for specialisations on later broadcast or reducing operations, e.g. for chunk reading with DiskArrays.jl, because we know the data originates from a single array.
Bins(f, bins; labels, pad)
Bins(bins; labels, pad)
Specify bins to reduce groups after applying function f
.
f
: a grouping function of the lookup values, by defaultidentity
.bins
:an
Integer
will divide the group values into equally spaced sections.an
AbstractArray
of values will be treated as exact matches for the return value off
. For example,1:3
will create 3 bins - 1, 2, 3.an
AbstractArray
ofIntervalSets.Interval
can be used to explicitly define the intervals. Overlapping intervals have undefined behaviour.
Keywords
pad
: fraction of the total interval to pad at each end whenBins
contains an
<<<<<<< HEAD Integer
. This avoids losing the edge values. Note this is a messy solution - it will often be preferable to manually specify a Vector
of chosen Interval
s rather than relying on passing an Integer
and pad
. ======= Integer
. This avoids losing the edge values. Note this is a messy solution - it will often be prefereble to manually specify a Vector
of chosen Interval
s rather than relying on passing an Integer
and pad
.
parent of c54bc6c (Fix whitespace)
labels
: a list of descriptive labels for the bins. The labels need to have the same length asbins
.
When the return value of f
is a tuple, binning is applied to the last value of the tuples.
ranges(A::AbstractRange{<:Integer})
Generate a Vector
of UnitRange
with length step(A)
intervals(A::AbstractRange)
Generate a Vector
of UnitRange
with length step(A)
CyclicBins(f; cycle, start, step, labels)
Cyclic bins to reduce groups after applying function f
. Groups can wrap around the cycle. This is used for grouping in seasons
, months
and hours
but can also be used for custom cycles.
f
: a grouping function of the lookup values, by defaultidentity
.
Keywords
cycle
: the length of the cycle, in return values off
.start
: the start of the cycle: a return value off
.step
the number of sequential values to group.labels
: either a vector of labels matching the number of groups, or a function that generates labels fromVector{Int}
of the selected bins.
When the return value of f
is a tuple, binning is applied to the last value of the tuples.
seasons(; [start=Dates.December, labels])
Generates CyclicBins
for three month periods.
Keywords
start
: By default seasons start in December, but any integer1:12
can be used.labels
: either a vector of four labels, or a function that generates labels
<<<<<<< HEAD from Vector{Int}
of the selected quarters. ======= from Vector{Int}
of the selected quartals.
parent of c54bc6c (Fix whitespace)
months(step; [start=Dates.January, labels])
Generates CyclicBins
for grouping to arbitrary month periods. These can wrap around the end of a year.
step
the number of months to group.
Keywords
start
: By default months start in January, but any integer1:12
can be used.labels
: either a vector of labels matching the number of groups, or a function that generates labels fromVector{Int}
of the selected months.
hours(step; [start=0, labels])
Generates CyclicBins
for grouping to arbitrary hour periods. These can wrap around the end of the day.
steps
the number of hours to group.
Keywords
start
: By default seasons start at0
, but any integer1:24
can be used.labels
: either a vector of four labels, or a function that generates labels fromVector{Int}
of the selected hours of the day.
Utility methods
For transforming DimensionalData objects:
set(x, val)
set(x, args::Pairs...) => x with updated field/s
set(x, args...; kw...) => x with updated field/s
set(x, args::Tuple{Vararg{Dimension}}; kw...) => x with updated field/s
set(dim::Dimension, index::AbstractArray) => Dimension
set(dim::Dimension, lookup::Lookup) => Dimension
set(dim::Dimension, lookupcomponent::LookupTrait) => Dimension
set(dim::Dimension, metadata::AbstractMetadata) => Dimension
Set the properties of an object, its internal data or the traits of its dimensions and lookup index.
As DimensionalData is so strongly typed you do not need to specify what field of a Lookup
to set
- there is no ambiguity.
To set fields of a Lookup
you need to specify the dimension. This can be done using X => val
pairs, X = val
keyword arguments, or X(val)
wrapped arguments.
When a Dimension
or Lookup
is passed to set
to replace the existing ones, fields that are not set will keep their original values.
Notes:
Changing a lookup index range/vector will also update the step size and order where applicable.
Setting the Order
like ForwardOrdered
will not reverse the array or dimension to match. Use reverse
and reorder
to do this.
Examples
julia> using DimensionalData; const DD = DimensionalData;
julia> da = DimArray(zeros(3, 4), (custom=10.0:010.0:30.0, Z=-20:010.0:10.0));
julia> set(da, ones(3, 4))
╭─────────────────────────╮
│ 3×4 DimArray{Float64,2} │
├─────────────────────────┴───────────────────────────────────────── dims ┐
↓ custom Sampled{Float64} 10.0:10.0:30.0 ForwardOrdered Regular Points,
→ Z Sampled{Float64} -20.0:10.0:10.0 ForwardOrdered Regular Points
└─────────────────────────────────────────────────────────────────────────┘
↓ → -20.0 -10.0 0.0 10.0
10.0 1.0 1.0 1.0 1.0
20.0 1.0 1.0 1.0 1.0
30.0 1.0 1.0 1.0 1.0
Change the Dimension
wrapper type:
julia> set(da, :Z => Ti, :custom => Z)
╭─────────────────────────╮
│ 3×4 DimArray{Float64,2} │
├─────────────────────────┴───────────────────────────────────── dims ┐
↓ Z Sampled{Float64} 10.0:10.0:30.0 ForwardOrdered Regular Points,
→ Ti Sampled{Float64} -20.0:10.0:10.0 ForwardOrdered Regular Points
└─────────────────────────────────────────────────────────────────────┘
↓ → -20.0 -10.0 0.0 10.0
10.0 0.0 0.0 0.0 0.0
20.0 0.0 0.0 0.0 0.0
30.0 0.0 0.0 0.0 0.0
Change the lookup Vector
:
julia> set(da, Z => [:a, :b, :c, :d], :custom => [4, 5, 6])
╭─────────────────────────╮
│ 3×4 DimArray{Float64,2} │
├─────────────────────────┴───────────────────────────────────────── dims ┐
↓ custom Sampled{Int64} [4, 5, 6] ForwardOrdered Regular Points,
→ Z Sampled{Symbol} [:a, :b, :c, :d] ForwardOrdered Regular Points
└─────────────────────────────────────────────────────────────────────────┘
↓ → :a :b :c :d
4 0.0 0.0 0.0 0.0
5 0.0 0.0 0.0 0.0
6 0.0 0.0 0.0 0.0
Change the Lookup
type:
julia> set(da, Z=DD.NoLookup(), custom=DD.Sampled())
╭─────────────────────────╮
│ 3×4 DimArray{Float64,2} │
├─────────────────────────┴───────────────────────────────────────── dims ┐
↓ custom Sampled{Float64} 10.0:10.0:30.0 ForwardOrdered Regular Points,
→ Z
└─────────────────────────────────────────────────────────────────────────┘
10.0 0.0 0.0 0.0 0.0
20.0 0.0 0.0 0.0 0.0
30.0 0.0 0.0 0.0 0.0
Change the Sampling
trait:
julia> set(da, :custom => DD.Irregular(10, 12), Z => DD.Regular(9.9))
╭─────────────────────────╮
│ 3×4 DimArray{Float64,2} │
├─────────────────────────┴─────────────────────────────────────────── dims ┐
↓ custom Sampled{Float64} 10.0:10.0:30.0 ForwardOrdered Irregular Points,
→ Z Sampled{Float64} -20.0:10.0:10.0 ForwardOrdered Regular Points
└───────────────────────────────────────────────────────────────────────────┘
↓ → -20.0 -10.0 0.0 10.0
10.0 0.0 0.0 0.0 0.0
20.0 0.0 0.0 0.0 0.0
30.0 0.0 0.0 0.0 0.0
rebuild(x; kw...)
Rebuild an object struct with updated field values.
x
can be a AbstractDimArray
, a Dimension
, Lookup
or other custom types.
This is an abstraction that allows inbuilt and custom types to be rebuilt to update their fields, as most objects in DimensionalData.jl are immutable.
Rebuild is mostly automated using ConstructionBase.setproperties
. It should only be defined if your object has fields with with different names to DimensionalData objects. Try not to do that!
The arguments required are defined for the abstract type that has a rebuild
method.
AbstractBasicDimArray
:
dims
: aTuple
ofDimension
AbstractDimArray
:
data
: the parent object - anAbstractArray
dims
: aTuple
ofDimension
refdims
: aTuple
ofDimension
name
: A Symbol, orNoName
andName
on GPU.metadata
: ADict
-like object
AbstractDimStack
:
data
: the parent object, often aNamedTuple
dims
,refdims
,metadata
Dimension
:
val
: anything.
Lookup
:
data
: the parent object, anAbstractArray
Note: argument
rebuild
is deprecated onAbstractDimArray
and
AbstractDimStack
in favour of always using the keyword version. In future the argument version will only be used on Dimension
, which only have one argument.
modify(f, A::AbstractDimArray) => AbstractDimArray
modify(f, s::AbstractDimStack) => AbstractDimStack
modify(f, dim::Dimension) => Dimension
modify(f, x, lookupdim::Dimension) => typeof(x)
Modify the parent data, rebuilding the object wrapper without change. f
must return a AbstractArray
of the same size as the original.
This method is mostly useful as a way of swapping the parent array type of an object.
Example
If we have a previously-defined DimArray
, we can copy it to an Nvidia GPU with:
A = DimArray(rand(100, 100), (X, Y))
modify(CuArray, A)
This also works for all the data layers in a DimStack
.
broadcast_dims(f, sources::AbstractDimArray...) => AbstractDimArray
Broadcast function f
over the AbstractDimArray
s in sources
, permuting and reshaping dimensions to match where required. The result will contain all the dimensions in all passed in arrays in the order in which they are found.
Arguments
sources
:AbstractDimArrays
to broadcast over withf
.
This is like broadcasting over every slice of A
if it is sliced by the dimensions of B
.
broadcast_dims!(f, dest::AbstractDimArray, sources::AbstractDimArray...) => dest
Broadcast function f
over the AbstractDimArray
s in sources
, writing to dest
. sources
are permuting and reshaping dimensions to match where required.
The result will contain all the dimensions in all passed in arrays, in the order in which they are found.
Arguments
dest
:AbstractDimArray
to update.sources
:AbstractDimArrays
to broadcast over withf
.
mergedims(old_dims => new_dim) => Dimension
Return a dimension new_dim
whose indices are a MergedLookup
of the indices of old_dims
.
mergedims(dims, old_dims => new_dim, others::Pair...) => dims_new
If dimensions old_dims
, new_dim
, etc. are found in dims
, then return new dims_new
where all dims in old_dims
have been combined into a single dim new_dim
. The returned dimension will keep only the name of new_dim
. Its coords will be a MergedLookup
of the coords of the dims in old_dims
. New dimensions are always placed at the end of dims_new
. others
contains other dimension pairs to be merged.
Example
julia> using DimensionalData
julia> ds = (X(0:0.1:0.4), Y(10:10:100), Ti([0, 3, 4]))
↓ X 0.0:0.1:0.4,
→ Y 10:10:100,
↗ Ti [0, 3, 4]
julia> mergedims(ds, (X, Y) => :space)
↓ Ti [0, 3, 4],
→ space MergedLookup{Tuple{Float64, Int64}} [(0.0, 10), (0.1, 10), …, (0.3, 100), (0.4, 100)] ↓ X, → Y
mergedims(A::AbstractDimArray, dim_pairs::Pair...) => AbstractDimArray
mergedims(A::AbstractDimStack, dim_pairs::Pair...) => AbstractDimStack
Return a new array or stack whose dimensions are the result of mergedims(dims(A), dim_pairs)
.
unmergedims(merged_dims::Tuple{Vararg{Dimension}}) => Tuple{Vararg{Dimension}}
Return the unmerged dimensions from a tuple of merged dimensions. However, the order of the original dimensions are not necessarily preserved.
unmergedims(A::AbstractDimArray, original_dims) => AbstractDimArray
unmergedims(A::AbstractDimStack, original_dims) => AbstractDimStack
Return a new array or stack whose dimensions are restored to their original prior to calling mergedims(A, dim_pairs)
.
reorder(A::Union{AbstractDimArray,AbstractDimStack}, order::Pair...)
reorder(A::Union{AbstractDimArray,AbstractDimStack}, order)
reorder(A::Dimension, order::Order)
Reorder every dims index/array to order
, or reorder index for the given dimension(s) in order
.
order
can be an Order
, Dimension => Order
pairs. A Tuple of Dimensions or any object that defines dims
can be used in which case the dimensions of this object are used for reordering.
If no axis reversal is required the same objects will be returned, without allocation.
Example
using DimensionalData
# Create a DimArray
da = DimArray([1 2 3; 4 5 6], (X(10:10:20), Y(300:-100:100)))
# Reverse it
rev = reverse(da, dims=Y)
# using `da` in reorder will return it to the original order
reorder(rev, da) == da
# output
true
Base methods
Base.cat(stacks::AbstractDimStack...; [keys=keys(stacks[1])], dims)
Concatenate all or a subset of layers for all passed in stacks.
Keywords
keys
:Tuple
ofSymbol
for the stack keys to concatenate.dims
: Dimension of child array to concatenate on.
Example
Concatenate the :sea_surface_temp and :humidity layers in the time dimension:
cat(stacks...; keys=(:sea_surface_temp, :humidity), dims=Ti)
Base.copy!(dst::AbstractArray, src::AbstractDimStack, key::Key)
Copy the stack layer key
to dst
, which can be any AbstractArray
.
Example
Copy the :humidity
layer from stack
to array
.
copy!(array, stack, :humidity)
Base.copy!(dst::AbstractDimStack, src::AbstractDimStack, [keys=keys(dst)])
Copy all or a subset of layers from one stack to another.
Example
Copy just the :sea_surface_temp
and :humidity
layers from src
to dst
.
copy!(dst::AbstractDimStack, src::AbstractDimStack, keys=(:sea_surface_temp, :humidity))
Base.eachslice(A::AbstractDimArray; dims,drop=true)
Create a generator that iterates over dimensions dims
of A
, returning arrays that select all the data from the other dimensions in A
using views.
The generator has size
and axes
equivalent to those of the provided dims
if drop=true
. Otherwise it will have the same dimensionality as the underlying array with inner dimensions having size 1.
Base.eachslice(stack::AbstractDimStack; dims, drop=true)
Create a generator that iterates over dimensions dims
of stack
, returning stacks that select all the data from the other dimensions in stack
using views.
The generator has size
and axes
equivalent to those of the provided dims
.
Examples
julia> ds = DimStack((
x=DimArray(randn(2, 3, 4), (X([:x1, :x2]), Y(1:3), Z)),
y=DimArray(randn(2, 3, 5), (X([:x1, :x2]), Y(1:3), Ti))
));
julia> slices = eachslice(ds; dims=(Z, X));
julia> size(slices)
(4, 2)
julia> map(dims, axes(slices))
↓ Z Base.OneTo(4),
→ X Base.OneTo(2)
julia> first(slices)
╭──────────────╮
│ 3×5 DimStack │
├──────────────┴─────────────────────────────────── dims ┐
↓ Y Sampled{Int64} 1:3 ForwardOrdered Regular Points,
→ Ti
├──────────────────────────────────────────────── layers ┤
:x eltype: Float64 dims: Y size: 3
:y eltype: Float64 dims: Y, Ti size: 3×5
└────────────────────────────────────────────────────────┘
Most base methods work as expected, using Dimension
wherever a dims
keyword is used. They are not all specifically documented here.
Name
Name <: AbstractName
Name(name::Union{Symbol,Name) => Name
Name(name::NoName) => NoName
Name wrapper. This lets arrays keep symbol names when the array wrapper needs to be isbits
, like for use on GPUs. It makes the name a property of the type. It's not necessary to use in normal use, a symbol is probably easier.
NoName <: AbstractName
NoName()
NoName specifies an array is not named, and is the default name
value for all AbstractDimArray
s.
Internal interface
DimArrayInterface
An Interfaces.jl Interface
with mandatory components (:dims, :refdims_base, :ndims, :size, :rebuild_parent, :rebuild_dims, :rebuild_parent_kw, :rebuild_dims_kw, :rebuild)
and optional components (:refdims, :name, :metadata)
.
This is an early stage of inteface definition, many things are not yet tested.
Pass constructed AbstractDimArrays as test data.
They must not be zero dimensional, and should test at least 1, 2, and 3 dimensions.
Extended help
Mandatory keys:
dims
:defines a
dims
methoddims are updated on getindex
refdims_base
:refdims
returns a tuple of Dimension or emptyndims
: number of dims matches dimensions of arraysize
: length of dims matches dimensions of arrayrebuild_parent
: rebuild parent from argsrebuild_dims
: rebuild paaarnet and dims from argsrebuild_parent_kw
: rebuild parent from argsrebuild_dims_kw
: rebuild dims from argsrebuild
: all rebuild arguments and keywords are accepted
Optional keys:
refdims
:refdims are updated in args rebuild
refdims are updated in kw rebuild
dropped dimensions are added to refdims
name
:rebuild updates name in arg rebuild
rebuild updates name in kw rebuild
metadata
:rebuild updates metadata in arg rebuild
rebuild updates metadata in kw rebuild
DimStackInterface
An Interfaces.jl Interface
with mandatory components (:dims, :refdims_base, :ndims, :size, :rebuild_parent, :rebuild_dims, :rebuild_layerdims, :rebuild_dims_kw, :rebuild_parent_kw, :rebuild_layerdims_kw, :rebuild)
and optional components (:refdims, :metadata)
.
This is an early stage of inteface definition, many things are not yet tested.
Pass constructed AbstractDimArrays as test data.
They must not be zero dimensional, and should test at least 1, 2, and 3 dimensions.
Extended help
Mandatory keys:
dims
:defines a
dims
methoddims are updated on getindex
refdims_base
:refdims
returns a tuple of Dimension or emptyndims
: number of dims matches ndims of stacksize
: length of dims matches size of stackrebuild_parent
: rebuild parent from argsrebuild_dims
: rebuild paaarnet and dims from argsrebuild_layerdims
: rebuild paaarnet and dims from argsrebuild_dims_kw
: rebuild dims from argsrebuild_parent_kw
: rebuild parent from argsrebuild_layerdims_kw
: rebuild parent from argsrebuild
: all rebuild arguments and keywords are accepted
Optional keys:
refdims
:refdims are updated in args rebuild
refdims are updated in kw rebuild
dropped dimensions are added to refdims
metadata
:rebuild updates metadata in arg rebuild
rebuild updates metadata in kw rebuild
rebuild_from_arrays(s::AbstractDimStack, das::NamedTuple{<:Any,<:Tuple{Vararg{AbstractDimArray}}}; kw...)
Rebuild an AbstractDimStack
from a Tuple
or NamedTuple
of AbstractDimArray
and an existing stack.
Keywords
Keywords are simply the fields of the stack object:
data
dims
refdims
metadata
layerdims
layermetadata
show_main(io::IO, mime, A::AbstractDimArray)
show_main(io::IO, mime, A::AbstractDimStack)
Interface methods for adding the main part of show
At the least, you likely want to call:
print_top(io, mime, A)
But read the DimensionalData.jl show.jl
code for details.
show_after(io::IO, mime, A::AbstractDimArray)
show_after(io::IO, mime, A::AbstractDimStack)
Interface methods for adding additional show
text for AbstractDimArray/AbstractDimStack subtypes.
Always include kw
to avoid future breaking changes
Additional keywords may be added at any time.
blockwidth
is passed in context
blockwidth = get(io, :blockwidth, 10000)
Note - a ANSI box is left unclosed. This method needs to close it, or add more. blockwidth
is the maximum length of the inner text.
Most likely you always want to at least close the show blocks with:
print_block_close(io, blockwidth)
But read the DimensionalData.jl show.jl
code for details.
refdims_title(A::AbstractDimArray)
refdims_title(refdims::Tuple)
refdims_title(refdim::Dimension)
Generate a title string based on reference dimension values.