DimensionalData.jl

Appearance

336

Xarray and PythonCall.jl ​

In the Python ecosystem Xarray is by far the most popular package for working with multidimensional labelled arrays. The main data structures it provides are:

DimensionalData integrates with PythonCall.jl to allow converting these Xarray types to their DimensionalData equivalent:

julia
import PythonCall: pyconvert

# By default this will share the underlying array
my_dimarray = pyconvert(DimArray, my_dataarray)

my_dimstack = pyconvert(DimStack, my_dataset)

DimensionalData also defines Py() conversion methods for DimArray and DimStack to convert them to a DataArray and Dataset respectively. These are opt-in because they require Xarray to be installed, so they must be enabled first:

julia
DimensionalData.set_xarray_convert!(true)

We recommend restarting Julia after changing the preference. Once it's enabled, passing a DimArray or DimStack to a Python function will automatically convert it. Otherwise they will be passed as opaque Julia objects, though a DimArray will still support the array interface in Python so it can be used with e.g. numpy.

The preference can be overridden for a single conversion with the xarray argument, which is useful if you only occasionally want an Xarray type:

julia
Py(my_dimarray; xarray=true)   # Always a DataArray
Py(my_dimstack; xarray=false)  # Never a Dataset
DimensionalData.set_xarray_convert! Function
julia
set_xarray_convert!(x::Bool)

Set whether converting an AbstractDimArray/AbstractDimStack to Python with PythonCall.Py() should create an xarray.DataArray/xarray.Dataset.

Defaults to false, in which case both are passed through as opaque Julia objects (an AbstractDimArray still supports the array interface in Python). Note that enabling it means that any conversion to Python will import xarray, so it must be installed.

The preference is set for the active project, and restarting Julia afterwards is recommended.

source

Here are some things to keep in mind when converting:

  • pyconvert(DimArray, x) is zero-copy by default, i.e. it will share the underlying array with Python and register itself with Pythons GC to ensure that the memory isn't garbage-collected prematurely. If you want to make a copy you can call it like pyconvert(DimArray, x; copy=true). The same is true for Py(x), it's zero-copy by default and you can pass copy=true to make it copy. An exception to the above is when converting x to x_jl, and x contains missing values. These cannot be represented in Python so in such cases a copy will always be made and the missing values will be converted to NaNs when possible. When conversion is not possible (e.g. because the array holds integers rather floats) an exception will be thrown.

  • When doing a zero-copy conversion from x to x_jl, parent(x_jl) will be a PyArray. In most situations there should be no overhead from this but note that a PyArray is not a DenseArray so some operations that dispatch on DenseArray may not be performant, e.g. BLAS calls. See these issues for more information:

    When copy=true, parent(x_jl) will always be a standard Array. However, we do not consider the type of parent array covered by semver so this may change in the future.

  • Python stores arrays in row-major order whereas Julia stores them in column-major order, hence the dimensions on a converted DimArray will be in reverse order from the original DataArray (and vice versa). This is done to ensure that the 'fast axis' to iterate over is the same dimension in both Julia and Python.

  • Only the metadata of the array/stack itself is converted to attrs. Any metadata attached to lookups (e.g. units) is not passed to the coordinates when converting to Python, and likewise coordinate attrs are not converted to lookup metadata when converting to Julia.