Flatten.jl

Flatten

Flatten.jl converts data from arbitrary nested structs to tuples, using flatten(), and rebuilds them using reconstruct(), update!(). modify() combines flatten and reconstruct with a function application to each element of the intermediate tuple.

This facilitates building modular, composable structs that can be queries, modified and rebuilt based on the types of their fields, without knowing field locations.

It also allows access to solvers and optimisers that require flat lists of parameters. Importantly, it's type-stable and very fast. It does not flatten the contents of Array or Dict (which is a container of Arrays), where the number of values are not known at compile time.

Flatten is also flexible. The types to return and ignore can be specified, and individual fields can be ignored using field-level traits like flattenable from FieldMetadata.jl. Method overrides can also be defined for custom types.

Type queries

Flatten allows a kind of querying to extract some types and ignore others, here using flatten:

julia> using Flatten

julia> struct Foo{A,B,C}
           a::A
           b::B
           c::C
       end

julia> struct Bar{X}
           x::X
       end

julia> obj = Foo(1, :two, Foo(Bar(3), 4.0, 5.0f0));

julia> use = Union{Int, Float32}; # Return Int and Float32 fields

julia> ignore = Bar;              # Dont return Bar or iterate over Bar fields

julia> flatten(obj, use, ignore)  # `flatten` all Int and Float32 except fields of Bar
(1, 5.0f0)

julia> modify(string, obj, Int)   # `modify`: convert all Int to String
Foo{String,Symbol,Foo{Bar{String},Float64,Float32}}("1", :two, Foo{Bar{String},Float64,Float32}(Bar{String}("3"), 4.0, 5.0f0))

The default type used is Real. These rules also apply in reconstruct, update! and modify.

Field removal

There are often cases where you want to ignore certain fields that have the same type as the fields you want to extract. Flatten.jl also FieldMetadata.jl to provide @flattenable macro and methods, allowing you to choose fields to include and remove from flattening. The default is true for all fields.

using Flatten
import Flatten: flattenable

@flattenable struct Bar{X,Y}
    x::X | true
    y::Y | false
end

flatten(Bar(1, 2))

# output
(1,)

Custom @metadata methods from FieldMetadata can be used, if they return a Bool. You can also use custom functions that follow the following form, returning a boolean:

f(::Type, ::Type{Var{:fieldname}}) = false

Metatdata flattening

Flatten also provides metaflatten() to flatten any FieldMetadata.jl metadata for the same fields flatten() returns. This can be useful for attaching information like descriptions or prior probability distributions to each field. Regular field data can also be collected with convenience versions of metaflatten: fieldnameflatten, parentflatten, fieldtypeflatten and parenttypeflatten functions provide lists of fieldnames and types that may be useful for building parameter display tables.

This package was started by Robin Deits (@rdeits), and its early development owes much to discussions and ideas from Jan Weidner (@jw3126) and Robin Deits. """

reconstruct and modify for StaticArrays

SArray and other objects from StaticArrays.jl can not be constructed from their fields. Dealing with this in the long term will require either a dependency on ConstructionBase.jl in StaticArrays.jl, or a glue package that provides the required constructorof methods, which for now you can define manually:

using StaticArrays, ConstructionBase, Flatten

struct SArrayConstructor{S,N,L} end
(::SArrayConstructor{S,N,L})(data::NTuple{L,T}) where {S,T,N,L} = SArray{S,T,N,L}(data)

ConstructionBase.constructorof(sa::Type{<:SArray{S,<:Any,N,L}}) where {S,N,L} = 
    SArrayConstructor{S,N,L}()

jldoctest fieldnameflatten(obj, args...)

Flatten the field names of an object. Args are passed to metaflatten.

Examples

julia> using Flatten

julia> struct Foo{A,B,C}
           a::A
           b::B
           c::C
       end

julia> fieldnameflatten(Foo(1, 2, 3))
(:a, :b, :c)

fieldtypeflatten(obj, args...)

Flatten the field types of an object. Args are passed to metaflatten.

Examples

julia> using Flatten

julia> struct Foo{A,B,C}
           a::A
           b::B
           c::C
       end

julia> fieldtypeflatten(Foo(1.0, :two, "Three"), Union{Real,String})
(Float64, String)

flatten(obj, [flattentrait::Function], [use::Type], [ignore::Type])

Flattening. Flattens a nested struct or tuple to a flat tuple. Query types and flatten trait arguments are optional, but you must pass use to pass ignore.

Arguments

• obj: The target type to be reconstructed
• flattentrait: A function returning a Bool, such as a FielMetadata method. With the form: f(::Type, ::Type{Val{:fieldname}}) = true
• use: Type or Union of types to return in the tuple.
• ignore: Types or Union of types to ignore completly. These are not reurned or recursed over.

Examples

julia> using Flatten

julia> struct Foo{A,B,C}
           a::A
           b::B
           c::C
       end

julia> foo = Foo(1, 2, 3)
Foo{Int64,Int64,Int64}(1, 2, 3)

julia> flatten(foo)
(1, 2, 3)

julia> nested = Foo(Foo(1, 2, 3), 4.0, 5.0)
Foo{Foo{Int64,Int64,Int64},Float64,Float64}(Foo{Int64,Int64,Int64}(1, 2, 3), 4.0, 5.0)

julia> flatten(nested)
(1, 2, 3, 4.0, 5.0)

To convert the tuple to a vector, simply use [flatten(x)...], or
using static arrays to avoid allocations: `SVector(flatten(x))`.

Fields can be excluded from flattening with the flattenable(struct, field) method. These are easily defined using @flattenable from FieldMetadata.jl, or defining your own custom function with FieldMetadata, or manually with the form:

julia> import Flatten: flattenable

julia> @flattenable struct Partial{A,B,C}
           a::A | true
           b::B | true
           c::C | false
       end

julia> nestedpartial = Partial(Partial(1.0, 2.0, 3.0), 4, 5)
Partial{Partial{Float64,Float64,Float64},Int64,Int64}(Partial{Float64,Float64,Float64}(1.0, 2.0, 3.0), 4, 5)

julia> nestedpartial = Partial(Partial(1.0, 2.0, 3.0), 4, 5)
Partial{Partial{Float64,Float64,Float64},Int64,Int64}(Partial{Float64,Float64,Float64}(1.0, 2.0, 3.0), 4, 5)

julia> flatten(nestedpartial)
(1.0, 2.0, 4)

metaflatten(obj, func, [flattentrait::Function], [use::Type], [ignore::Type])

Metadata flattening. Flattens data attached to a field using a passed in function Query types and flatten trait arguments are optional, but you must pass use to pass ignore.

Arguments

• obj: The target type to be reconstructed
• func: A function with the form: f(::Type, ::Type{Val{:fieldname}}) = metadata
• flattentrait: A function returning a Bool, such as a FielMetadata method. With the form: f(::Type, ::Type{Val{:fieldname}}) = true
• use: Type or Union of types to return in the tuple.
• ignore: Types or Union of types to ignore completly. These are not reurned or recursed over.

We can flatten this @foobar metadata:

julia> using Flatten, FieldMetadata

julia> import Flatten: flattenable

julia> @metadata foobar :foo;

julia> @foobar struct Foo{A,B,C}
           a::A | :bar
           b::B | :foobar
           c::C | :foo
       end;

julia> @foobar struct Bar{X,Y}
           x::X | :foobar
           y::Y | :bar
       end;

julia> metaflatten(Foo(1, 2, Bar(3, 4)), foobar)
(:bar, :foobar, :foobar, :bar)

modify(func, obj, args...)

Modify field in a type with a function

parentnameflatten(obj, args...)

Flatten the name of the parent type of an object. Args are passed to metaflatten.

Examples

julia> using Flatten

julia> struct Foo{A,B,C}
           a::A
           b::B
           c::C
       end

julia> struct Bar{X,Y}
           x::X
           y::Y
       end

julia> parentnameflatten(Foo(1, 2, Bar(3, 4)))
(:Foo, :Foo, :Bar, :Bar)

parenttypeflatten(obj, args...)

Flatten the parent type of an object. Args are passed to metaflatten.

Examples

julia> using Flatten

julia> struct Foo{A,B,C}
           a::A
           b::B
           c::C
       end

julia> struct Bar{X,Y}
           x::X
           y::Y
       end

julia> parenttypeflatten(Foo(1, 2, Bar(3, 4)))
(Foo{Int64,Int64,Bar{Int64,Int64}}, Foo{Int64,Int64,Bar{Int64,Int64}}, Bar{Int64,Int64}, Bar{Int64,Int64})

reconstruct(obj, data, [flattentrait::Function], [use::Type], [ignore::Type])

Reconstruct an object from Tuple or Vector data and an existing object. Data should be at least as long as the queried fields in the obj. Query types and flatten trait arguments are optional, but you must pass use to pass ignore.

Arguments

• obj: The target type to be reconstructed
• data: Replacement data - an AbstractArray, Tuple or type that defines getfield.
• flattentrait: A function returning a Bool, such as a FielMetadata method. With the form: f(::Type, ::Type{Val{:fieldname}}) = true
• use: Type or Union of types to return in the tuple.
• ignore: Types or Union of types to ignore completly. These are not reurned or recursed over.

Examples

julia> struct Foo{A,B,C}
           a::A
           b::B
           c::C
       end

julia> reconstruct(Foo(1, 2, 3), (1, :two, 3.0))
Foo{Int64,Symbol,Float64}(1, :two, 3.0)

update!(obj, data, [flattentrait::Function], [use::Type], [ignore::Type])

Update a mutable object with a Tuple or Vector of data. Query types and flatten trait arguments are optional, but you must pass use to pass ignore.

Arguments

• obj: The target type to be reconstructed
• data: Replacement data - an AbstractArray, Tuple or type that defines getfield.
• flattentrait: A function returning a Bool, such as a FielMetadat method. With the form: f(::Type, ::Type{Val{:fieldname}}) = true
• use: Types to return in the tuple.
• ignore: Types to ignore completly. These are not reurned or recursed over.

Examples

julia> using Flatten

julia> mutable struct MutableFoo{A,B,C}
           a::A
           b::B
           c::C
       end

julia> mufoo = MutableFoo(1, 2, 3)
MutableFoo{Int64,Int64,Int64}(1, 2, 3)

julia> update!(mufoo, (2, 4, 6))
MutableFoo{Int64,Int64,Int64}(2, 4, 6)

julia> mufoo = MutableFoo(1, 2, :three)
MutableFoo{Int64,Int64,Symbol}(1, 2, :three)

julia> update!(mufoo, (:foo,), Symbol)
MutableFoo{Int64,Int64,Symbol}(1, 2, :foo)