Introduction

ThorsSerializer A modern declarative C++ serialization library.

API

JSON

NameSpace:

ThorsAnvil::Serialize

Headers:

ThorSerialize/JsonThor.h

Exporter<T, Json> jsonExporter ( T const& value )
Importer<T, Json> jsonImporter ( T& value )

YAML

NameSpace:

ThorsAnvil::Serialize

Headers:

ThorSerialize/YamlThor.h

Exporter<T, Yaml> yamlExporter ( T const& value )
Importer<T, Yaml> yamlImporter ( T& value )

Binary

NameSpace:

ThorsAnvil::Serialize

Headers:

ThorSerialize/BinaryThor.h

Exporter<T, Binary> binaryExporter ( T const& value )
Importer<T, Binary> binaryImporter ( T& value )

Macros

NameSpace:

Headers:

ThorSerialize/Traits.h

ThorsAnvil_MakeEnum ( EnumType,EnumValues... )
ThorsAnvil_MakeTraitCustom ( Type )
ThorsAnvil_PointerAllocator ( Type,Action )
ThorsAnvil_MakeTrait ( Type,fields... )
ThorsAnvil_ExpandTrait ( ParentType,Type,fields... )
ThorsAnvil_Template_MakeTrait ( TemplateParameterCount,Type,fields... )
ThorsAnvil_Template_ExpandTrait ( TemplateParameterCount,ParentType,Type,fields... )
ThorsAnvil_PolyMorphicSerializer ( Type )
ThorsAnvil_RegisterPolyMorphicType ( Type )

Exporter

NameSpace:

ThorsAnvil::Serialize

Headers:

ThorSerialize/Exporter.h

std::ostream& operator<< ( std::ostream& stream,Exporter const& data )

Importer

NameSpace:

ThorsAnvil::Serialize

Headers:

ThorSerialize/Importer.h

std::istream& operator>> ( std::istream& stream,Importer const& data )

Usage

Serializing

There are two serialization formats supported out o the box (JSON/YAML) and an experimental binary format (I would love if somebody added the google Protocol Buffers).

#include "ThorSerialize/JsonThor.h"

using ThorsAnvil::Serialize::jsonExporter;
using ThorsAnvil::Serialize::jsonImporter;

int main()
{
    std::cout << jsonExporter(12) << "\n";

    int value;
    std::cin >> jsonImporter(value);
}

Each format has two commands <format>Importer(<object>) and <format>Exporter(<object>). e.g. jsonExporter(12).

These functions return a very lightweight object (it simply contains a reference to the object being serialized) that can be passed to the standard stream operators. Thats at it for a the user of serialization library.

To include all the functionality all you need to do is include #include "ThorSerialize/<format>Thor.h" and link against libThorSerialize.so.

Format

    std::cout << jsonExporter(mark) << "\n";
...
    {
        "name": "mark",
        "score": 10,
        "damage": 5,
        "team":
        {
            "red": 66,
            "green": 42,
            "blue": 23
        }
    }
...
    using OutputType = ThorsAnvil::Serialize::PrinterInterface::OutputType;
    std::cout << jsonExporter(mark, OutputType::Stream) << "\n";
...
{"name":"mark","score":10,"damage":5,"team":{"red":66,"green":42,"blue":23}}

By default the generated JSON is verbose and easy for humans to read. This also makes it longer than required. You can compact the output by OutputType::Stream in the jsonFormat().

Strictness

    std::cin << jsonImporter(dst) << "\n";
...
    using ParseType = ThorsAnvil::Serialize::ParserInterface::ParseType;
    std::cin >> jsonImporter(dst, ParseType::Strict);

By default the parser is forgiving; extra or missing fields are simply ignored. If you want to use Strict parsing then you specify this as part of the jsonImporter(). In this mode all fields are required no additional fields are allowed. If either of these constraints are broken then an exception is thrown.

Exceptions

    using OutputType = ThorsAnvil::Serialize::PrinterInterface::OutputType;
    while(std::cout << jsonExporter(12, OutputType::Default, true) << "\n")
    {
        // Successfully wrote an object to the output
    }
    ...
    using ParseType = ThorsAnvil::Serialize::ParserInterface::ParseType;
    while(std::cin >> jsonImporter(dst, ParseType::Weak, true))
    {
        // Successfully read an object from the input
    }

By default the ThorsSerializer will throw an exception when it encounters a parsing error (it also sets the stream state to fail). If you would prefer for the stream to swallow the exception (but still set the stream state to fail) then you can modify this behavior in jsonImporter() and jsonExporter().

Declarations

Standard Types

#include "ThorSerialize/JsonThor.h"

using ThorsAnvil::Serialize::jsonExporter;
using ThorsAnvil::Serialize::jsonImporter;

int main()
{
    int         v1 = 12;        // All the different types of int are supported)
    double      v2 = 13.5;      // float/double/long double all supported
    bool        v3 = true;
    std::string v4 = "A string";

    std::cout << jsonExporter(v1) << " "
              << jsonExporter(v2) << " "
              << jsonExporter(v3) << " "
              << jsonExporter(v4) << "\n";
}
...
 12  13.5  true  "A string"

The built-in types integer/float/bool/std::string are serialalable out of the box. You may notice a couple of notable exceptions char and char*. The char type is not supported as it is very easily confused with an integer and char* is not supported because I did not want to encourage C-Strings when std::string is available (sorry).

Note: I know this is not very useful by itself. Bare with me it becomes useful when you start composing types.

Standard Containers

#include "ThorSerialize/JsonThor.h"
#include "ThorSerialize/SerUtil.h"

using ThorsAnvil::Serialize::jsonExporter;
using ThorsAnvil::Serialize::jsonImporter;

int main()
{
    std::vector<int>    data = {1,2,3,4,5,6,7};
    std::cout << jsonExporter(data) << "\n";
}
...
 [ 1, 2, 3, 4, 5, 6, 7]

The standard library container types are all supported. You simply need to include #include "ThorSerialize/SerUtil.h".

std::array<T,N>
std::list<T>
std::vector<T, Allocator>
std::deque<T, Allocator>
std::pair<A,B>
std::set<K>
std::multiset<K>
std::map<K,V>
std::multimap<K,V>
std::tuple<Args...>
std::unordered_set<K,V>
std::unordered_multiset<K>
std::unordered_map<K,V>
std::unordered_multimap<K,V>
std::initializer_list<T>
std::unique_ptr<T>

Enum

#include "ThorSerialize/JsonThor.h"
#include "ThorSerialize/Traits.h"

using ThorsAnvil::Serialize::jsonExporter;
using ThorsAnvil::Serialize::jsonImporter;

enum Color { red, green, blue };
ThorsAnvil_MakeEnum(Color, red, green, blue);

int main()
{
    Color c = red;
    std::cout << jsonExporter(c) << "\n";
}
...
 "red"

In C++ enums are serialized as integer types. This can work but you loose meaning in this translation. It also binds you contractually to never changing the order of any enum items in the type.

The ThorsSerializer library provides you a mechanism to stream the type as a string. This maintains its symantic meaning while in the JSON format and when de-serialized is converted back to the correct enum value automatically.

For each enum type that you want to serialize simply use ThorsAnvil_MakeEnum() macros to declare the enum and all valid streamable values in the enum range. You simply need to include #include "ThorSerialize/Traits.h".

Structure

// Shirt.h
#include "ThorSerialize/Traits.h"
struct Shirt
{
    int     red;
    int     green;
    int     blue;
};
ThorsAnvil_MakeTrait(Shirt, red, green, blue);

#include "ThorSerialize/JsonThor.h"
#include "Shirt.h"

using ThorsAnvil::Serialize::jsonExporter;
using ThorsAnvil::Serialize::jsonImporter;

int main()
{
    Shirt          shirt{10, 20, 45};
    std::cout << jsonExporter(shirt) << "\n";
}
...
    {
        "red": 10,
        "green": 20,
        "blue": 45
    }

In C++ structures are not serializable by default (but often have the appropriate stream operators defined). The advantage of ThorsSerializer library is that it adds a serialization feature to a class without altering the class in any way. This is achieved by building a Serialization::Traits<> type for your class. We will go into more detail on how this works in the implementation section.

The easiest way to build the appropriate Serialization::Traits<> type is to use the macro ThorsAnvil_MakeTrait() naming the type and all members you want to serialize as part of the object. You simply need to include #include "ThorSerialize/Traits.h".

Private Members

// TeamMember.h
#include "Shirt.h"
class TeamMember
{
    std::string     name;
    int             score;
    int             damage;
    Shirt           team;
    public:
    TeamMember(std::string const& name, int score, int damage, Shirt const& team)
        : name(name)
          , score(score)
          , damage(damage)
          , team(team)
    {}
    // Define the trait as a friend to get accesses to private Members.
    friend class ThorsAnvil::Serialize::Traits<TeamMember>;
};
ThorsAnvil_MakeTrait(TeamMember, name, score, damage, team);

#include "ThorSerialize/JsonThor.h"
#include "TeamMember.h"

using ThorsAnvil::Serialize::jsonExporter;
using ThorsAnvil::Serialize::jsonImporter;

int main()
{
    TeamMember          mark("mark", 10, 5, Shirt{66, 42, 23});
    std::cout << jsonExporter(mark) << "\n";
}
...
    {
        "name": "mark",
        "score": 10,
        "damage": 5,
        "team":
        {
            "red": 66,
            "green": 42,
            "blue": 23
        }
    }

If you have a class/struct were the members that need to be serialized are private then there is no automatic accesses for the ThorsSerializer library to these members. To give the library accesses your class must declare the Serialization::Traits<> type as a friend of the class. This will allow the class to directly accesses these members and both read/write them during the serialization/de-serialization processes.

Inheritance

// ExtendedTeamMember.h
#include "TeamMember.h"
class ExtendedTeamMember: public TeamMember
{
    bool    extension;
    public:
        ExtendedTeamMember(std::string const& name, int score, int damage, Shirt const& team, bool ex)
            : TeamMember(name, score, damage, team)
            , extension(ex)
        {}
    friend class ThorsAnvil::Serialize::Traits<ExtendedTeamMember>;
};
ThorsAnvil_ExpandTrait(TeamMember, ExtendedTeamMember, extension);

#include "ThorSerialize/JsonThor.h"
#include "ExtendedTeamMember.h"

using ThorsAnvil::Serialize::jsonExporter;
using ThorsAnvil::Serialize::jsonImporter;

int main()
{
    ExtendedTeamMember          rangers("Rangers", 10, 5, Shirt{66, 42, 23}, true);
    std::cout << jsonExporter(rangers) << "\n";
}
...
    {
        "name": "Rangers",
        "score": 10,
        "damage": 5,
        "team":
        {
            "red": 66,
            "green": 42,
            "blue": 23
        },
        "extension": true
    }

The only difference from a base type is that we use ThorsAnvil_ExpandTrait() rather than ThorsAnvil_MakeTrait(). The difference between these two is that ThorsAnvil_ExpandTrait() takes the parent class as the first parameter followed by the child type as the second parameter.

Multiple Inheritance

#include "ThorSerialize/JsonThor.h"
#include "ThorSerialize/Traits.h"

using ThorsAnvil::Serialize::jsonExporter;
using ThorsAnvil::Serialize::jsonImporter;

struct Base1
{
    int         data;
};
struct Base2
{
    std::string name;
};
struct Object: public Base1, public Base2
{
    bool        good;
};
ThorsAnvil_MakeTrait(Base1, data);
ThorsAnvil_MakeTrait(Base2, name);

// The macro ThorsAnvil_ExpandTrait() only allows you to declare 1 base type name.
// To get around this limitation you need place both base types into `Serialize::Parents<>` template.
// This can then be used as the parent type in the macro ThorsAnvil_ExpandTrait()␣
using Base1Base2Parent = ThorsAnvil::Serialize::Parents<Base1, Base2>;
ThorsAnvil_ExpandTrait(Base1Base2Parent, Object, good);

int main()
{
    Object  object{12, "Test", true};
    std::cout << jsonExporter(object) << "\n";
}
...
    {
        "data": 12,
        "name": "Test",
        "good": true
    }

Multiple enheritance is a tiny bit of a hack.
Multiple parent types must be combined using the template class ThorsAnvil::Serialize::Parents and then passed to the ThorsAnvil_ExpandTrait() macro as if that was the parent type.

Pointers

#include "ThorSerialize/JsonThor.h"
#include "ThorSerialize/SerUtil.h"
#include "TeamMember.h"


using ThorsAnvil::Serialize::jsonExporter;
using ThorsAnvil::Serialize::jsonImporter;

int main()
{
    std::unique_ptr<TeamMember>     data;
    std::cout << jsonExporter(data) << "\n";
    std::cout << jsonExporter(data.get()) << "\n";

    data.reset(new TeamMember{"Tim", 33, 2, Shirt{12, 13, 14}});
    std::cout << jsonExporter(data) << "\n";
    std::cout << jsonExporter(data.get()) << "\n";
}
...
 null
 null

    {
        "name": "Tim",
        "score": 33,
        "damage": 2,
        "team":
        {
            "red": 12,
            "green": 13,
            "blue": 14
        }
    }

    {
        "name": "Tim",
        "score": 33,
        "damage": 2,
        "team":
        {
            "red": 12,
            "green": 13,
            "blue": 14
        }
    }

No extra work needs to be done to serialize pointers. If the pointer is nullptr it will serialize to the null type of the serialization format. If the pointer is not null then it will serialize as an object.

std::unique_ptr

The std::unique_ptr<> behaves just like a pointer during serialization.

std::shared_ptr

The std::shared_ptr is not currently supported.

Polymorphic Objects

#include "ThorSerialize/JsonThor.h"
#include "TeamMember.h"

using ThorsAnvil::Serialize::jsonExporter;
using ThorsAnvil::Serialize::jsonImporter;

struct Vehicle
{
    Vehicle(){}
    Vehicle(int speed)
        : speed(speed)
    {}
    virtual ~Vehicle() {}
    int     speed;
    ThorsAnvil_PolyMorphicSerializer(Vehicle);
};
struct Car: public Vehicle
{
    Car(){}
    Car(int speed, std::string const& make)
        : Vehicle(speed)
          , make(make)
    {}
    std::string     make;
    ThorsAnvil_PolyMorphicSerializer(Car);
};
struct Bike: public Vehicle
{
    Bike(){}
    Bike(int speed, int stroke)
        : Vehicle(speed)
          , stroke(stroke)
    {}
    int     stroke;
    ThorsAnvil_PolyMorphicSerializer(Bike);
};
ThorsAnvil_MakeTrait(Vehicle, speed);
ThorsAnvil_ExpandTrait(Vehicle, Car, make);
ThorsAnvil_ExpandTrait(Vehicle, Bike, stroke);

int main()
{
    Vehicle* v = new Bike(12,3);
    std::cout << jsonExporter(v) << "\n";
}
...
    {
        "__type": "Bike",
        "speed": 12,
        "stroke": 3
    }

Polymorphic objects are supported. BUT require an intrusive change in the type. To mark objects as polymorphic you need to add the macro ThorsAnvil_PolyMorphicSerializer(). This macro adds a couple of virtual methods to your class (but no data members). Additonally the resulting JSON has an extra field __ type that contains the name of the type. This allows the serialization library to dynamically create the correct type at runtime.

Value

#include "ThorSerialize/JsonThor.h"
#include "ThorSerialize/Traits.h"

using ThorsAnvil::Serialize::jsonExporter;
using ThorsAnvil::Serialize::jsonImporter;

struct ID
{
    int id;
    friend std::ostream& operator<<(std::ostream& s, ID const& data) {return s << data.id;}
    friend std::istream& operator>>(std::istream& s, ID& data)       {return s >> data.id;}
};
ThorsAnvil_MakeTraitCustom(ID);

int main()
{
    ID  id{23};
    std::cout << jsonExporter(id) << "\n";
}
...
 23

In situations where your class already has appropriate input and output operators (that generate JSON like values (Bool,Integer/Float/String) then you can simply declare your type as serializeable using the macro ThorsAnvil_MakeTraitCustom().

Implementation

Traits

namespace ThorsAnvil
{
    namespace Serialize
    {

template<typename T>
class Traits
{
    public:
        static constexpr TraitType type = TraitType::Invalid;
};

    }
}

The serialization processes is built around around a traits class Traits<T> that is specialized for each type. This is similar in technique to the std::iterator_traits<> used by the standard library.

The generic (and thus default) Traits<T> has a single member type that has the value TraitType::Invalid.

The type member of the Traits<> specialization indicates how the member will be serialized/de-serialized and defines what other members of the Traits<> class are needed for that speialization.

The following values are allowed: {Invalid, Parent, Value, Map, Array, Enum, Pointer, Serialize}

Traits Value

template<>
class Traits<FundamentType>
{
    public:
        static constexpr TraitType type = TraitType::Value;
};

This is used for all the fundamental types. These types use the ParserInterface and PrinterInterface to serialize/de-serialize. You should only use this value if your value can be parsed via: ParserInterface::getValue() and printed via PrinterInterface::addValue().

Traits Enum

template<>
class Traits<Enum_Type>
{
    public:
        static constexpr TraitType type = TraitType::Enum;

        static char const* const*   getValues();
        static Enum_Type            getValue(std::string const& val, std::string const& msg);
};

In this case ThorsSerializer expects the Traits<T> class to have two extra static methods: getValues() and getValue().

getValues()
Is used for serializing the value by providing an array of char const* that represent the text version of the enum value.
getValue()
Is used for deserializing a specific string into a specifc enum value.

    ThorsAnvil_MakeEnum(<EnumType>, <EnumValues>...)

The easy way to generate the Traits<> specialization for an enum with these fields is via the macro ThorsAnvil_MakeEnum().

Traits Serialize

template<>
class Traits<StremableType>
{
    public:
        static constexpr TraitType type = TraitType::Serialize;
};

In this case ThorsSerializer expects the object to serialize itself via operator<< and operator>>. The resulting value should be a simple JSON value (Bool/Integer/Float/String).

    ThorsAnvil_MakeTraitCustom(<SerializableType>);

The easy way to generate the Traits<> specialization for a Serialize type is via the macro ThorsAnvil_MakeTraitCustom().

Traits Pointer

template<typename T>
class Traits<T*>
{
    public:
        static constexpr TraitType type = TraitType::Pointer;
        static T*   alloc();
        static void release(T* p);
};

In this case ThorsSerializer expects the Traits<T> class to have two extra static methods: alloc() and release().

alloc()
Is used to allocate an object of type T
release()
Is used to release an onbject of type T that was allocated via the alloc() function.

If you simply want to use new and delete then the default partial specialization of thie object works out of the box with no extra configuration needed.

ThorsAnvil_PointerAllocator

strict MyDataMemoryPool
{
    MyData* alloc();
    void release(MyData*);
};
ThorsAnvil_PointerAllocator(MyData, MyDataMemoryPool);

The easy way to generate the Traits<> specialization for a pointer that has its own allocators is via the macro ThorsAnvil_PointerAllocator().

Traits Map

template<>
class Traits<MapType>
{
    public:
        static constexpr TraitType type = TraitType::Map;   // or TraitType::Parent

        static ExtractorType const& getMembers();
};
//  ExtractorType   => std::tuple<Item...>
//                  or MemberExtractor
//
//  Item    =>  std::pair<char const*, M*>              // M*            pointer to a static member of the class.
//          =>  std::pair<char const*, M MapType::*>    // M MapType::*  pointer to a member of the class
//              struct MemberExtractor
//              {
//                  void operator()(PrinterInterface& printer, ArrayType const& object) const;
//                  void operator()(ParserInterface& parser,  std::string const& key, ArrayType& object) const;
//              };

If the Traits<>::type is TraitType::Map or TraitType::Parent then the Traits<> specialization is expected to have a static getMembers() method that returns an Extractor object (see below for details).

When serializing an object where the Traits<>::type is TraitType::Map or TraitType::Parent the library will call openMap() on the printerInterface (generates a '{' in JSON) and conversely will call closeMap() (generates a '}' in JSON) after all the members have been serialized. Conversily when de-serializing into an object with these Traits<>::type the parser will expect an mapOpen/mapClose marker (in JSON '{' amd '}' respecively).

Tuple of pair

    ThorsAnvil_MakeTrait(DataType, ...);
    ThorsAnvil_ExpandTrait(ParentType, DataType, ...);

If the getMembers() method returns a std::tuple with a set of std::pair<>. Then each pair represents a member of the map that needs to be serialized. The pair<>::first represents the key and is serialized as string and when de-serializing we search the std::tuple to see if we can match the key that was read from the input. The pair<>::second is a pointer to the object that represents the value. A recursive call to de-serialize the value is made.

The easy way to generate the Traits<> specialization for an Map with fields is via the macro ThorsAnvil_MakeTrait() or ThorsAnvil_ExpandTrait().

MemberExtractor

If the getMembers() method returns any other type it is expected to have an interface that matchs the MemberExtractor definition. The method using PrinterInterface is called once and is expected to serialize the object using this interface. The method using the ParserInterface is called after each key has been extracted and is expected to de-serialize the value directly into the destination object.

Traits Array

template<>
class Traits<ArrayType>
{
    public:
        static constexpr TraitType type = TraitType::Array;

        struct MemberExtractor
        {
            void operator()(PrinterInterface& printer, ArrayType const& object) const;
            void operator()(ParserInterface& parser, std::size_t index, ArrayType& object) const;
        };
        static ExtractorType const& getMembers()
};

If the Traits<>::type is TraitType::Array then the Traits<> specialization is expected to have a static getMembers() method that returns an MemberExtractor object (see below for details).

When serializing an object where the Traits<>::type is TraitType::Array the library will call openArray() on the printerInterface (generates a '[' in JSON) and conversely will call closeArray() (generates a ']' in JSON) after all the members have been serialized. Conversily when de-serializing into an object with these Traits<>::type the parser will expect an arrayOpen/arrayClose marker (in JSON '[' amd ']' respecively).

The method using PrinterInterface is called once and is expected to serialize the object using this interface. The method using the ParserInterface is called for each new value that is available on the stream and is expected to de-serialize the value directly into the destination object. Note: each time it is called the index is passed.

APi Generated

NameSpace:

ThorsAnvil::Serialize

Headers:

ThorSerialize

BinaryThor.h

Exporter<Binary binExporter ()
Importer<Binary binImporter ()

JsonThor.h

Exporter<Json, T> jsonExporter ()
Importer<Json, T> jsonImporter ()

YamlThor.h

Exporter<Yaml, T> yamlExporter ()
Importer<Yaml, T> yamlImporter ()

binExporter

Header:

ThorSerialize/BinaryThor.h

NameSpace:

ThorsAnvil::Serialize

Template:

template<typename T>

Return Value: Exporter, T>

Object that can be passed to operator<< for serialization.

Parameters

T const& value

The object to be serialized.

bool catchExceptions

binImporter

Header:

ThorSerialize/BinaryThor.h

NameSpace:

ThorsAnvil::Serialize

Template:

template<typename T>

Return Value: Importer, T>

Object that can be passed to operator>> for de-serialization.

Parameters

T& value

The object to be de-serialized.

bool catchExceptions

jsonExporter

Header:

ThorSerialize/JsonThor.h

NameSpace:

ThorsAnvil::Serialize

Template:

template<typename T>

Return Value: Exporter

Object that can be passed to operator<< for serialization.

Parameters

T const& value

The object to be serialized.

PrinterInterface::OutputType characteristics

Default: is verbose and logical. Stream: remove all white space.

bool catchExceptions

jsonImporter

Header:

ThorSerialize/JsonThor.h

NameSpace:

ThorsAnvil::Serialize

Template:

template<typename T>

Return Value: Importer

Object that can be passed to operator>> for de-serialization.

Parameters

T& value

The object to be de-serialized.

ParserInterface::ParseType parseStrictness

Weak: ignore missing extra fields. Strict: Any missing or extra fields throws exception.

bool catchExceptions

yamlExporter

Header:

ThorSerialize/YamlThor.h

NameSpace:

ThorsAnvil::Serialize

Template:

template<typename T>

Return Value: Exporter

Object that can be passed to operator<< for serialization.

Parameters

T const& value

The object to be serialized.

PrinterInterface::OutputType characteristics

Default: is verbose and logical. Stream: remove all white space.

bool catchExceptions

yamlImporter

Header:

ThorSerialize/YamlThor.h

NameSpace:

ThorsAnvil::Serialize

Template:

template<typename T>

Return Value: Importer

Object that can be passed to operator>> for de-serialization.

Parameters

T& value

The object to be de-serialized.

ParserInterface::ParseType parseStrictness

Weak: ignore missing extra fields. Strict: Any missing or extra fields throws exception.

bool catchExceptions