Overview

Metaplay ships with a powerful and easy-to-use built-in serializer. The serializer was implemented with the following design goals in mind:

• Minimize boilerplate: Serialization is controlled by specifying attributes to types that need to be serialized. The game logic types themselves are used for serialization, so no additional copies of the types are needed for serialization. All serialization code is programmatically generated to avoid tedious and error-prone manual implementation of serialization logic.
• Easy to use: Game logic types only need to be tagged with the proper attributes and the rest is handled by Metaplay. No external tools or additional message specifications are required.
• Multi-purpose: The same format is used to communicate between client and server, as well as to persist data into the database. Attributes can be used to control which data is persisted in the database and which data is shared by the client and the server.
• Compact format: The binary format used is inspired by Google ProtoBufs. It is roughly 3-5x more compact than JSON.
• Efficient: All serialization code is generated and compiled using the Roslyn compiler, which results in very efficient implementation. For Unity IL2CPP targets, the serializer code is generated as a build step and included in the final build as a .dll file.
• Backward-compatible: All types and members in the serialized data is tagged with metadata so that members can be added or removed. Supporting more complex changes does require hand-written migration code.
• Stand-alone decoding: Serialized data contains enough metadata for it to be be traversed and printed for debugging purposes.

Supported Types

The following types are supported by the serializer:

• Primitive types: integer types, chars, strings, enums, bools, floats, and doubles.
• Fixed-point scalars and vectors: F32, F64, F32Vec2, F32Vec3, F64Vec2, and F64Vec3.
• Nullable variants of primitive value types, as well as the fixed-point types listed above. For example: int?, F64?.
• Type-safe string identifiers: StringId<T>s.
• Classes and structs, including nested objects (i.e., objects contained within object objects) and hierarchies of classes.
• Tuples and ValueTuples: analogously to Classes and Structs, the order of the elements corresponds with the TagId order, hence you should not change the order at a later time. Since Tuples and ValueTuples have the same serialized format as Classes and Structs, you can change the member to be a class or a struct as long as you maintain the TagId order.
• Nullable structs T? where T is a struct type supported by the serializer.
• Interfaces, analogously to abstract classes.
• Arrays and nested arrays, e.g.: int[] or List<bool>[][] (but not multi-dimensional arrays int[,]).
• Value and key-value collections, e.g.: List<T>, Dictionary<Key, Value>, OrderedSet<T>, OrderedDictionary<Key, Value>, ReadOnlyCollection<T>, ReadOnlyDictionary<Key, Value>.
• System Types: Guid, Version, DateTimeOffset, and TimeSpan.
• Special MetaSerialized<T> type which allows including nested serialized data. This is useful when, for example, the PlayerModel class is included in a MetaMessage as a member. When using MetaSerialized<PlayerModel>, the contained PlayerModel does not need to be deserialized
• Game config data references: references to IGameConfigData<TKey>-implementing classes are serialized as just the config key instead of the object's contents. Upon deserialization, the key is used to resolve the concrete reference into a config library. See Game Config References for a few details.

Limitations:

• While nested objects are supported, only a single reference to each object may be used. If multiple references to a given object exist, a separate copy is generated as a result of a serialize-deserialize cycle.
• Type conversions are not supported. If a member is serialized with a given type, it must also be serialized back to the same type. The same applies when persisting data into the database: when reading back the data at a later time, all the member types must still match.

Classes and Structs

Example of declaring a serializable class with some members with different behaviors:

[MetaSerializable]
public class ExampleClass
{
  // Persisted member, stored in database and sent to the client
  [MetaMember(4)] public int Experience { get; private set; }

  // Transient member, shared with the client, but not persisted in database
  [MetaMember(5), Transient] public int SessionTick { get; private set; }

  // Persisted in database, but excluded from any Desync checksum checks
  [MetaMember(6), NoChecksum] public int PendingGoldGift { get; private set; }

  // Persisted in database, not sent to the client
  [MetaMember(7), ServerOnly] public List<int> SecretList { get; private set; }

  // An empty constructor or a deserialization constructor (using `MetaSerializableFlags.AutomaticConstructorDetection` or the `MetaDeserializationConstructor` attribute) is required by the serializer.
  // If no constructors are explicitly defined, the compiler implicitly
  // creates an empty constructor which is enough for Metaplay.
  // The constructor can be protected or private.
  public ExampleClass() {}
}

[MetaSerializable] Attribute

The class itself is tagged with the [MetaSerializable] attribute, which informs the serializer that serialization code should be generated for the type.

The following MetaSerializableFlags can be given to [MetaSerializable] attribute:

• MetaSerializableFlags.ImplicitMembers will implicitly tag all of the type's members with [MetaMember(tagId)] using linearly incrementing tagIds. This flag is used by default for MetaMessages and ModelActions, to avoid boilerplate.
  • When using ImplicitMembers, any non-concrete class in the class hierarchy that declares any MetaMembers must have the [MetaImplicitMembersRange] attribute to specify the range of tagIds to use.
  • Concrete classes don't need the [MetaImplicitMembersRange] attribute, but instead the base class can specify the [MetaImplicitMembersDefaultRangeForMostDerivedClass] attribute, which will specify the range that gets by default used for concrete classes.
  • The purpose of these attributes is to allocate some tagId space in base classes in order to allow adding new members to them without changing the tagIds in derived classes.
• MetaSerializableFlags.AutomaticConstructorDetection will automatically try to detect the best matching constructor (based on type and member name), the serializer will call this constructor when deserializing into an object. Alternatively the [MetaDeserializationConstructor] attribute can be used to specify the constructor which will be used for deserialization.

[MetaMember] Attribute

The [MetaMember(tagId)] attribute is used to declare members of the class to be included in the serialization. The tagId is used to identify the given member in the serialized format, and during deserialization, the tagId is used to match the data from the serialized format into the class members. The tagId used must be a positive non-zero integer.

It is safe to introduce new members with previously unused tagId. During deserialization, if no data is found matching the given tagId, the member is left to its default value (as initialized by the empty-argument constructor). Similarly, members can also be safely removed from the class: the deserializer will ignore any data that it cannot find a member with a matching tagId.

INFO

Note: The tagIds should never be re-used within the same class as it can create a situation where old data can be accidentally deserialized to overwrite a member with the conflicting tagId.

The following attributes can be combined with the [MetaMember] attribute:

• [Transient] causes the member to be excluded when persisting the given class into the database. It will still be sent to the client and included in any checksum checks (for detecting Desyncs). It is most useful when declaring session-specific variables.
• [NoChecksum] causes the member to be excluded from any checksum checks for detecting Desyncs. Its most common use case is for members that are modified via ServerActions, which are not synchronized to execute on the same tick at both ends. Most typical use cases would be when the player receives gifts, mail, gacha results, or similar events from the server.
• [ServerOnly] causes the member to be excluded when serializing the class for sending to the client. It is convenient for keeping hidden data in a Model state, which the Players should never see or be able to access. The member is also excluded from checksum checks.
• [MaxCollectionSize(32768)] causes the maximum collection size limit to be overridden. The default limit is 16384. You should use this attribute to set reasonable limits for collections coming from the client (which cannot be trusted) to minimize the impact of resource starvation attacks. For trusted data (e.g. read-only), you can use this attribute to allow larger-than-default collection sizes.

INFO

Note: All serializable members must be mutable. I.e., readonly members are not supported.

[MetaDeserializationConstructor] Attribute

Constructors can be tagged with the [MetaDeserializationConstructor] attribute to instruct the serializer to invoke this constructor instead of deserializing by members. This enables the usage of read-only/init-only properties.

The serializer can only use a constructor to deserialize if it has a parameter for each MetaMember with a matching name and type, the order however does not have to be the same.

For example, in the Idler sample, we are using constructor based deserialization to make sure that the Info property is read only.

[MetaSerializable]
public class ProducerModel
{
    [MetaMember(1)] public ProducerInfo     Info        { get; }
    [MetaMember(2)] public int              Level       { get; set; } = 1;
    [MetaMember(3)] public MetaTime         StartedAt   { get; set; }

    [MetaDeserializationConstructor]
    public ProducerModel(ProducerInfo info, int level, MetaTime startedAt)
    {
        Info = info;
        Level = level;
        StartedAt = startedAt;
    }
}

Serialization of Actions

For Actions, no explicit attributes are needed. The [ModelAction] attribute implicitly adds the [MetaSerializable] attribute to the class itself, as well as [MetaMember] attribute to all the fields in the Action class.

Example:

[ModelAction(12345)]
public class MyAction : PlayerAction
{
    // Note: [MetaMember] attributes with linearly incrementing tagIds
    // are added implicitly to Action members (shown in comments).
    /*[MetaMember(1)]*/ public string Name { get; private set; }
    /*[MetaMember(2)]*/ public int    Cost { get; private set; }

    public MyAction() {}
    public MyAction(string name, int cost) { Name = name; Cost = cost; }

    public override ActionResult Execute(PlayerModel player, bool commit)
    {
        // ...
    }
}

Class Hierarchy Serialization

Metaplay supports the serializing of class hierarchies with some limitations:

// The base class is abstract, which tells Metaplay to serialize any concrete derived
// classes of it based on their declared TypeCodes.
[MetaSerializable]
public abstract class BuildingBase
{
    // All tagIds must be unique in the whole chain of derived classes.
    // This is by design, as it makes it easy to move members between the base
    // and derived classes. It is a convenient pattern to start the tagIds from
    // some high number (eg, 100) in the base class to avoid accidental conflicts
    // with members of the derived classes.
    [MetaMember(100)] public int BuildingId { get; set; }

    public BuildingBase() {}
}

// Using typeCode==1 for the first building example. The typeCodes must be unique
// for all classes deriving from any given base class, but need not be unique
// globally.
[MetaSerializableDerived(1)]
public class Granary : BuildingBase
{
    [MetaMember(1)] public int FoodAmount  { get; set; }
    [MetaMember(2)] public int MaxCapacity { get; set; }
}

// Another derived class with typeCode==2.
[MetaSerializableDerived(2)]
public class Library : BuildingBase
{
    [MetaMember(1)] public int NumBooks { get; set; }
}

// Example of usage.
[MetaSerializable]
public class BuildingInventory
{
    // References to the base class are supported by the serializer. It will
    // use the typeCode of the concrete class to identify the types of Building
    // in the list.
    [MetaMember(10)] public List<BuildingBase> Buildings;
}

This is achieved by adding the [MetaSerializable] attribute to the abstract base class and [MetaSerializableDerived(typeCode)] attribute to all the concrete classes deriving from the abstract base class. The typeCode is stored in the serialized format to identify which concrete class instance is stored in the serialized data.

INFO

💡 Note: typeCodes must be unique for all classes deriving from any given abstract base class, but typeCodes need not be unique globally (i.e., many classes can share the same typeCode as long as they don't derive from the same base class).

Multiple abstract classes may be derived from a chain, from which a set of non-abstract classes must inherit. The non-abstract classes may not further inherit from other non-abstract classes.

Interface Serialization

Serialization is supported for interfaces in a manner similar to base classes described above:

[MetaSerializable]
public interface IMyInterface
{
    int GetValue();
}

[MetaSerializableDerived(1)]
public class MyFirstClass : IMyInterface
{
    [MetaMember(1)] public int Foo { get; set; }
    [MetaMember(2)] public int Bar { get; set; }
    public int GetValue () => Foo + Bar;
}

[MetaSerializableDerived(2)]
public class MySecondClass : IMyInterface
{
    [MetaMember(1)] public int Baz { get; set; }
    public int GetValue () => Baz;
}

[MetaSerializable]
public class ContainingClass
{
    [MetaMember(10)] public List<IMyInterface> MyObjects;
}

Custom ISerializableTypeCodeProvider

It is also possible to use the ISerializableTypeCodeProvider interface to declare custom attributes, which provide typeCodes without the [MetaSerializableDerived] attribute.

For example, the [ModelAction] attribute uses this to provide typeCodes for all the Actions in a game.

Example usage:

// Declare custom attribute, which is used to give typeCodes to implementation classes.
[AttributeUsage(AttributeTargets.Class)]
public class MyTypeAttribute : Attribute, ISerializableTypeCodeProvider
{
    // Property getter implements ISerializableTypeCodeProvider
    public int TypeCode { get; }

    public MyTypeAttribute(int typeCode) { TypeCode = typeCode; }
}

// Declare base class (must be abstract).
// Using MetaSerializable on abstract class expects the non-abstract implementation
// classes to have an attribute which provides a typeCode for it.
[MetaSerializable]
public abstract class MyBaseType { ... }

// Declare some concrete types of MyBaseType.
// The identifiers must be positive integers.
[MyType(1)] public class MyTypeA : MyBaseType { ... }
[MyType(2)] public class MyTypeB : MyBaseType { ... }

Game Config References

Game config data types have special serialization behavior, in that they are serialized as references instead of as their contents. When a value of a type implementing IGameConfigData<TKey> is serialized, only its key (the TKey-typed property ConfigKey) is serialized. Accordingly, when it is deserialized, the key is used to look up the item in the appropriate GameConfigLibrary.

When TKey is nullable (i.e. a reference type or a Nullable<>), a null reference of type IGameConfigData<TKey> is serialized as the null value of type TKey. However, when TKey is not nullable (for example int, an enum, or a struct), null references are by default not permitted because they cannot be represented using the TKey type; trying to serialize such a null reference will throw an exception. To allow serializing a null reference when TKey is not nullable, you can define a TKey-typed static property named ConfigNullSentinelKey in your IGameConfigData<TKey>-implementing class. Then, null references will be serialized using that sentinel key, and the config library checks that there is no item with that key.

When a game config reference is itself contained within game config data, it needs to be wrapped in the MetaRef<> type. When game configs are deserialized, contained MetaRefs are not resolved immediately, but only after all config libraries have been loaded. This allows forward references as well as references between items in the same config library. See Config Item References Contained in Game Config Data for an example.

Game Config Archives

While Game Config Data (i.e., types implementing the IGameConfigData<> interface) are typically parsed from source formats other than Metaplay's binary format (such as .csv or .json), the ConfigArchive itself is format-agnostic. Configs can thus be stored either in their original source format, or they can be binary-serialized. The need for the [MetaSerializable] and [MetaMember] attributes in Game Config classes depends on whether they use binary-serialization.

Parsing binary-serialized data is faster than parsing .csv or .json files, and has fewer formatting issues with types like DateTimes and fixed-point values.

Serializing Data

Serialization API

The MetaSerialization class is the main API for serializing data via its multiple variants of SerializeTagged() and DeserializeTagged() methods for different scenarios.

INFO

Note: The term Tagged appears in the serialization code. It refers to the currently used ProtoBuf-like wire format. This is in anticipation of the Compact format which relies on both the serializer and the deserializer having the exact same schema for the data, and can, therefore, omit all the metadata included in the Tagged format.

Serialization Use Cases

The MetaSerializationFlags enum is used to specify the purpose of the serialization or deserialization of data, with the following options:

• IncludeAll indicates that all data should be included in serialization.
• SendOverNetwork indicates that the payload is about to be transferred over the network (from server to client or vice versa), and any members marked with the [ServerOnly] attribute should be excluded from the serialization.
• ComputeChecksum indicates that the serialization is performed for the purposes of computing a checksum from the state. Any members with either the [ServerOnly] or [NoChecksum] attribute are excluded from the serialization.
• Persisted indicates that the serialization is performed in order to persist the data into the database. Any members with the [Transient] attribute are excluded from the serialization.

Serialization Troubleshooting

The Tagged format contains enough metadata for it to be parsed without knowing the C# types that the serialized data represents. The TaggedWireSerializer.ToString(byte[]) can be used to convert serialized data to a string for debugging purposes.

If a deserialization operation fails for any reason, it can be helpful to log the contents of the byte array using the method above.

Public vs Private Types

Metaplay tags all serializable types as Public or Private. The Public types are ones that are part of the communication protocol between the client and the server. Metaplay computes a hash of all the Public types. It can be useful in development builds to identify potential client/server incompatibility based on the hash values at each end.

All types in the namespaces Metaplay.Core are marked Public. Additional public namespaces can be specified in MetaplayCoreOptions.

Custom Deserialization Failure Handling

By default, MetaSerialization.DeserializeTagged throws if any part of the input fails to deserialize. Occasionally, to recover from programming mistakes, you may need to handle the deserialization failure of a specific class member without failing the entire deserialization operation. This can be done with the [MetaOnMemberDeserializationFailure(...)] attribute.

For example, imagine you had the following class:

[MetaSerializable]
public class MyClass
{
    [MetaMember(1)] public int Number;
}

Let’s assume that this class is stored inside persistent data such as PlayerModel, and therefore its schema needs to be backwards-compatible.

Imagine you then decided to retire the Number member, and later went on to add a new Name member of type string. By mistake, you re-used the same tagId for the [MetaMember]:

[MetaSerializable]
public class MyClass
{
    // Oops - we changed the type and meaning of the MetaMember without
    // changing the tagId number. This was a mistake!
    [MetaMember(1)] public string Name = "InitialName";
}

DANGER

Warning: Re-using [MetaMember] tag ids should be avoided when there is a backwards-compatibility requirement, because it changes the interpretation of existing data. The tag id re-use shown here is specifically an example of a mistake. To avoid such mistakes, it’s a good practice to use the [MetaBlockedMembers(...)] attribute when retiring old members.

If existing data has already been serialized with the int Number member, then deserializing that data with this new class definition will result in a MetaWireDataTypeMismatchDeserializationException being thrown, because a serialized int cannot be deserialized as a string.

Assuming you cannot fix the problem by simply changing the tag id (for example because some new data has already been persisted with the string Name member by the time you notice the problem), you can consider the following approach:

[MetaSerializable]
public class MyClass
{
    // Specify a method that will get called if the deserialization of the
    // Name member fails. The method will return a replacement value for
    // the Name member.
    [MetaOnMemberDeserializationFailure("FixNameMistake")]
    [MetaMember(1)] public string Name = "InitialName";

    public static string FixNameMistake(MetaMemberDeserializationFailureParams failureParams)
    {
        // Check that the error was what we expect.
        if (failureParams.Exception
            is MetaWireDataTypeMismatchDeserializationException mismatch)
        {
            // Failure was due to a type mismatch.
            // Tolerate the failure, and just use the initial name.
            return "InitialName";
        }
        else
        {
            // The failure was due to something else, unexpected.
            throw failureParams.Exception;
        }
    }
}

Now, a deserialization failure of the Name member will not cause the deserialization of the entire MyClass to fail, since it is contained by the FixNameMistake handler (unless the handler itself throws).

Changing an Existing Serialized Type

Sometimes you may want to change an existing serialized type in a way that changes its serialization format, while retaining the ability to deserialize existing data. Examples of such changes are:

• Changing a concrete class to a base class, of which one subclass corresponds to the original concrete class.
• Changing a struct to a class.

By default, such changes are not backwards-compatible if some data has already been persisted with the old types, because structs, concrete classes, and abstract classes have different serialization formats. However, a few kinds of "deserialization converters" are provided to support these common changes. These converters are specified as attributes on the serializable type.

Concrete Class to Base Class

The [MetaDeserializationConvertFromConcreteDerivedType(typeof(DerivedType))] attribute allows changing a concrete class to a base class. For example, let's say your PlayerModel currently contains a list of SwordModels. (PlayerModel is used for context in this example, but converters apply to all deserialization, not just things within PlayerModel.)

[MetaSerializable]
public class SwordModel
{
    [MetaMember(1)] public int SwordId;
    [MetaMember(2)] public SwordTypeInfo SwordType;
    [MetaMember(3)] public int Level;

    ...
}

[MetaSerializableDerived(1)]
[...]
public class PlayerModel : ...
{
    ...

    [MetaMember(100)] public List<SwordModel> Swords;
}

Now, you decide to introduce also other types of items in your game, such as shields. You'd like to hold all types of items in PlayerModel as an abstract ItemModel class, of which SwordModel and ShieldModel are subclasses. At the same time, existing players have been serialized using the plain concrete SwordModel, and the game must be still be able to deserialize such existing players. This change can be achieved as follows:

[MetaSerializable]
// Using the MetaDeserializationConvertFromConcreteDerivedType attribute,
// we tell the deserializer that ItemModel should accept not only abstract
// objects (the normal format for base classes), but also concrete objects.
// We give it typeof(SwordModel) to specify that concrete objects should
// be deserialized as SwordModels, since that is what the previously-
// serialized data contains.
[MetaDeserializationConvertFromConcreteDerivedType(typeof(SwordModel))]
public abstract class ItemModel
{
    // Note that ItemId here corresponds to SwordId from the earlier
    // snippet; it uses the same MetaMember tag number (1). This is
    // just an example to illustrate that members can be moved from
    // the old concrete class to the base class, if desired.
    [MetaMember(1)] public int ItemId;
}

// SwordModel has been changed to be a subclass of ItemModel:
// it uses [MetaSerializableDerived(...)] instead of [MetaSerializable],
// and one example field has been moved to the base class.
// Other than that, it is the same as in the earlier snippet,
// so that it can be deserialized from existing data.
[MetaSerializableDerived(1)]
public class SwordModel : ItemModel
{
    [MetaMember(2)] public SwordTypeInfo SwordType;
    [MetaMember(3)] public int Level;

    ...
}

// This is a newly-introduced class.
[MetaSerializableDerived(2)]
public class ShieldModel : ItemModel
{
    [MetaMember(2)] public ShieldTypeInfo ShieldType;

    ...
}

[MetaSerializableDerived(1)]
[...]
public class PlayerModel : ...
{
    ...

    // The Swords list has been changed to an Items list, containing
    // ItemModels instead of SwordModels. Normally, the deserializer
    // would here fail to parse the abstract ItemModels from data that
    // was serialized from concrete SwordModels, but thanks to the
    // converter, it is now able to parse them.
    [MetaMember(100)] public List<ItemModel> Items;
}

Struct to Class

The [MetaDeserializationConvertFromStruct] attribute allows changing a struct to a class. Structs (non-nullable) and classes (nullable) have different serialization formats, so by default they are not compatible with each other. With this converter, classes can be deserialized from data that was previously serialized using structs. Note that the other direction is not supported: classes cannot be converted to structs, because structs cannot represent the null value that classes can.

If your old type was a struct:

[MetaSerializable]
public struct MyType
{
    [MetaMember(1)] public int MyField;

    ...
}

You can change it to a class like this:

[MetaSerializable]
// This attribute permits the deserializer to convert from
// serialized struct format to class.
[MetaDeserializationConvertFromStruct]
public class MyType
{
    [MetaMember(1)] public int MyField;

    ...
}