Dependencies

• Working Knowledge of Programming on the Metaplay Server - The Workload balancing system works by distributing entities between nodes. If you're not yet familiar with how Metaplay's game servers work, check out Introduction to Entities and Actors.

• Familiarity with Custom Server Entities - Entities are actors responsible for implementing gameplay logic on the server. The Custom Server Entities page describes in detail how to add custom server entities.

• Sharding Strategies - Workload balancing is only available for static entities that use the StaticModuloShardingStrategy or the StaticServiceShardingStrategy. For dynamic entities, the DynamicServiceShardingStrategy and ManualShardingStrategy strategies are supported.

Key Concepts

• A Workload is a user-defined data structure that defines how much work is running on a specific node.
• Collecting a Workload happens when the Workload collector queries every node in the cluster to return the requested data. It happens periodically, and the data is then synced to all other nodes in the network.

Overview

When creating new entities, it can be desirable to evenly distribute them across different nodes. This is especially important when they are doing resource-intensive work and evenly distributing the Workload can increase performance.

Workloads can set soft and hard limits to the work being scheduled on a node. The soft limit will block new work from being scheduled (when using WorkloadSchedulingUtility) until all nodes have hit the soft limit. The hard limit, however, will stop any more work from being scheduled whatsoever. These can be used to create an elastic limit, where nodes can set their "optimal" load level but stretch (with potential degraded performance) during peak load.

Currently, persisted entities that are restored from the database are not evenly distributed across nodes, meaning this is mostly useful for shorter-lived persisted entities or ephemeral entities.

We have a few different tools to assist in Workload tracking:

• The WorkloadSchedulingUtility can be used to create EntityIds that will run on the node with the lowest Workload. A sample of how it's used will be available in the example below.
• WorkloadCollector is responsible for Collecting the Workload of the current node.
• The WorkloadBase is the data structure that keeps track of the Workload and is serialized and synced across nodes. It is required to implement IComparable<WorkloadBase>. We recommend keeping the size of this type small as it is often serialized and synced over the network. If you're not familiar with how IComparable<T> works, take a look at the .NET docs.
• WorkloadGauge and WorkloadCounter are utility metric types that we recommend you use. -WorkloadGauge is a single numerical value that can arbitrarily be set, a good use-case of this would be the CPU usage of a node. -WorkloadCounter is a counter that can increase and decrease. It is internally represented by 2 monotonically increasing counters. When no more work is running on a node, this should evaluate to 0, meaning you've done the same amount of increases and decreases.

Example Implementation

Let's say our game has a "battle" mechanic, and we want to create a Workload that reports the number of battles happening on the current node:

BattleWorkload.cs

[MetaSerializableDerived(1)]
public class BattleWorkload : WorkloadBase
{
    [MetaMember(1)] public WorkloadCounter Battles;

    [MetaDeserializationConstructor]
    public BattleWorkload(WorkloadCounter battles)
    {
        Battles = battles;
    }
    
    public override bool IsWithinSoftLimit(NodeSetConfig nodeSetConfig, ClusterNodeAddress nodeAddress)
    {
        // In this case we do a very simple check if a specific node has a certain port,
        // however you can also do more complicated checks here. For example, looking up the
        // current CPU usage.
        if (nodeAddress.Port == 6020)
            return Battles.Value < 2;
        return true;
    }

    // The CompareTo method defines sorting orther.
    public override int CompareTo(WorkloadBase other)
    {
        if (other is not BattleWorkload workload)
            return 1;

        return CompareTo(workload);
    }

    public int CompareTo(BattleWorkload other)
    {
        if (ReferenceEquals(this, other)) return 0;
        if (ReferenceEquals(null, other)) return 1;
        return Comparer<WorkloadCounter>.Default.Compare(Battles, other.Battles);
    }
}

Then, we define a WorkloadCollector that has a static WorkloadCounter that keeps track of the current amount of battles and can be increased or decreased from within the BattleActor entity.

BattleWorkloadCollector.cs

public class BattleWorkloadCollector : WorkloadCollector
{
    public static WorkloadCounter Battles = new();
 
    public override Task<WorkloadBase> Collect()
    {
        return Task.FromResult<WorkloadBase>(new BattleWorkload(Battles));
    }
}

We increase the counter when an entity is created, and decrement when the entity is shutdown.

BattleActor.cs

public sealed class BattleActor : PersistedMultiplayerEntityActorBase<BattleModel, BattleAction, PersistedBattle>, IBattleModelServerListener
{
    protected override Task OnEntityInitialized()
    {
        BattleWorkloadCollector.Battles.Increment();
        return base.OnEntityInitialized();
    }

    ...
    
    protected override Task OnShutdown()
    {
        BattleWorkloadCollector.Battles.Decrement();
        return base.OnShutdown();
    }
}

Lastly, we'll use the WorkloadSchedulingUtility to create the BattleActor entity on the right node. Note that the CreateEntityIdOnBestNode method is async and does not complete instantly as it requests the latest workload status.

async Task<(EntityId, BattleSetupParams)> StartBattleAsync(BattleEntry battleEntry)
{
    // Create a new battle entity
    var newBattleId = await WorkloadSchedulingUtility.CreateEntityIdOnBestNode(EntityKindGame.Battle);
    if (!await DatabaseEntityUtil.CreateNewEntityAsync<PersistedBattle>(newBattleId))
        throw new InvalidOperationException("Could not create BattleActor on node provided by load tracking");
    
    BattleMemberAvatar[] avatars = GetAvatarsForBattleEntry(battleEntry).ToArray();

    BattleSetupParams setupParams = new BattleSetupParams(avatars);
    _ = await EntityAskAsync<InternalEntitySetupResponse>(newBattleId,
        new InternalEntitySetupRequest(setupParams));

    return (newBattleId, setupParams);
}