Dependencies

• Understanding entities - The Scan Jobs work by processing the entities in your game's database. If you're not yet familiar with how Metaplay's game servers work, check out Introduction to Entities and Actors.

Key Concepts

• A Database Scan Job is an operation that scans and processes each entity in an entity database table. A Scan Job happens over time and is not instantaneous. After being started, it runs to completion unless explicitly canceled. A running Scan Job can become paused due to higher-priority Jobs.

• A Maintenance Job is a special type of Scan Job that requires less configuration and has to be run manually from the LiveOps Dashboard. They are meant to run during a game's regular operation, so they run relatively slowly to avoid consuming too many resources.

Overview

Scan Jobs are useful when you need to operate on some or all entities of a specific kind. They are especially helpful when there is no simple way pre-filter the entities down to the ones you want to target, but instead, need to evaluate each one.

Some examples of default Jobs the SDK provides are notification campaigns and player deletion.

Here are the basic components that make up the Scan Jobs system:

• Scan coordinator - An entity responsible for coordinating the execution of Jobs and communicating with the scan workers. There is only one scan coordinator in a game's backend.
• Scan workers - The entities that execute the scanning and processing for a Job. Multiple scan workers in the backend work alongside each other to carry out the Scan Jobs' processing.
• Job manager - An object that manages a specific kind of Job. It holds information such as when the Job should start and the Job specifications. A Job manager is not an entity but is part of the scan coordinator, and each type of Scan Job should have its own manager.
• Job Specification - Specifies the type of entity the Job concerns and can create new scan processor instances for the Job.
• Scan processor - An object that implements the actual processing code of a specific kind of Job. A processor takes in batches of scanned items and processes them. A scan processor is not an entity of its own but is contained within a scan worker. Each kind of Job has a corresponding type of processor.

Scan Jobs System vs. Maintenance Jobs

Besides regular Database Scan Jobs, you can create Maintenance Jobs. Maintenance Jobs are a subcategory of Scan Jobs that are initiated manually from the Scan Jobs page in the LiveOps Dashboard and are executed in a simple queued order. If a Maintenance Job is launched while another is active, the new one enters a queue. At most, one Maintenance Job can be active at a time.

Since maintenance Jobs are intended to be run during a game server's regular operation, they run relatively slowly to avoid consuming too many resources. For this reason, they have a lower priority than other types of Scan Jobs and are therefore paused if another Scan Job is running.

Maintenance Jobs are considerably simpler to implement when compared to regular Scan Jobs, so if they meet all of your requirements, we recommend you use them. Maintenance Jobs are appropriate if:

• The Job is meant to be triggered manually on an ad-hoc basis instead of automatically (e.g., periodically).
• It is acceptable for the job to get queued together with the other maintenance jobs.
• The built-in maintenance job UI in the dashboard is sufficient. In particular, the job does not have complex parameterization.

There are a few built-in Maintenance Jobs, like the schema migrator and the entity refresher.

• Schema migrator Job - Migrates all target entities to the latest schema version. It does this by waking up actors for entities that need to be added to the latest schema version. The job produces a summary describing the count of successful migrations and the count of each schema version it encountered.
• Entity refresher Job - Wakes up all target entities. This is useful for executing any code that runs automatically when an actor is initialized or persisted. It is similar to the schema migrator job but also wakes up entities on the latest schema version.

Implementing a Custom Maintenance Job

Defining custom Maintenance Jobs entails creating subclasses of DatabaseScanProcessor and MaintenanceJobSpec, that implement the scan processor and Job specification, respectively.

The DatabaseScanProcessor subclass implements the StartProcessItemBatchAsync method that processes items for the Job. Additionally, a DatabaseScanProcessor specifies some values that control the speed and batch size of the scanning. A DatabaseScanProcessor can contain a state, but it may not be required for all kinds of Jobs. We recommend you derive your class from DatabaseScanProcessor<TScannedItem>, which automatically implements some boilerplate parts of the code based on the item being scanned. Here's some sample code of a processor for a custom Scan Job that just goes over all players and counts them:

CustomMaintenanceJob.cs

[MetaSerializableDerived(202)]
public class CustomMaintenanceJobProcessor : DatabaseScanProcessor<PersistedPlayerBase>
{
    [MetaMember(1)] CustomMaintenanceJobProcessingStatistics _statistics;

    // Control speed and batch size of scanning
    public override int DesiredScanBatchSize => 100;
    public override TimeSpan ScanInterval => TimeSpan.FromSeconds(0.1);
    public override TimeSpan PersistInterval => TimeSpan.FromMilliseconds(1000);
    public override TimeSpan TickInterval => TimeSpan.FromSeconds(1.0);

    public override bool CanCurrentlyProcessMoreItems => true;
    public override bool HasCompletedAllWorkSoFar => true;
    public override DatabaseScanProcessingStatistics Stats => _statistics;

    public CustomMaintenanceJobProcessor() { }
    public CustomMaintenanceJobProcessor(CustomMaintenanceJobProcessingStatistics initialStatisticsMaybe)
    {
        _statistics = initialStatisticsMaybe ?? new CustomMaintenanceJobProcessingStatistics();
    }

    public override Task StartProcessItemBatchAsync(IContext context, IEnumerable<PersistedPlayerBase> persistedPlayers)
    {
        foreach (PersistedPlayerBase persistedPlayer in persistedPlayers)
        {
            // Do something.
            _statistics.PlayersScanned++;
        }

        return Task.CompletedTask;
    }

    public override Task TickAsync(IContext context)
    {
        return Task.CompletedTask;
    }

    public override void Cancel(IContext context)
    {
    }
}

The CustomMaintenanceJobProcessingStatistics inherits from DatabaseScanProcessingStatistics and is used to keep track of whatever statistics you wish. In this case we're using it to count the amount of players scanned.

CustomMaintenanceJob.cs

[MetaSerializableDerived(202)]
public class CustomMaintenanceJobProcessingStatistics : DatabaseScanProcessingStatistics
{
    [MetaMember(1)] public int PlayersScanned = 0;

    public static CustomMaintenanceJobProcessingStatistics ComputeAggregate(IEnumerable<CustomMaintenanceJobProcessingStatistics> parts)
    {
        CustomMaintenanceJobProcessingStatistics aggregate = new CustomMaintenanceJobProcessingStatistics();

        foreach (CustomMaintenanceJobProcessingStatistics part in parts)
        {
            aggregate.PlayersScanned += part.PlayersScanned;
        }

        return aggregate;
    }
}

Next, we implement the MaintenanceJobSpec subclass for our Job specification:

CustomMaintenanceJob.cs

    [MetaSerializableDerived(202)]
    public class CustomMaintenanceJobSpec : MaintenanceJobSpec
    {
        public override string JobTitle => $"Custom Maintenance Job";
        public override string JobDescription => $"Counts number of players scanned.";
        public override string MetricsTag => $"Custom Maintenance Job";
        public override int Priority => DatabaseScanJobPriorities.CustomMaintenanceJob;
        public override EntityKind EntityKind => EntityKindCore.Player;

        public override object JobKindDiscriminator => null;

        CustomMaintenanceJobSpec() { }

        public static CustomMaintenanceJobSpec CreateDefault()
        {
            return new CustomMaintenanceJobSpec();
        }

        public override DatabaseScanProcessor CreateProcessor(DatabaseScanProcessingStatistics initialStatisticsMaybe)
        {
            return new CustomMaintenanceJobProcessor((CustomMaintenanceJobProcessingStatistics)initialStatisticsMaybe);
        }

        public override OrderedDictionary<string, object> CreateSummary(DatabaseScanProcessingStatistics statsParam)
        {
            CustomMaintenanceJobProcessingStatistics stats = (CustomMaintenanceJobProcessingStatistics)statsParam;

            OrderedDictionary<string, object> summary = new()
            {
                { "Player Scanned", stats.PlayersScanned },
            };
            return summary;
        }

        public override DatabaseScanProcessingStatistics ComputeAggregateStatistics(IEnumerable<DatabaseScanProcessingStatistics> parts)
        {
            return CustomMaintenanceJobProcessingStatistics.ComputeAggregate(parts.Cast<CustomMaintenanceJobProcessingStatistics>());
        }
    }

You'll also need to register the maintenance Job and assign a Job priority to it.

To register the maintenance Job, mark the affected PersistedEntityConfigs with the EntityMaintenanceJob attribute:

PlayerActor.cs

[EntityConfig]
[EntityMaintenanceJob("Custom", typeof(CustomMaintenanceJobSpec))]
public class PlayerConfig : PlayerConfigBase
{
    public override Type EntityActorType => typeof(PlayerActor);
}

Finally, each Job has a priority, given by its Job Specification (specifically, the Priority property of DatabaseScanJobSpec). When a Job is already active, the Scan Coordinator will only start a new Job if the priority of the new Job is higher than the existing Job. It will then pause the execution of the lower-priority Job until the new Job competes.

You might have noticed that in the DatabaseScanJobSpec code, we added the priority as DatabaseScanJobPriorities.CustomMaintenanceJob. Now we can actually define it in the DatabaseScanJobPriorities class. The values are defined in this centralized file so that their ordering is easy to manage.

DatabaseScanJobPriorities.cs

/// <summary>
/// Centralized registry for priorities of database scan jobs.
/// Higher value means higher priority, i.e. takes precedence over lower-priority jobs.
/// </summary>
public static class DatabaseScanJobPriorities
{
    public const int ScheduledPlayerDeletion    = 0;
    public const int EntitySchemaMigrator       = 1;
    public const int EntityRefresher            = 1;
    public const int CustomMaintenanceJob       = 1;
    public const int NotificationCampaign       = 2;
}

This mechanism supports long-running background Jobs while still allowing more urgent Jobs to start without delay. For example, a running player deletion job can be paused when a new notification campaign starts. Once the notification campaign ends, the deletion job will resume.

Defining a Custom Database Scan Job

To define a full-blow Database Scan Job as opposed to a Maintenance Job, there are a few extra steps involved.

Firstly, you need to create a DatabaseScanJobManager class. In particular, the subclass will implement the TryGetNextDueJob method, which will return a Job Specification for a Job that is due and whether it is valid to start or null if no Job is available. Additionally, the subclass will implement communication methods with the rest of the game backend and the Scan Coordinator.

Take a look at NotificationCampaignJob.cs and ScheduledPlayerDeletionJob.cs for examples of how their Job managers are implemented. It's also worthwhile to study DatabaseScanUser.cs since this is where all of the base classes you will derive your custom classes from are defined.

After creating your custom Job manager, you must add it to the Scan coordinator. Currently, this is a somewhat manual process; you'll need to add some things in DatabaseScanCoordinator.cs:

First, add a new member to the DatabaseScanJobManagerKind enum, representing your Job Manager type. Please mind the numbering advice in the comments.

DatabaseScanCoordinator.cs

[MetaSerializable]
public enum DatabaseScanJobManagerKind
{
    NotificationCampaign = 0,
    ScheduledPlayerDeletion = 1,
    MaintenanceJobManager = 2,

    TestManager = 99,

    // Please add any game-specific manager kinds here with their numbers starting from 1000.
    // This will help mitigate future conflicts.
}

Then, add a new member to DatabaseScanCoordinatorState for your Job manager using your concrete subclass of DatabaseScanJobManager, with an appropriate initializer for the member.

DatabaseScanCoordinator.cs

[MetaSerializable]
[SupportedSchemaVersions(1, 4)]
public class DatabaseScanCoordinatorState : ISchemaMigratable
{
    ...

    // Add your scan job manager here.
    [MetaMember(100)] public NotificationCampaign.NotificationCampaignManager NotificationCampaignManager { get; private set; } = new NotificationCampaign.NotificationCampaignManager();
    [MetaMember(101)] public ScheduledPlayerDeletion.ScheduledPlayerDeletionManager ScheduledPlayerDeletionManager { get; private set; } = new ScheduledPlayerDeletion.ScheduledPlayerDeletionManager();
    [MetaMember(102)] public MaintenanceJob.MaintenanceJobManager MaintenanceJobManager { get; private set; } = new MaintenanceJob.MaintenanceJobManager();
   
}

Finally, add a case in DatabaseScanCoordinatorActor.GetJobManagerOfKind that maps the newly-added enum member to the newly-added member in DatabaseScanCoordinatorState.

DatabaseScanCoordinator

DatabaseScanJobManager GetJobManagerOfKind(DatabaseScanJobManagerKind kind)
{
    switch (kind)
    {
        case DatabaseScanJobManagerKind.NotificationCampaign: return _state.NotificationCampaignManager;
        case DatabaseScanJobManagerKind.ScheduledPlayerDeletion: return _state.ScheduledPlayerDeletionManager;
        case DatabaseScanJobManagerKind.MaintenanceJobManager: return _state.MaintenanceJobManager;
        case DatabaseScanJobManagerKind.TestManager: return _state.TestManager;
        default:
            throw new InvalidEnumArgumentException(nameof(kind), (int)kind, typeof(DatabaseScanJobManagerKind));
    }
}

That's it! You're all done.

Further Reading

Following this article, it might be a good idea to take a look at the built-in Scan Jobs the Metaplay SDK offers:

• The Schema Migrator - You can learn more about schemas and the schema migrator on the Entity Schema Versions and Migrations page.
• Player Deletion - Player deletion uses a maintenance Job to go through players scheduled for deletion and remove them. Check out Working with Player Deletion for more details.

Here are some more pages on server programming that relate to entities and can be of interest following Scan Jobs:

• Custom Server Entities - Entities are actors responsible for implementing gameplay logic on the server. Custom Server Entities describes in detail how to add your own custom server entities.
• Entity-to-Entity Communication - Entities can communicate with each other to create more sophisticated game logic patterns. Read more about it on Deep Dive: Entity-to-Entity Communication.