Dependencies

• Understanding entities - The Scan Jobs work by processing the entities in your game's database. If you're not yet familiar with how Metaplay's game servers work, check out Introduction to Entities and Actors.

Key Concepts

• A Database Scan Job is an operation that scans and processes each entity in an entity database table. A Scan Job happens over time and is not instantaneous. After being started, it runs to completion unless explicitly canceled. A running Scan Job can become paused due to higher-priority Jobs.

• A Maintenance Job is a special type of Scan Job that requires less configuration and has to be run manually from the LiveOps Dashboard. It is meant to run during a game's regular operation, so it runs relatively slowly to avoid consuming too many resources.

Overview

Scan Jobs are useful when you need to operate on some or all entities of a specific kind. They are especially helpful when there is no simple way to pre-filter the entities down to the ones you want to target, but instead, need to evaluate each one.

Some examples of default Jobs the SDK provides are notification campaigns and player deletion.

Here are the basic components that make up the Scan Jobs system:

• Scan coordinator - An entity responsible for coordinating the execution of Jobs and communicating with the scan workers. There is only one scan coordinator in a game's backend.
• Scan workers - The entities that execute the scanning and processing for a Job. Multiple scan workers in the backend work alongside each other to carry out the Scan Jobs' processing.
• Job manager - An object that manages a specific kind of Job. It holds information such as when the Job should start and the Job specifications. A Job manager is not an entity but is part of the scan coordinator, and each type of Scan Job should have its own manager.
• Job Specification - Specifies the type of entity the Job concerns and can create new scan processor instances for the Job.
• Scan processor - An object that implements the actual processing code of a specific kind of Job. A processor takes in batches of scanned items and processes them. A scan processor is not an entity of its own but is contained within a scan worker. Each kind of Job has a corresponding type of processor.

Scan Jobs System vs. Maintenance Jobs

Besides regular Database Scan Jobs, you can create Maintenance Jobs. Maintenance Jobs are a subcategory of Scan Jobs that are initiated manually from the Scan Jobs page in the LiveOps Dashboard and are executed in a simple queued order. If a Maintenance Job is launched while another is active, the new one enters a queue. At most, one Maintenance Job can be active at a time.

Since maintenance Jobs are intended to be run during a game server's regular operation, they run relatively slowly to avoid consuming too many resources. For this reason, they have a lower priority than other types of Scan Jobs and are therefore paused if another Scan Job is running.

Maintenance Jobs are considerably simpler to implement when compared to regular Scan Jobs, so if they meet all of your requirements, we recommend you use them. Maintenance Jobs are appropriate if:

• The Job is meant to be triggered manually on an ad-hoc basis instead of automatically (e.g., periodically).
• It is acceptable for the job to get queued together with the other maintenance jobs.
• The built-in maintenance job UI in the dashboard is sufficient. In particular, the job does not have complex parameterization.

There are a few built-in Maintenance Jobs, like the schema migrator and the entity refresher.

• Schema migrator Job - Migrates all target entities to the latest schema version. It does this by waking up actors for entities that need to be added to the latest schema version. The job produces a summary describing the count of successful migrations and the count of each schema version it encountered.
• Entity refresher Job - Wakes up all target entities. This is useful for executing any code that runs automatically when an actor is initialized or persisted. It is similar to the schema migrator job but also wakes up entities on the latest schema version.

Implementing a Custom Maintenance Job

Defining custom Maintenance Jobs entails creating subclasses of DatabaseScanProcessor and MaintenanceJobSpec, that implement the scan processor and Job specification, respectively.

The DatabaseScanProcessor subclass implements the StartProcessItemBatchAsync method that processes items for the Job. Additionally, a DatabaseScanProcessor specifies some values that control the speed and batch size of the scanning. A DatabaseScanProcessor can contain a state, but it may not be required for all kinds of Jobs. We recommend you derive your class from DatabaseScanProcessor<TScannedItem>, which automatically implements some boilerplate parts of the code based on the item being scanned. Here's some sample code of a processor for a custom Scan Job that just goes over all players and counts them:

CustomMaintenanceJob.cs

[MetaSerializableDerived(202)]
public class CustomMaintenanceJobProcessor : DatabaseScanProcessor<PersistedPlayerBase>
{
    [MetaMember(1)] CustomMaintenanceJobProcessingStatistics _statistics;

    // Control speed and batch size of scanning
    public override int DesiredScanBatchSize => 100;
    public override TimeSpan ScanInterval => TimeSpan.FromSeconds(0.1);
    public override TimeSpan PersistInterval => TimeSpan.FromMilliseconds(1000);
    public override TimeSpan TickInterval => TimeSpan.FromSeconds(1.0);

    public override bool CanCurrentlyProcessMoreItems => true;
    public override bool HasCompletedAllWorkSoFar => true;
    public override DatabaseScanProcessingStatistics Stats => _statistics;

    public CustomMaintenanceJobProcessor() { }
    public CustomMaintenanceJobProcessor(CustomMaintenanceJobProcessingStatistics initialStatisticsMaybe)
    {
        _statistics = initialStatisticsMaybe ?? new CustomMaintenanceJobProcessingStatistics();
    }

    public override Task StartProcessItemBatchAsync(IContext context, IEnumerable<PersistedPlayerBase> persistedPlayers)
    {
        foreach (PersistedPlayerBase persistedPlayer in persistedPlayers)
        {
            // Do something.
            _statistics.PlayersScanned++;
        }

        return Task.CompletedTask;
    }

    public override Task TickAsync(IContext context)
    {
        return Task.CompletedTask;
    }

    public override void Cancel(IContext context)
    {
    }
}

The CustomMaintenanceJobProcessingStatistics inherits from DatabaseScanProcessingStatistics and is used to keep track of whatever statistics you wish. In this case, we're using it to count the number of players scanned.

CustomMaintenanceJob.cs

[MetaSerializableDerived(202)]
public class CustomMaintenanceJobProcessingStatistics : DatabaseScanProcessingStatistics
{
    [MetaMember(1)] public int PlayersScanned = 0;

    public static CustomMaintenanceJobProcessingStatistics ComputeAggregate(IEnumerable<CustomMaintenanceJobProcessingStatistics> parts)
    {
        CustomMaintenanceJobProcessingStatistics aggregate = new CustomMaintenanceJobProcessingStatistics();

        foreach (CustomMaintenanceJobProcessingStatistics part in parts)
        {
            aggregate.PlayersScanned += part.PlayersScanned;
        }

        return aggregate;
    }
}

Next, we implement the MaintenanceJobSpec subclass for our Job specification:

CustomMaintenanceJob.cs

[MetaSerializableDerived(202)]
public class CustomMaintenanceJobSpec : MaintenanceJobSpec
{
    public override string JobTitle => "Custom Maintenance Job";
    public override string JobDescription => "Counts the number of players scanned.";
    public override string MetricsTag => "CustomMaintenanceJob";
    public override int Priority => DatabaseScanJobPriorities.ScheduledPlayerDeletion;
    public override EntityKind EntityKind => EntityKindCore.Player;

    public override object JobKindDiscriminator => null;

    CustomMaintenanceJobSpec() { }

    public static CustomMaintenanceJobSpec CreateDefault()
    {
        return new CustomMaintenanceJobSpec();
    }

    public override DatabaseScanProcessor CreateProcessor(DatabaseScanProcessingStatistics initialStatisticsMaybe)
    {
        return new CustomMaintenanceJobProcessor((CustomMaintenanceJobProcessingStatistics)initialStatisticsMaybe);
    }

    public override MetaDictionary<string, object> CreateSummary(DatabaseScanProcessingStatistics statsParam)
    {
        CustomMaintenanceJobProcessingStatistics stats = (CustomMaintenanceJobProcessingStatistics)statsParam;

        MetaDictionary<string, object> summary = new()
        {
            { "Players Scanned", stats.PlayersScanned },
        };
        return summary;
    }

    public override DatabaseScanProcessingStatistics ComputeAggregateStatistics(IEnumerable<DatabaseScanProcessingStatistics> parts)
    {
        return CustomMaintenanceJobProcessingStatistics.ComputeAggregate(parts.Cast<CustomMaintenanceJobProcessingStatistics>());
    }
}

Each Job has a priority, given by the Priority property of DatabaseScanJobSpec. When a Job is already active, the Scan Coordinator will only start a new Job if the priority of the new Job is higher than the existing Job. It will then pause the execution of the lower-priority Job until the new Job completes. In the above snippet, we set the priority to the same value as used by the SDK's player deletion Job, but you can set it to any appropriate value to define its priority relative to other types of Jobs.

You'll also need to register the maintenance Job. Mark the affected PersistedEntityConfigs with the EntityMaintenanceJob attribute:

PlayerActor.cs

[EntityConfig]
[EntityMaintenanceJob("Custom", typeof(CustomMaintenanceJobSpec))]
public class PlayerConfig : PlayerConfigBase
{
    public override Type EntityActorType => typeof(PlayerActor);
}

:::

This mechanism supports long-running background Jobs while still allowing more urgent Jobs to start without delay. For example, a running player deletion job can be paused when a new notification campaign starts. Once the notification campaign ends, the deletion job will resume.

Defining a Custom Database Scan Job

To define a full-blown Database Scan Job as opposed to a Maintenance Job, you need to create a subclass of DatabaseScanJobManager. In particular, the subclass will implement the TryGetNextDueJob method, which will return a Job Specification for a Job that is due and whether it is valid to start or null if no Job is available. Additionally, the subclass will implement communication methods with the rest of the game backend and the Scan Coordinator.

Take a look at NotificationCampaignJob.cs and ScheduledPlayerDeletionJob.cs for examples of how their Job managers are implemented. It's also worthwhile to study DatabaseScanUser.cs since this is where all of the base classes you will derive your custom classes from are defined.

The SDK detects all defined manager classes and automatically creates one instance of each. The manager's serializable state will be automatically persisted in the database.

Further Reading

Following this article, it might be a good idea to take a look at the built-in Scan Jobs the Metaplay SDK offers:

• The Schema Migrator - You can learn more about schemas and the schema migrator on the Entity Schema Versions and Migrations page.
• Player Deletion - Player deletion uses a maintenance Job to go through players scheduled for deletion and remove them. Check out Working with Player Deletion for more details.

Here are some more pages on server programming that relate to entities and can be of interest following Scan Jobs:

• Custom Server Entities - Entities are actors responsible for implementing gameplay logic on the server. Custom Server Entities describes in detail how to add your own custom server entities.
• Entity-to-Entity Communication - Entities can communicate with each other to create more sophisticated game logic patterns. Read more about it on Deep Dive: Entity-to-Entity Communication.