Key Concepts

• Analytics Event (or just Event) is something that happens to or is performed by a Game Entity. For example, a player spending diamonds might be a player Event. A guild member gifting an item to another member might be a guild Event.
• Analytics Dispatcher is a global service that collects Analytics Events from all the Game Entities running on a given node and allows writing them into external storage or data stream via Analytics Sinks.
• Analytics Sink (or just Sink) is a component that is registered to the Analytics Dispatcher and writes Analytics Events into External Storage. It is up to the Sink to convert the incoming Events into any format that it wants (eg, JSON or Parquet). Metaplay ships with a built-in sink that writes JSON files into S3 (with optional compression). Custom Sinks can also be implemented to support new data formats or storage types.
• External Storage refers to 3rd party storage, database, or data stream. Some typical analytics-oriented databases are Google BigQuery and AWS RedShift. Blob storages like S3 and GCP can be used in data-lake approaches. Also, data streams like AWS Kinesis or Kafka can be written to. The data could also be written to the local or remote disk, or pushed over HTTP(S) to any external service.
• Event Batch (or Batch of Events) means a collection of Events collected from a single Game Entity which is flushed to the Analytics Dispatcher in a single operation.
• Event Chunk is a collection of Events that the Sinks use as the granularity of flushing to External Storage. A Sink might collect multiple Batches to form a single Chunk before flushing it out.

Overview

Metaplay supports streaming Analytics Events from the Game Servers to various External Storages. This allows integrating with various 3rd party or custom-made analytics pipelines and ETL solutions to process and visualize the Events. Metaplay itself doesn't implement any ETL capabilities directly.

To better understand how to start collecting Analytics Events in your game, see Implementing Analytics Events in Game Logic.

Event Collection Flow

The flow of Analytics Events from the source to the Sinks is described below. This is mainly for informational purposes, to better understand the system and the latencies involved.

1. Events are collected from the source Game Entities. This can happen either from the shared game logic via, eg, PlayerModel.EventStream.Event(new PlayerEventXyz(...)) or directly on the server via, eg, PlayerActor._analyticsEventBatcher.Enqueue(new PlayerEventXyz(..)).
2. The Event is buffered up in the Actor for up to 30 seconds (this may change in the future), or until the maximum number (currently, 100) of Events have been queued up. When either condition is triggered, the Events are flushed in a single Batch to the Analytics Dispatcher.
3. The Analytics Dispatcher then forwards the Event Batches to all the Analytics Sinks.
4. Each Sink should then immediately convert the Events to its target format (eg, JSON) and buffer up the Events as a Chunk. When enough Events have accumulated, or enough time has passed, the Sink should flush the Chunk into its External Storage.

INFO

Currently, collecting Analytics Events is only supported in PlayerActor and GuildActor. In the future, this will be generalized to more Entities.

Event Structure

The metadata for an Event is represented by the AnalyticsEventEnvelope class, with the following members (matching the JSON):

• Source is the EntityId of the Game Entity where the Event was originated. Events are always associated with an Entity.
• ModelTime is the logical time when the Event happens (taken from PlayerModel.CurrentTime). Due to buffering and network latencies, the actual time when the event is captured on the server might actually happen somewhat after the logical time.
• (CollectedAt is the timestamp when the Event was collected at the Game Server. This is currently disabled, as it wasn't considered to be useful. If it feels useful, let us know and we can enable it.)
• UniqueId is a fully unique identifier assigned to the Event. It's mainly intended for de-duplicating Events in case the Sink implementation can cause duplicates due to retry-based error handling. It can also be used to correlate Events in the Entity's Event Log with those outputted by the Sinks.
• EventType is the name of the C# class that represents the Event type.
• SchemaVersion is the version of the Event. It is useful for making it easier to handle changes in the Event structure when processing in an ETL pipeline.
• Payload contains the Event-specific parameters.
• Context contains the Source-specific context data. For players, this contains the current sessionNumber or null if event was emitted outside a session, and the active experiments and for each the active variant.
• Labels contains the arbitrary custom data emitted by the source Game Entity.

Some example Events (after conversion to JSON & pretty-printed on multiple lines):

Player example event:
{
  "source": "Player:X9Pq78AMqb",
  "modelTime": "2021-09-28T15:11:28.6980000Z",
  "uniqueId": "0000017C2CF66B0A-3E044240ECE36CAC",
  "eventType": "PlayerEventClientDisconnected",
  "schemaVersion": 1,
  "payload": {
    "sessionToken": "54579D4B2A2EEF2B"
  },
  "context": {
    "sessionNumber": 2,
    "experiments": {
      "_egf": "_fast"
    }
  },
  "labels": {
    "Location": "FI"
  }
}

Guild example event:
{
  "source": "Guild:ExeVdW76C3",
  "modelTime": "2021-09-28T15:33:46.8140000Z",
  "uniqueId": "0000017C2D0A85B3-51862C7D1EF5F073",
  "eventType": "GuildEventMemberPoked",
  "schemaVersion": 1,
  "payload": {
    "pokingPlayerId": "Player:X9Pq78AMqb",
    "pokingPlayerMemberInstanceId": 0,
    "pokingPlayerName": "Guest 80142",
    "pokedPlayerId": "Player:X9Pq78AMqb",
    "pokedPlayerMemberInstanceId": 0,
    "pokedPlayerName": "Guest 80142",
    "pokedCountAfter": 1
  },
  "context": {},
  "labels": {}
}

Analytics Sinks

The purpose of the Sinks is to take the incoming Event Batches, convert them to desired format (eg, JSON or Parquet), optionally collect multiple Batches to larger Chunks, and then flush the Chunks out to the External Storage of choice (eg, S3, Kinesis, BigQuery, Amplitude, or similar).

It is up to the Sink how this process should be done, what formats and External Storage it supports, what (if any) compression methods it supports, how it handles failures and retries, and so on. Metaplay ships with a built-in JSON-to-S3 and a built-in BigQuery Sinks. The JSON-to-S3 is additionally an reference implementation of a Sink that works as an example of how to solve the common problems in a realistic manner and acts as a starting point for implementing your custom Sinks.

Built-in JSON-to-S3 Sink

Metaplay ships with a built-in Sink AnalyticsDispatcherSinkJsonBlobStorage that supports writing JSON data into an S3 bucket.

It is mainly intended to be a reference implementation of how a Sink can be implemented, but may also be useful in some production scenarios.

Instead of writing directly to S3, the Sink uses the IBlobStorage interface, which will write to the disk when running the server locally. The files are written to a configurable path, by default Game.Server/bin/PrivateBlobStorage with paths like ServerAnalytics/2021/12/31/<hostname>-235901-b7ad85cd.json.gz.

It demonstrates the following key concepts:

• Converting Analytics Events into JSON, including the built-in converters for custom data types.
• Flushing the data out to S3 either at a) reaching a specified number of events, or b) maximum time an Event has been pending, whichever happens first.
• Using multiple buffers (MemoryStreams) to write out data in the background, in parallel. The uploading is retried a few times to avoid temporary glitches from causing data loss. A fixed number of buffers is kept to force an upper limit of memory usage for the Events, to avoid the server running out of memory. The implementation can only handle short periods of inability to write before it starts dropping events.
• Converting incoming Analytics Events into JSON as the events come in. This allows using recycled byte buffers for the stored data instead of keeping the Event class instances in memory.
• Optional compression of the outgoing to GZIP format. Also done in streaming fashion to converse memory usage and to amortize the CPU usage over time.
• A simple partitioning of the written Events by time is supported. Note that the partitioning only happens based on the time when the given Chunk of Events is ready, not on the individual Events' timestamps.
• Outputting relevant Prometheus metrics.

While the JSON-to-S3 Sink is mainly intended as a reference implementation for building custom Sinks, it may still be useful in real-life production settings as well. For example, it it possible to hook an AWS Lambda Function to trigger when files are written to the S3 bucket, and then perform initial processing of the Events and write them out to any External Storage.

Configuring the JSON-to-S3 Sink

The sink can be configured via the Runtime Options, e.g. Options.base.yaml:

AnalyticsSinkJsonBlobStorage:
  Enabled:            false
  FilePath:           "ServerAnalytics/{Year}/{Month}/{Day}/{HostName}-{Hour}{Minute}{Second}-{UniqueId}.json{CompressSuffix}"
  CompressMode:       Gzip      # None to disable compression
  EventsPerChunk:     100000    # Target number of events per chunk (100k ~= 20MB uncompressed, assuming 200 bytes/event)
  MaxPendingDuration: 00:05:00  # Maximum amount of time an event can be buffered before the batch is flushed
  NumChunkBuffers:    10        # Number of chunk upload buffers to use (if all become filled due to slow/failed uploads, subsequent events are dropped)

  # Use these to write to a custom S3 bucket
  #S3BucketName: custom-bucket-name
  #S3Region:     eu-west-1

Built-in BigQuery Sink

Metaplay ships with a built-in Sink AnalyticsDispatcherSinkBigQuery that supports writing analytics event data into an BigQuery table. Each event is stored in the BigQuery table as a single row. Since this row-format does not support arbitrarily nested data structures, all Payload data is flattened into a list of key-value pairs. The format is described in:

BigQuery Analytics Event Format

To set up the target BigQuery table, you may use the Metaplay BigQuery Analytics infra module. The process is described in the infra documentation. Using the built-in module is recommended as it will automatically set a proper schema table and set up the table with sensible default values.

Configuring the BigQuery Sink

The sink can be configured via the Runtime Options. If you are using the built-in infra modules for setting up the table, as defined above, the Project, Dataset and Table information and Credentials will be automatically set.

AnalyticsSinkBigQuery:
    Enabled:            true

    # Maximum number of events per insert batch
    EventsPerChunk:     100

    # Maximum amount of time an event can be buffered before the batch is flushed
    MaxPendingDuration: 00:00:30

    # Number of chunk upload buffers to use (if all become filled due to slow/failed
    # uploads, subsequent events are dropped)
    NumChunkBuffers:    10

    # Enables the native BigQuery row deduplication based on insertId. If this is
    # disabled, data processor should deduplicate events based on event_id column.
    # This is enabled by default.
    BigQueryEnableRowDeduplication: false

    # Target table
    BigQueryProjectId: "myproject"
    BigQueryDatasetId: "mydataset"
    BigQueryTableId:   "mytable"

    # file path to the credentials, or aws-sm:// url for credential from AWS Secrets
    # Manager
    BigQueryCredentialsJsonPath: xxx

Implementing a Custom Sink

The simplest way to get started with a custom Sink is to copy the existing AnalyticsDispatcherSinkJsonBlobStorage and use it as a basis. It already solves many common issues that Sinks need to address, so no need to re-invent the wheel on those.

An instance of the custom sink needs to be registered to the AnalyticsDispatcher on server start. This is done by introducing a class that derives from AnalyticsDispatcherSinkFactory and returns the initialized custom sink in its CreateSinksAsync method:

public class MyAnalyticsDispatcherSinkFactory : AnalyticsDispatcherSinkFactory
{
    public override Task<IEnumerable<AnalyticsDispatcherSinkBase>> CreateSinksAsync()
    {
        AnalyticsDispatcherSinkBase customSink = new MyCustomAnalyticsDispatcher();
        return Task.FromResult(Enumerable.Repeat(customSink, 1));
    }
}

The main interface for a Sink is the EnqueueBatches() method. The Analytics Dispatcher uses it to append any new Event Batches to the Sink. The method is called frequently (once per second) even if no Events have occurred, so it can be used to implement time-based flushing. It is up to the Sink to do whatever it wishes with the Events.

There's a convenience ChunkBufferManager class, which can be used to keep a fixed-size pool of buffers (MemoryStreams) for writing the Events into, and then flushing to the External Storage in the background.

Configuring the Sink is best done using the Runtime Options. Unfortunately, there's no reference for this at this time.

Things to consider:

• Format of Events: JSON is supported out-of-the-box, but is not the most efficient format for processing the data. Often this isn't a problem, though, as the raw output is probably processed to another format anyway, something more optimized for querying, and the transformation can be done during that processing to something more efficient like Parquet.
• Chunking granularity: What's the desired size of the Chunks and how often should they be flushed even if the target size isn't reached? For example, AWS Athena recommends around 128MB blobs to optimize efficiency. Then again, in some scenarios, a shorter latency might be more relevant to optimize for.
• Partitioning: How should the output data be partitioned? Should the Sink handle per-Event partitioning or is it enough to partition on the level of Event Chunks and let the downstream pipeline handle the per-Event partitioning?
• Robustness: The External Storages are not all equal, and can vary in uptime guarantees or have other potential failure modes, like running out of space.
• Garbage collector pressure: It's recommended to convert all the input data immediately upon receiving to a format that doesn't put as much pressure on the garbage collector as the Event data classes do. In practice, this means converting to JSON (or similar) immediately upon receiving the Events, so they can be kept in reusable byte buffers.
• Memory usage: It's a good idea to limit the maximum memory usage of the Sink even in cases where the downstream External Storage isn't accepting any more input, due to transient failure or other reasons like running out of space. Otherwise, the memory usage will keep growing until the machine runs out of memory and terminates the whole Game Server.
• Compression: It's probably a good idea to compress the data into long-term storage. GZIP is a good all-around compression method, but if performance is of utmost importance, there are faster methods like Snappy. This is a speed-vs-size trade-off.
• Metrics: The Sink should output enough relevant Prometheus metrics to make it observable and allow alerting on in production.

Metadata Events

In addition to game-specific events, Metaplay emits certain metadata events to enable integration with Analytics Pipelines. For these events, the Source entity may not be defined. Current metadata events are:

• PlayerDeleted event is emitted when Player is being deleted, possibly as a result to a GDPR data removal request (Right to Erasure). Upon receiving this event, the consuming pipeline should make sure all Personal Data of this player is deleted. The Source of this event is the PlayerId of the player being removed.

  Note that Metaplay does not remove any events from this player from BigQuery Table automatically. It is expected that the developer has either:

  1. Configured the destination BigQuery Table with sufficient Partition Expiration such that data in BigQuery Table can be considered ephemeral and will be removed sufficiently quickly. The built-in Metaplay infra module allows configuring Partition Expiration as desired.
  2. Configured the consuming Analytics pipeline to remove the consumed data from BigQuery Table (for example by submitting partition deletions after consuming the data) such that data in BigQuery Table can be considered ephemeral.
  3. Avoided recording any Personal Data in Analytics Event data.

• PlayerEventFacebookAuthenticationRevoked with Source = DataDeletionRequest is emitted when a User that logged in with a Facebook Account chooses to Submit Facebook Data Deletion Request on Facebook Settings & Privacy > Settings > Apps and Websites page. Upon receiving this event, the consuming pipeline should make sure all Facebook-sourced data of this player is deleted. The Source of this event is the PlayerId of the player user has logged on to.

  Currently only data collected is the App-Scoped User Id which may be present in Social Login related Analytics Events. As the User ID is App-Scoped, i.e. all Apps use a separate User ID namespaces, it cannot be associated with any natural person and is not Personal Data.

• ServerEventExperimentInfo and ServerEventExperimentVariantInfo events are emitted on server startup and whenever Game Configs are changed. ServerEventExperimentInfo is emitted for each Experiment and it contains information about the experiment, such as the human-readable display name, description, internal Experiment Id, and the Analytics ID. Similarly, ServerEventExperimentVariantInfo is emitted for each variant of each active Experiment, and it contains both the Analytics ID and the internal Id of the experiment.