Overview

When Metaplay Analytics Events are exported via BigQuery Analytics Sink into a BigQuery Table, the event format is modified to roughly emulate the native export format of Firebase Analytics. The intention is to make integration easy with Firebase-aligned tooling. As with Firebase, Metaplay writes each Analytics Event as a single row in the BigQuery Table and the values in a AnalyticsEventEnvelope are exported as follows:

• Source is mapped to source_id.
• UniqueId is mapped to event_id. This value is also used as the BigQuery deduplication key if native BigQuery deduplication is enabled.
• ModelTime is mapped to event_timestamp.
• EventType is mapped to event_name
• SchemaVersion is mapped to event_schema_version
• Payload is flattened into event_params as key-value pairs. The process is described in more detail below.
• Context is mapped to the proper source-specific fields as defined in the schema. Currently only Player-entities have a context. The player context is exported as follows:
  • sessionNumber is mapped into player.session_number
  • active experiments are mapped into player.experiments list where each experiment is represented by { "experiment": analyticsId, "variant": analyticsId } pair.
• Labels is mapped to a list where each element is represented by a { "name": name, "string_value": value } pair.

Example event:

{
  "source_id": "Player:123",
  "event_id": "11223344AABBCCDD66778899EEFF0011",
  "event_timestamp": "2020-10-10 14:40:12.000000",
  "event_name": "PlayerEventClientDisconnected",
  "event_schema_version": 1,
  "event_params": [
    {
      "key": "SessionToken",
      "string_value": "0000000000BC614E"
    }
  ],
  "player": {
    "session_number": 4,
    "experiments": [
      {
        "experiment": "experimentId",
        "variant": "variantId"
      }
    ]
  },
  "labels": [
    {
      "name": "Location",
      "string_value": "FI"
    }
  ]
}

Payload Representation and Flattening

All Event Payload fields are represented as key-value pairs. The key is always a string and the value is either an integer, a floating point value, a string value, or a null value. This is represented as struct of {"key": key, "string_value": value, "int_value": value, "double_value": value }. When exported, a single field will have only a single *_value field set, or none in the case of a null value.

When an Event is exported, each field in the Payload object is mapped to a single key-value pair in the exported row. For flat Payload objects, such as the example PlayerEventClientDisconnected, each field is trivially mapped to a key-value pair without any transformation. In the case of the example, the SessionToken field is mapped to a SessionToken key. But if the Payload contains either (1) Aggregate types, (2) Array types, or (3) Dynamic Types, a the object tree is walked and flattened to key-value pairs as follows:

1. Aggregate types, i.e. fields that contain multiple fields, are represented as multiple key-value pairs where each subfield uses a name of <aggregate_field_name>:<subfield_name>. For example, a Pair pair field where Pair is struct Pair { int A, int B } would be represented as pair:A and pair:B int-valued key-value pairs. If the field is null, it is interpreted as if all subfields were null.
2. Array types are represented as their elements where each element use a name of <array_field_name>:<array_index>. For example, int[] value would be represented as value:0, value:1, ... fields.
3. Dynamic types, i.e. field has is declared with a base type and the contents depend on the runtime value of the field, are represented like Aggregate Types but with an additional synthetic $t subfield. The value $t string is the runtime type of the field. For example, a Basetype field with a value of class ConcreteType : BaseType { int A } would be exported as field:$t = "ConcreteType" and field:A subfields. A null value is interpreted as if there are no field.

The rules are applied recursively, and single top-level field the use of all of the rules. For example an single MetaReward field named as ResolvedContent might expand to:

{
    "key": "ResolvedContent:$t",
    "string_value": "ResolvedPurchaseMetaRewards"
},
{
    "key": "ResolvedContent:Rewards:0:$t",
    "string_value": "RewardGems"
},
{
    "key": "ResolvedContent:Rewards:0:Amount",
    "int_value": 123
},
{
    "key": "ResolvedContent:Rewards:1:$t",
    "string_value": "RewardGold"
},
{
    "key": "ResolvedContent:Rewards:1:Amount",
    "int_value": 123
},

The flattening of individual types and members can be controlled with a BigQueryAnalyticsFormat attribute which allows choosing the flattening mode. Currently, the only custom mode is Ignore, which removes the annotated Type or Member from the BigQuery flattening result. This can be used as follows:

class Event
{
    //...

    // this field is not exported via BigQuery
    [BigQueryAnalyticsFormat(BigQueryAnalyticsFormatMode.Ignore)]
    public int Field;

    // this field is also not exported via BigQuery
    public DataFragment fragment;
}

[BigQueryAnalyticsFormat(BigQueryAnalyticsFormatMode.Ignore)]
class DataFragment { }

You can use the [BigQueryAnalyticsName(...)] attribute to override the name of an event when you want it to be different from the C# class:

// Default: this event will have name "MyEvent" in BigQuery analytics.
[AnalyticsEvent(PlayerEventCodes...)]
public class MyEvent : PlayerEventBase
{
    // ... members ...
}

// Overridden event name: this event will have name "my_other_event" in BigQuery analytics.
[BigQueryAnalyticsName("my_other_event")]
[AnalyticsEvent(PlayerEventCodes...)]
public class MyOtherEvent : PlayerEventBase
{
    // ... members ...
}

You can use the same attribute for overriding the name of an event parameter, which would otherwise get its name from the C# member:

[AnalyticsEvent(PlayerEventCodes...)]
public class MyEvent : PlayerEventBase
{
    // Default: this member will appear as "MyMember" in the BigQuery analytics event.
    [MetaMember(1)]
    public string MyMember { get; private set; }

    // Overridden parameter name: this member will appear "my_other_member" in the BigQuery analytics event.
    [BigQueryAnalyticsName("my_other_member")]
    [MetaMember(2)]
    public int MyOtherMember { get; private set; }

    // ...
}

Example Event Generator

Due to the complexity of the key-value mapping, predicting the exact data format for a Event can be difficult. To help with inspecting and validating the mappings, Metaplay LiveOps Dashboard ships with a example event generator. By selecting the desired Analytics Event on the Analytics Events page, an example JSON-formatted BigQuery Event can be inspected. The tool is only intended for inspecting the data format; the example contents in each field are mock values.

Appendices

• BigQuery Table Schema:

  )

  [
      {
          "name": "source_id",
          "type": "STRING",
          "mode": "REQUIRED",
          "description": "EntityId of the source entity. For players, this is for example Player:0a23456789. Can be other entities as well, like Guild:XXX."
      },
      {
          "name": "event_id",
          "type": "STRING",
          "mode": "REQUIRED",
          "description": "An unique ID of the event."
      },
      {
          "name": "event_timestamp",
          "type": "TIMESTAMP",
          "mode": "REQUIRED",
          "description": "Timestamp of the event."
      },
      {
          "name": "event_name",
          "type": "STRING",
          "mode": "REQUIRED",
          "description": "Name of the event. The name of the analytics event type in server code."
      },
      {
          "name": "event_schema_version",
          "type": "INTEGER",
          "mode": "REQUIRED",
          "description": "Schema version of the event parameter data. Developer bumps this if the event class in code changes shape."
      },
      {
          "name": "event_params",
          "type": "RECORD",
          "mode": "REPEATED",
          "description": "Event-specific key-to-value mapping",
          "fields":
          [
              {
                  "name": "key",
                  "type": "STRING",
                  "mode": "REQUIRED",
                  "description": "Name of the event param."
              },
              {
                  "name": "string_value",
                  "type": "STRING",
                  "mode": "NULLABLE",
                  "description": "String value of the event param if param is string."
              },
              {
                  "name": "int_value",
                  "type": "INTEGER",
                  "mode": "NULLABLE",
                  "description": "Integer value of the event param if param is integer."
              },
              {
                  "name": "double_value",
                  "type": "FLOAT",
                  "mode": "NULLABLE",
                  "description": "Floating point value of the event param if param is floating point."
              }
          ]
      },
      {
          "name": "player",
          "type": "RECORD",
          "mode": "NULLABLE",
          "description": "The context of the player entity. Null for non-player events.",
          "fields":
          [
              {
                  "name": "session_number",
                  "type": "INTEGER",
                  "mode": "NULLABLE",
                  "description": "The (nullable) session number of the player. For the first session of the player, this will be 1, and increase by one for each subsequent session. If there is no session, the value is null."
              },
              {
                  "name": "experiments",
                  "type": "RECORD",
                  "mode": "REPEATED",
                  "description": "The active experiments of the player. Empty list for non-player events.",
                  "fields":
                  [
                      {
                          "name": "experiment",
                          "type": "STRING",
                          "mode": "REQUIRED",
                          "description": "The Analytics ID of the experiment."
                      },
                      {
                          "name": "variant",
                          "type": "STRING",
                          "mode": "REQUIRED",
                          "description": "The Analytics ID of the experiment variant."
                      }
                  ]
              }
          ]
      },
      {
          "name": "labels",
          "type": "RECORD",
          "mode": "REPEATED",
          "description": "The custom labels assigned by the source entity.",
          "fields":
          [
              {
                  "name": "name",
                  "type": "STRING",
                  "mode": "REQUIRED",
                  "description": "The name of the custom label."
              },
              {
                  "name": "string_value",
                  "type": "STRING",
                  "mode": "NULLABLE",
                  "description": "The string value of the custom label."
              }
          ]
      }
  ]

• Example dump for a simple session:

  A player logs in, purchases an item and logs out. Player is in a running experiment, and we can see the experiment metadata events.

  {
    "source_id": "GlobalStateManager:0000000000",
    "event_id": "0000017C2E005FC46B684835D11C663D",
    "event_timestamp": "2021-09-28 20:02:28.932000",
    "event_name": "ServerEventExperimentInfo",
    "event_schema_version": 1,
    "event_params": [
      {
        "key": "ExperimentId",
        "string_value": "EarlyGameFunnel"
      },
      {
        "key": "ExperimentAnalyticsId",
        "string_value": "_egf"
      },
      {
        "key": "IsActive",
        "int_value": 1
      },
      {
        "key": "IsRollingOut",
        "int_value": 0
      },
      {
        "key": "DisplayName",
        "string_value": "Faster early game funnel"
      },
      {
        "key": "Description",
        "string_value": "If we tweak the early game funnel, then we can discover which settings work best to retain players, because players will either prefer a slower or faster early game experience."
      }
    ]
  }
  {
    "source_id": "GlobalStateManager:0000000000",
    "event_id": "0000017C2E005FC6C9BB324455A432B9",
    "event_timestamp": "2021-09-28 20:02:28.934000",
    "event_name": "ServerEventExperimentVariantInfo",
    "event_schema_version": 1,
    "event_params": [
      {
        "key": "ExperimentId",
        "string_value": "EarlyGameFunnel"
      },
      {
        "key": "ExperimentAnalyticsId",
        "string_value": "_egf"
      },
      {
        "key": "VariantId",
        "string_value": "Fast"
      },
      {
        "key": "VariantAnalyticsId",
        "string_value": "_fast"
      }
    ]
  }
  {
    "source_id": "GlobalStateManager:0000000000",
    "event_id": "0000017C2E005FC616FC1F5DD6A9CA26",
    "event_timestamp": "2021-09-28 20:02:28.934000",
    "event_name": "ServerEventExperimentVariantInfo",
    "event_schema_version": 1,
    "event_params": [
      {
        "key": "ExperimentId",
        "string_value": "EarlyGameFunnel"
      },
      {
        "key": "ExperimentAnalyticsId",
        "string_value": "_egf"
      },
      {
        "key": "VariantId",
        "string_value": "Slow"
      },
      {
        "key": "VariantAnalyticsId",
        "string_value": "_slow"
      }
    ]
  }
  {
    "source_id": "Player:X9Pq78AMqb",
    "event_id": "0000017C2E0783BA084AD329DCEBCAAD",
    "event_timestamp": "2021-09-28 20:10:16.834000",
    "event_name": "PlayerEventClientConnected",
    "event_schema_version": 1,
    "event_params": [
      {
        "key": "SessionToken",
        "string_value": "EECBC270F71FA6F6"
      },
      {
        "key": "DeviceId",
        "string_value": "Vr0296eTOgX0JmagTlllBWseGIdtRgp5cCiNEvT2RVL8JQCk"
      },
      {
        "key": "DeviceModel",
        "string_value": "Precision 7540 (Dell Inc.)"
      },
      {
        "key": "LogicVersion",
        "int_value": 5
      },
      {
        "key": "TimeZoneInfo:CurrentUtcOffset:Milliseconds",
        "int_value": 10800000
      },
      {
        "key": "Location:Country:IsoCode",
        "string_value": null
      },
      {
        "key": "ClientVersion",
        "string_value": "0.1"
      }
    ],
    "player": {
      "session_number": 6,
      "experiments": [
        {
          "experiment": "_egf",
          "variant": "_fast"
        }
      ]
    }
  }
  {
    "source_id": "Player:X9Pq78AMqb",
    "event_id": "0000017C2E07A0F2A8C477A5C9C46B0E",
    "event_timestamp": "2021-09-28 20:10:24.134000",
    "event_name": "PlayerEventPendingStaticPurchaseContextAssigned",
    "event_schema_version": 1,
    "event_params": [
      {
        "key": "ProductId",
        "string_value": "GemPackLarge"
      },
      {
        "key": "DeviceId",
        "string_value": "Vr0296eTOgX0JmagTlllBWseGIdtRgp5cCiNEvT2RVL8JQCk"
      },
      {
        "key": "GameProductAnalyticsId",
        "string_value": "GemPackLarge"
      },
      {
        "key": "PurchaseContext:$t",
        "string_value": "IdlerPurchaseAnalyticsContext"
      },
      {
        "key": "PurchaseContext:Placement",
        "string_value": "ShopTab"
      },
      {
        "key": "PurchaseContext:Group",
        "string_value": "Basic"
      }
    ],
    "player": {
      "session_number": 6,
      "experiments": [
        {
          "experiment": "_egf",
          "variant": "_fast"
        }
      ]
    }
  }
  {
    "source_id": "Player:X9Pq78AMqb",
    "event_id": "0000017C2E07A5436E376BBF50A37C96",
    "event_timestamp": "2021-09-28 20:10:25.234000",
    "event_name": "PlayerEventInAppValidationStarted",
    "event_schema_version": 1,
    "event_params": [
      {
        "key": "ProductId",
        "string_value": "GemPackLarge"
      },
      {
        "key": "Platform",
        "string_value": "Development"
      },
      {
        "key": "TransactionId",
        "string_value": "fakeTxn449467"
      },
      {
        "key": "PlatformProductId",
        "string_value": "dev.GemPackLarge"
      },
      {
        "key": "ReferencePrice",
        "double_value": 9.989999999990687
      },
      {
        "key": "GameProductAnalyticsId",
        "string_value": "GemPackLarge"
      },
      {
        "key": "PurchaseContext:$t",
        "string_value": "IdlerPurchaseAnalyticsContext"
      },
      {
        "key": "PurchaseContext:Placement",
        "string_value": "ShopTab"
      },
      {
        "key": "PurchaseContext:Group",
        "string_value": "Basic"
      }
    ],
    "player": {
      "session_number": 6,
      "experiments": [
        {
          "experiment": "_egf",
          "variant": "_fast"
        }
      ]
    }
  }
  {
    "source_id": "Player:X9Pq78AMqb",
    "event_id": "0000017C2E07A95272F8B44CF259FE0C",
    "event_timestamp": "2021-09-28 20:10:25.234000",
    "event_name": "PlayerEventInAppValidationComplete",
    "event_schema_version": 1,
    "event_params": [
      {
        "key": "Result",
        "string_value": "Valid"
      },
      {
        "key": "ProductId",
        "string_value": "GemPackLarge"
      },
      {
        "key": "Platform",
        "string_value": "Development"
      },
      {
        "key": "TransactionId",
        "string_value": "fakeTxn449467"
      },
      {
        "key": "PlatformProductId",
        "string_value": "dev.GemPackLarge"
      },
      {
        "key": "ReferencePrice",
        "double_value": 9.989999999990687
      },
      {
        "key": "GameProductAnalyticsId",
        "string_value": "GemPackLarge"
      },
      {
        "key": "PurchaseContext:$t",
        "string_value": "IdlerPurchaseAnalyticsContext"
      },
      {
        "key": "PurchaseContext:Placement",
        "string_value": "ShopTab"
      },
      {
        "key": "PurchaseContext:Group",
        "string_value": "Basic"
      }
    ],
    "player": {
      "session_number": 6,
      "experiments": [
        {
          "experiment": "_egf",
          "variant": "_fast"
        }
      ]
    }
  }
  {
    "source_id": "Player:X9Pq78AMqb",
    "event_id": "0000017C2E07A98B966982742D5AD08E",
    "event_timestamp": "2021-09-28 20:10:26.334000",
    "event_name": "PlayerEventInAppPurchased",
    "event_schema_version": 2,
    "event_params": [
      {
        "key": "ProductId",
        "string_value": "GemPackLarge"
      },
      {
        "key": "Platform",
        "string_value": "Development"
      },
      {
        "key": "TransactionId",
        "string_value": "fakeTxn449467"
      },
      {
        "key": "PlatformProductId",
        "string_value": "dev.GemPackLarge"
      },
      {
        "key": "ReferencePrice",
        "double_value": 9.989999999990687
      },
      {
        "key": "GameProductAnalyticsId",
        "string_value": "GemPackLarge"
      },
      {
        "key": "PurchaseContext:$t",
        "string_value": "IdlerPurchaseAnalyticsContext"
      },
      {
        "key": "PurchaseContext:Placement",
        "string_value": "ShopTab"
      },
      {
        "key": "PurchaseContext:Group",
        "string_value": "Basic"
      },
      {
        "key": "ResolvedContent:$t",
        "string_value": "ResolvedPurchaseGameContent"
      },
      {
        "key": "ResolvedContent:NumGems",
        "int_value": 100
      },
      {
        "key": "ResolvedContent:NumGold",
        "int_value": 10000
      }
    ],
    "player": {
      "session_number": 6,
      "experiments": [
        {
          "experiment": "_egf",
          "variant": "_fast"
        }
      ]
    }
  }
  {
    "source_id": "Player:X9Pq78AMqb",
    "event_id": "0000017C2E081B3B8CC57015FAFBB917",
    "event_timestamp": "2021-09-28 20:10:26.334000",
    "event_name": "PlayerEventClientDisconnected",
    "event_schema_version": 1,
    "event_params": [
      {
        "key": "SessionToken",
        "string_value": "EECBC270F71FA6F6"
      }
    ],
    "player": {
      "session_number": 6,
      "experiments": [
        {
          "experiment": "_egf",
          "variant": "_fast"
        }
      ]
    }
  }