Appearance
Skip to content This document describes the data format used in Analytics Events that are written with the Built-in Analytics sink.
Overview
When Metaplay Analytics Events are exported via BigQuery Analytics Sink into a BigQuery Table, the event format is modified to roughly emulate the native export format of Firebase Analytics. The intention is to make integration easy with Firebase-aligned tooling. As with Firebase, Metaplay writes each Analytics Event as a single row in the BigQuery Table and the values in a AnalyticsEventEnvelope are exported as follows:
Sourceis mapped tosource_id.UniqueIdis mapped toevent_id. This value is also used as the BigQuery deduplication key if native BigQuery deduplication is enabled.ModelTimeis mapped toevent_timestamp.EventTypeis mapped toevent_nameSchemaVersionis mapped toevent_schema_versionPayloadis flattened intoevent_paramsas key-value pairs. The process is described in more detail below.Contextis mapped to the proper source-specific fields as defined in the schema. Currently only Player-entities have a context. The player context is exported as follows:sessionNumberis mapped intoplayer.session_number- active
experimentsare mapped intoplayer.experimentslist where each experiment is represented by{ "experiment": analyticsId, "variant": analyticsId }pair.
Labelsis mapped to a list where each element is represented by a{ "name": name, "string_value": value }pair.
Example event:
C#
{
"source_id": "Player:123",
"event_id": "11223344AABBCCDD66778899EEFF0011",
"event_timestamp": "2020-10-10 14:40:12.000000",
"event_name": "PlayerEventClientDisconnected",
"event_schema_version": 1,
"event_params": [
{
"key": "SessionToken",
"string_value": "0000000000BC614E"
}
],
"player": {
"session_number": 4,
"experiments": [
{
"experiment": "experimentId",
"variant": "variantId"
}
]
},
"labels": [
{
"name": "Location",
"string_value": "FI"
}
]
}Payload Representation and Flattening
All Event Payload fields are represented as key-value pairs. The key is always a string and the value is either an integer, a floating point value, a string value, or a null value. This is represented as struct of {"key": key, "string_value": value, "int_value": value, "double_value": value }. When exported, a single field will have only a single *_value field set, or none in the case of a null value.
When an Event is exported, each field in the Payload object is mapped to a single key-value pair in the exported row. For flat Payload objects, such as the example PlayerEventClientDisconnected, each field is trivially mapped to a key-value pair without any transformation. In the case of the example, the SessionToken field is mapped to a SessionToken key. But if the Payload contains either (1) Aggregate types, (2) Array types, or (3) Dynamic Types, a the object tree is walked and flattened to key-value pairs as follows:
- Aggregate types, i.e. fields that contain multiple fields, are represented as multiple key-value pairs where each subfield uses a name of
<aggregate_field_name>:<subfield_name>. For example, aPair pairfield where Pair isstruct Pair { int A, int B }would be represented aspair:Aandpair:Bint-valued key-value pairs. If the field is null, it is interpreted as if all subfields were null. - Array types are represented as their elements where each element use a name of
<array_field_name>:<array_index>. For example,int[] valuewould be represented asvalue:0,value:1, ... fields. - Dynamic types, i.e. field has is declared with a base type and the contents depend on the runtime value of the field, are represented like Aggregate Types but with an additional synthetic
$tsubfield. The value$tstring is the runtime type of the field. For example, aBasetype fieldwith a value ofclass ConcreteType : BaseType { int A }would be exported asfield:$t = "ConcreteType"andfield:Asubfields. A null value is interpreted as if there are no field.
The rules are applied recursively, and single top-level field the use of all of the rules. For example an single MetaReward field named as ResolvedContent might expand to:
C#
{
"key": "ResolvedContent:$t",
"string_value": "ResolvedPurchaseMetaRewards"
},
{
"key": "ResolvedContent:Rewards:0:$t",
"string_value": "RewardGems"
},
{
"key": "ResolvedContent:Rewards:0:Amount",
"int_value": 123
},
{
"key": "ResolvedContent:Rewards:1:$t",
"string_value": "RewardGold"
},
{
"key": "ResolvedContent:Rewards:1:Amount",
"int_value": 123
},The flattening of individual types and members can be controlled with a BigQueryAnalyticsFormat attribute which allows choosing the flattening mode. Currently, the only custom mode is Ignore, which removes the annotated Type or Member from the BigQuery flattening result. This can be used as follows:
C#
class Event
{
//...
// this field is not exported via BigQuery
[BigQueryAnalyticsFormat(BigQueryAnalyticsFormatMode.Ignore)]
public int Field;
// this field is also not exported via BigQuery
public DataFragment fragment;
}
[BigQueryAnalyticsFormat(BigQueryAnalyticsFormatMode.Ignore)]
class DataFragment { }You can use the [BigQueryAnalyticsName(...)] attribute to override the name of an event when you want it to be different from the C# class:
C#
// Default: this event will have name "MyEvent" in BigQuery analytics.
[AnalyticsEvent(PlayerEventCodes...)]
public class MyEvent : PlayerEventBase
{
// ... members ...
}
// Overridden event name: this event will have name "my_other_event" in BigQuery analytics.
[BigQueryAnalyticsName("my_other_event")]
[AnalyticsEvent(PlayerEventCodes...)]
public class MyOtherEvent : PlayerEventBase
{
// ... members ...
}You can use the same attribute for overriding the name of an event parameter, which would otherwise get its name from the C# member:
C#
[AnalyticsEvent(PlayerEventCodes...)]
public class MyEvent : PlayerEventBase
{
// Default: this member will appear as "MyMember" in the BigQuery analytics event.
[MetaMember(1)]
public string MyMember { get; private set; }
// Overridden parameter name: this member will appear "my_other_member" in the BigQuery analytics event.
[BigQueryAnalyticsName("my_other_member")]
[MetaMember(2)]
public int MyOtherMember { get; private set; }
// ...
}Example Event Generator
Due to the complexity of the key-value mapping, predicting the exact data format for a Event can be difficult. To help with inspecting and validating the mappings, Metaplay LiveOps Dashboard ships with a example event generator. By selecting the desired Analytics Event on the Analytics Events page, an example JSON-formatted BigQuery Event can be inspected. The tool is only intended for inspecting the data format; the example contents in each field are mock values.
Appendices
BigQuery Table Schema:
)C#[ { "name": "source_id", "type": "STRING", "mode": "REQUIRED", "description": "EntityId of the source entity. For players, this is for example Player:0a23456789. Can be other entities as well, like Guild:XXX." }, { "name": "event_id", "type": "STRING", "mode": "REQUIRED", "description": "An unique ID of the event." }, { "name": "event_timestamp", "type": "TIMESTAMP", "mode": "REQUIRED", "description": "Timestamp of the event." }, { "name": "event_name", "type": "STRING", "mode": "REQUIRED", "description": "Name of the event. The name of the analytics event type in server code." }, { "name": "event_schema_version", "type": "INTEGER", "mode": "REQUIRED", "description": "Schema version of the event parameter data. Developer bumps this if the event class in code changes shape." }, { "name": "event_params", "type": "RECORD", "mode": "REPEATED", "description": "Event-specific key-to-value mapping", "fields": [ { "name": "key", "type": "STRING", "mode": "REQUIRED", "description": "Name of the event param." }, { "name": "string_value", "type": "STRING", "mode": "NULLABLE", "description": "String value of the event param if param is string." }, { "name": "int_value", "type": "INTEGER", "mode": "NULLABLE", "description": "Integer value of the event param if param is integer." }, { "name": "double_value", "type": "FLOAT", "mode": "NULLABLE", "description": "Floating point value of the event param if param is floating point." } ] }, { "name": "player", "type": "RECORD", "mode": "NULLABLE", "description": "The context of the player entity. Null for non-player events.", "fields": [ { "name": "session_number", "type": "INTEGER", "mode": "NULLABLE", "description": "The (nullable) session number of the player. For the first session of the player, this will be 1, and increase by one for each subsequent session. If there is no session, the value is null." }, { "name": "experiments", "type": "RECORD", "mode": "REPEATED", "description": "The active experiments of the player. Empty list for non-player events.", "fields": [ { "name": "experiment", "type": "STRING", "mode": "REQUIRED", "description": "The Analytics ID of the experiment." }, { "name": "variant", "type": "STRING", "mode": "REQUIRED", "description": "The Analytics ID of the experiment variant." } ] } ] }, { "name": "labels", "type": "RECORD", "mode": "REPEATED", "description": "The custom labels assigned by the source entity.", "fields": [ { "name": "name", "type": "STRING", "mode": "REQUIRED", "description": "The name of the custom label." }, { "name": "string_value", "type": "STRING", "mode": "NULLABLE", "description": "The string value of the custom label." } ] } ]Example dump for a simple session:
A player logs in, purchases an item and logs out. Player is in a running experiment, and we can see the experiment metadata events.
C#{ "source_id": "GlobalStateManager:0000000000", "event_id": "0000017C2E005FC46B684835D11C663D", "event_timestamp": "2021-09-28 20:02:28.932000", "event_name": "ServerEventExperimentInfo", "event_schema_version": 1, "event_params": [ { "key": "ExperimentId", "string_value": "EarlyGameFunnel" }, { "key": "ExperimentAnalyticsId", "string_value": "_egf" }, { "key": "IsActive", "int_value": 1 }, { "key": "IsRollingOut", "int_value": 0 }, { "key": "DisplayName", "string_value": "Faster early game funnel" }, { "key": "Description", "string_value": "If we tweak the early game funnel, then we can discover which settings work best to retain players, because players will either prefer a slower or faster early game experience." } ] } { "source_id": "GlobalStateManager:0000000000", "event_id": "0000017C2E005FC6C9BB324455A432B9", "event_timestamp": "2021-09-28 20:02:28.934000", "event_name": "ServerEventExperimentVariantInfo", "event_schema_version": 1, "event_params": [ { "key": "ExperimentId", "string_value": "EarlyGameFunnel" }, { "key": "ExperimentAnalyticsId", "string_value": "_egf" }, { "key": "VariantId", "string_value": "Fast" }, { "key": "VariantAnalyticsId", "string_value": "_fast" } ] } { "source_id": "GlobalStateManager:0000000000", "event_id": "0000017C2E005FC616FC1F5DD6A9CA26", "event_timestamp": "2021-09-28 20:02:28.934000", "event_name": "ServerEventExperimentVariantInfo", "event_schema_version": 1, "event_params": [ { "key": "ExperimentId", "string_value": "EarlyGameFunnel" }, { "key": "ExperimentAnalyticsId", "string_value": "_egf" }, { "key": "VariantId", "string_value": "Slow" }, { "key": "VariantAnalyticsId", "string_value": "_slow" } ] } { "source_id": "Player:X9Pq78AMqb", "event_id": "0000017C2E0783BA084AD329DCEBCAAD", "event_timestamp": "2021-09-28 20:10:16.834000", "event_name": "PlayerEventClientConnected", "event_schema_version": 1, "event_params": [ { "key": "SessionToken", "string_value": "EECBC270F71FA6F6" }, { "key": "DeviceId", "string_value": "Vr0296eTOgX0JmagTlllBWseGIdtRgp5cCiNEvT2RVL8JQCk" }, { "key": "DeviceModel", "string_value": "Precision 7540 (Dell Inc.)" }, { "key": "LogicVersion", "int_value": 5 }, { "key": "TimeZoneInfo:CurrentUtcOffset:Milliseconds", "int_value": 10800000 }, { "key": "Location:Country:IsoCode", "string_value": null }, { "key": "ClientVersion", "string_value": "0.1" } ], "player": { "session_number": 6, "experiments": [ { "experiment": "_egf", "variant": "_fast" } ] } } { "source_id": "Player:X9Pq78AMqb", "event_id": "0000017C2E07A0F2A8C477A5C9C46B0E", "event_timestamp": "2021-09-28 20:10:24.134000", "event_name": "PlayerEventPendingStaticPurchaseContextAssigned", "event_schema_version": 1, "event_params": [ { "key": "ProductId", "string_value": "GemPackLarge" }, { "key": "DeviceId", "string_value": "Vr0296eTOgX0JmagTlllBWseGIdtRgp5cCiNEvT2RVL8JQCk" }, { "key": "GameProductAnalyticsId", "string_value": "GemPackLarge" }, { "key": "PurchaseContext:$t", "string_value": "IdlerPurchaseAnalyticsContext" }, { "key": "PurchaseContext:Placement", "string_value": "ShopTab" }, { "key": "PurchaseContext:Group", "string_value": "Basic" } ], "player": { "session_number": 6, "experiments": [ { "experiment": "_egf", "variant": "_fast" } ] } } { "source_id": "Player:X9Pq78AMqb", "event_id": "0000017C2E07A5436E376BBF50A37C96", "event_timestamp": "2021-09-28 20:10:25.234000", "event_name": "PlayerEventInAppValidationStarted", "event_schema_version": 1, "event_params": [ { "key": "ProductId", "string_value": "GemPackLarge" }, { "key": "Platform", "string_value": "Development" }, { "key": "TransactionId", "string_value": "fakeTxn449467" }, { "key": "PlatformProductId", "string_value": "dev.GemPackLarge" }, { "key": "ReferencePrice", "double_value": 9.989999999990687 }, { "key": "GameProductAnalyticsId", "string_value": "GemPackLarge" }, { "key": "PurchaseContext:$t", "string_value": "IdlerPurchaseAnalyticsContext" }, { "key": "PurchaseContext:Placement", "string_value": "ShopTab" }, { "key": "PurchaseContext:Group", "string_value": "Basic" } ], "player": { "session_number": 6, "experiments": [ { "experiment": "_egf", "variant": "_fast" } ] } } { "source_id": "Player:X9Pq78AMqb", "event_id": "0000017C2E07A95272F8B44CF259FE0C", "event_timestamp": "2021-09-28 20:10:25.234000", "event_name": "PlayerEventInAppValidationComplete", "event_schema_version": 1, "event_params": [ { "key": "Result", "string_value": "Valid" }, { "key": "ProductId", "string_value": "GemPackLarge" }, { "key": "Platform", "string_value": "Development" }, { "key": "TransactionId", "string_value": "fakeTxn449467" }, { "key": "PlatformProductId", "string_value": "dev.GemPackLarge" }, { "key": "ReferencePrice", "double_value": 9.989999999990687 }, { "key": "GameProductAnalyticsId", "string_value": "GemPackLarge" }, { "key": "PurchaseContext:$t", "string_value": "IdlerPurchaseAnalyticsContext" }, { "key": "PurchaseContext:Placement", "string_value": "ShopTab" }, { "key": "PurchaseContext:Group", "string_value": "Basic" } ], "player": { "session_number": 6, "experiments": [ { "experiment": "_egf", "variant": "_fast" } ] } } { "source_id": "Player:X9Pq78AMqb", "event_id": "0000017C2E07A98B966982742D5AD08E", "event_timestamp": "2021-09-28 20:10:26.334000", "event_name": "PlayerEventInAppPurchased", "event_schema_version": 2, "event_params": [ { "key": "ProductId", "string_value": "GemPackLarge" }, { "key": "Platform", "string_value": "Development" }, { "key": "TransactionId", "string_value": "fakeTxn449467" }, { "key": "PlatformProductId", "string_value": "dev.GemPackLarge" }, { "key": "ReferencePrice", "double_value": 9.989999999990687 }, { "key": "GameProductAnalyticsId", "string_value": "GemPackLarge" }, { "key": "PurchaseContext:$t", "string_value": "IdlerPurchaseAnalyticsContext" }, { "key": "PurchaseContext:Placement", "string_value": "ShopTab" }, { "key": "PurchaseContext:Group", "string_value": "Basic" }, { "key": "ResolvedContent:$t", "string_value": "ResolvedPurchaseGameContent" }, { "key": "ResolvedContent:NumGems", "int_value": 100 }, { "key": "ResolvedContent:NumGold", "int_value": 10000 } ], "player": { "session_number": 6, "experiments": [ { "experiment": "_egf", "variant": "_fast" } ] } } { "source_id": "Player:X9Pq78AMqb", "event_id": "0000017C2E081B3B8CC57015FAFBB917", "event_timestamp": "2021-09-28 20:10:26.334000", "event_name": "PlayerEventClientDisconnected", "event_schema_version": 1, "event_params": [ { "key": "SessionToken", "string_value": "EECBC270F71FA6F6" } ], "player": { "session_number": 6, "experiments": [ { "experiment": "_egf", "variant": "_fast" } ] } }