Dependencies

• You should have game configs implemented - Look at Introduction to Game Configs for an introduction, and Working with Game Config Data to get an in-depth understanding.

Key Concepts

• An Experiment (sometimes known as an A/B Test or a Split Test) is a way to test different Variants on an Audience with the intention of finding out which Variant performs the best.
• A Variant is a patch to a game config that changes some of the config's variables. An Experiment may involve one or more Variants.
• The Audience describes a section of your player base. This is defined as a percentage of all players (eg: "50% of all of your players") and/or using Segments (eg: "all players from Finland").
• The Control Group is a section of your Audience who are served the base game config instead of a Variant. This gives you a baseline to compare your Variants against.
• A Tester is a player specifically designated to be able to try out an Experiment before it is enabled for all players.

Overview

Experiments allow you to test different game configs on different groups of players. Each Experiment contains one or more Variants and each Variant contains a set of alternative values to the base game configs.

Players are enrolled into active Experiments if they meet the Audience criteria. They are assigned into either the Control Group or to one of the Variants. This assignment is random, but you can control the relative distribution of players in each group using weights.

Each player model retains a full history of the Experiments that they have participated in. Players generate analytics events when joining an Experiment and this information is also included in all analytic events that the player generates. You can use this information in your analytics pipeline to later study the efficacy of your Variants.

Introduction to Audiences

A player is eligible to be enrolled in an Experiment if they meet the Audience criteria. There are a few different ways to define an Audience for each Experiment, and they can be used together to make a powerful filter for selecting the perfect group of players:

• Rollout Enabled determines whether players will be allowed to join the Experiment while it is active. This is set to true by default, meaning that players will join the Experiment while the rest of the rollout criteria are met. You can switch this to false to easily manually stop rollout without, for instance, entering a new rollout percentage or capacity limit.
• Rollout percentage lets you decide what portion of your player base is eligible to join the Experiment.
• A capacity limit can be set to cap the total number of players allowed to join the Experiment.
• Targeting allows you to specify which segments the players must belong to. You can reference as many segments as you like. Note that if a player leaves a segment after being enrolled then they will not be removed from the Experiment. For more information on segments see Implementing Player Segments.
• Account age lets you choose between all or just new players. ”New players” are players who are starting your game for the first time. This is great if you want to test things like early game progression, tutorials or balance a player's initial resource allocation without disrupting older players who have already played the game for a bit.

Introduction to the Lifecycle of Experiments

Every Experiment has a lifecycle and, normally, an Experiment will progress forwards one phase at a time through this lifecycle. We'll cover this in more detail later on (including reasons why you might want to take a backward step) but for now, it helps to have a general idea of the phases that an Experiment passes through:

The lifecycle as visualized on the LiveOps Dashboard

1. Testing. The Experiment has been created but is not yet active. It is only visible to players who have been set as a Tester for this experiment. This is a chance for you to test out the Variants on a few hand-picked players (for example, your QA team) before rolling them out to the wider player base.
2. Active. The Experiment is now live! Players will be enrolled into the Experiment if they meet the audience criteria. Once they are enrolled they will start seeing their Variants.
3. Paused: The Experiment is temporarily suspended for non-Tester players. Players assigned into this experiment do not see their Variant Game Config but instead see unmodified Game Config, as if they were in the Control Group. No new players are enrolled in this phase. Note that this phase is not shown in the diagram above, but replaces the Active phase.
4. Concluded. The Experiment is over. It's time to look at the results and draw some conclusions!

Creating an Experiment

Choose a Hypothesis

The first thing to do before starting to build an Experiment is to decide what you want to test, how you will test it, and how to interpret the results. A great way to do this is to develop a solid hypothesis that clearly states what you think the outcome of the change will be. A good hypothesis should answer the question, "What are we hoping to learn from this Experiment?"

INFO

Remember: Your Experiment may not always give the expected outcome, but you should always try to construct it in such a way that you can learn something from the results, regardless of the outcome.

Using the Metaplay Idler reference game, imagine a situation where players are quickly bouncing from your game and you believe that this is because the game progression is too slow. You might want to see if making the early-game progression faster will improve the early player retention rate. An example hypothesis for this might be: "If we make the first three producers cheaper, then the day one retention rates will increase because players enjoy progressing further". Based on this hypothesis, we know what to change and what metrics to track to conduct the Experiment.

INFO

Plan Your Experiments: You could use another framework for defining your Experiments, or you could use no framework at all. But whatever you do, make sure that you understand how to interpret the results before you start!

Decide on your Variants

Once you have a good hypothesis, deciding on your Variants should be easy. Remember that the Variants are modifications from your existing Game Config. In the simplest scenarios, a single Variant should change one thing only, and a set of Variants should change the same thing but in different ways.

For example, in the above scenario we might define a single Variant:

1. Fast Early-Game Producers: The first three Producers are 50% cheaper to unlock and upgrade.

Or we might want to go further and produce two Variants:

1. Fast Early-Game Producers: The first three Producers are 50% cheaper to unlock and upgrade.
2. Slow Early-Game Producers: The first three Producers are 100% more expensive to unlock and upgrade.

INFO

About Multiple Variants: While an Experiment in the Metaplay SDK can have more than one Variant, it's often a good idea to stick with just one - it's much easier to get to statistical significance and analyze the attribution of your results when you make fewer changes at once.

Create Your Experiment in the Game Config

Now that we have decided on our hypothesis and our Variants, we can move onto defining them in our Game Config. As an example, we will use the hypothesis that we developed earlier on for testing early-game progression in Idler. The Game Config in Idler is defined using Google Sheets and, in general, the Experiment data would be entered into the sheets by a game designer.

First, we'll set up our Experiment. In Idler's config sheet, we'll add an entry for our new Experiment in the Experiment sheet.

INFO

Good to know: We'll cover Analytics IDs later in this document.

Next, we'll create a Variant. To override a config value, simply add the ExperimentId to the /Variant column and fill in the ConfigKey and whichever values you want to override.

You can modify as many config items and values as you like. Just make sure the Experiment doesn't change too many things at once to keep your data analyst happy!

Note that you can also define the Variant row anywhere in the sheet, as long as you add the ConfigKey correctly.

In case many of your variants share the configuration for a particular row, you can also list multiple variants as a comma-separated list in the /Variant column to reduce the number of rows in your sheet!

The first time you're setting up experiments in your project, remember to define the PlayerExperiments library in ServerGameConfig. That may also be the first time you define the ServerGameConfig class in the first place.

GameConfig.cs

public class ServerGameConfig : ServerGameConfigBase
{
    [GameConfigEntry(PlayerExperimentsEntryName)]
    public GameConfigLibrary<PlayerExperimentId, PlayerExperimentInfo> PlayerExperiments { get; private set; }
}

Now you can build and publish your new game config, as shown in Working with Game Config Data.

The SDK also support defining the variants on the rows directly under the affected item, as such:

WARNING

Heads up: Remember that publishing a new game config with an Experiment doesn't change anything for existing players yet - all it does is defining a new Experiment and Variants for later rollout.

Using Variant Columns

When running experiments involving changing a small number of values of a large number of config rows, it is often more convenient to describe the variant in the spreadsheet by using Variant Columns.

Modifying our previous Experiment scenario, let's say that we'd like to only experiment with the UpgradeCost property but modify the values of all Producers up to Haskell. Our experiment plan would then be:

1. Fast Early-Game Producers: The 6 first Producers are 20% cheaper to upgrade.
2. Slow Early-Game Producers: The 6 first Producers are 20% more expensive to upgrade.

We could naturally declare the variants in the same fashion as before, but now our sheet would contain more variant rows than regular rows and maintaining the sheet would start to become cumbersome. Using Variant Columns instead, our sheet is transformed to look like this:

Variant Columns are declared by the special header column prefix "/:". A Variant Column contains override values for the first regular column on its left, and as in the example multiple variants can be declared for a single property by separate Variant Columns. Similarly to Variant Rows, empty values in the column result in the property for the row to not be overridden.

View Your Experiment in the LiveOps Dashboard

Once your Experiment has been defined in a Game Config and the same Game Config has been published, you can start using the LiveOps Dashboard to interact with it. First, let's navigate to the Experiments page:

Click on "View experiment" to take you through to a page containing more detailed information:

The main part of this page is the overview card, which shows you the current status of this Experiment and lets you control it - we'll cover this in the next section. For now, we can focus on the Variants card:

Now is a great time to check that your Variants are configured the way that you expect them to be. By clicking on "View config diff" you can examine the differences between a Variant and the Game Config that it is based on. If something's not quite right, you can update the Variant configuration in the Game Configs and rebuild. Once you're happy with the data, it's time to move on to running the Experiment.

Running an Experiment

At this point, the Experiment and its Variants have been created and configured correctly by the game's designer - the Experiment is almost ready to go! In this section we'll walk through the process of running an Experiment through to conclusion. This would usually be done by someone in the LiveOps team, but in a smaller team that might well be the same person as the designer of the Experiment.

We introduced the lifecycle model earlier on in this document. Now we'll look at each one of those phases in more detail, and we'll do all of that from the Experiment overview card on the detail page of the LiveOps Dashboard. Remember that the lifecycle looks like:

Testing → Active → Concluded

1. Testing

The Experiment is in the testing phase - it is not visible and has no effect, except for Tester players.

This is a great time to test out your Variants on some hand-picked players - maybe accounts owned by your test department. You can do this by navigating to a player's page and using the "Join Experiment" button in the admin tools section, and enable "Tester" switch.

If testing reveals any problems with the Variants, now is the time to go back to the sheets to get them right. Take as many iterations as you need to get things just right before you roll the Experiment out to your players.

Once you are happy that the Experiment and its Variants work as intended, you can configure the Experiment rollout parameters by clicking on "Configure".

In the "Configure Experiment for Testing" modal, you can configure an Experiment's Rollout Settings and Audience. Together these define how much of your player base will participate in the Experiment. Additionally, you can adjust how the participating players are distributed across the variants.

INFO

Heads up: You can edit the Experiment in any phase, but you probably shouldn't. Once the Experiment is rolled out, making changes may make it hard to understand the analytics data.

Finally, hitting the "Rollout" button will enable for the chosen audience.

2. Active

Now your Experiment is live and players are starting to see the Variants. The "Statistics" panel shows you how many players are in your Experiment. More detailed rollout info is also available in the Variants card.

Once your Experiment has been running long enough to gather all the analytics data that you need, you can hit "Conclude" to end it. If you need to temporarily stop players from seeing an Experiment (maybe you spotted a mistake in one of the Variants?) you can click "Pause".

3. Paused

The Experiment is temporarily suspended. Enrolling new players is disabled, and all non-Tester players will no longer see the Experiment. For the Testers of this Experiment, however, the Experiment is still active.

This is useful for example if an unexpected bug causes a problem in a production environment. If you need to update your Game Config to fix the issue then you can, and only your Tester players will see the changes - as long as the Experiment remains Paused.

Once you have fixed the temporary issue with the Experiment and are satisfied with the updated configuration, you may now click "Continue" to resume the Experiment and go back to the Active phase. Alternatively, you could also click "Conclude" to end the experiment immediately.

4. Concluded

Now the Experiment has finished - it's time to go and look at the results from your analytics pipeline. At this point, all of the players (including the Tester players) in the Experiment will start seeing the base Game Config again.

If you concluded an Experiment by accident, you can use the "Restart" button to put it back into the "Testing" phase, and from there you can roll out the Experiment again.

Measuring Results with Analytics

Players are constantly generating analytic events while they are playing your game. When they are in an experiment, these events are tagged with metadata about the experiment. You can measure the impact of variants on players' behavior by analyzing and comparing them against players in the control group.

Metaplay is not a data analysis platform but instead it allows for integrating with existing 3rd party and custom solutions. For the Experiments, this is enabled by providing the Analytics IDs we defined in the Game Config to the Analytics solution.

To integrate Experiment Analytics Events with your Analytics solution, see Analytics Processing.

Behaviour of Invalid Experiment Configurations

An Experiment has a failure mode, which is triggered if the Experiment Config is changed in an incompatible way while the Experiment is running. The cases are as follows:

• Removed Experiment Config: If the Experiment is removed from Game Config in a non-Concluded phase, the whole Experiment is marked as invalid until the config is reintroduced. An invalid Experiment is not visible to any player, including Testers, and it has no effect.

• Removed Variant Config: If a Variant is removed from the Game Config, the Experiment will remain running but the removed variant is marked as invalid and the system behaves as if the Variant did not exist. This means that players who have been assigned into this Variant will instead see the base Game Config. Additionally, the assignment weight of the Variant behaves as if it were set to 0, preventing any new players from being enrolled into the Variant. The weights of the other Variants will be adjusted accordingly.

  When the Variant is re-introduced into the Config, the effective weights are restored and players in the Variant start seeing the variant config again.

• No Variant Groups: If the Experiment has no valid Variants (i.e. it effectively consists only of a Control Group) then it is marked as invalid and disabled. The Experiment is not visible to any player, and it has no effect. When Variant Groups are reintroduced, the Experiment becomes valid again.

Audit Logs

It's good to know that whenever you make changes to an Experiment through the LiveOps Dashboard, an audit trail is left behind. This makes it easy to track changes over time. The trail for an individual Experiment can be seen in the admin section of each Experiment's detail page:

For more information on audit logs, see LiveOps Dashboard Audit Logs.

Running Multiple Experiments Simultaneously

The Metaplay SDK allows you to define as many Experiments as you like - so what happens if you have more than one running at the same time? The answer is straightforward: if a player is enrolled in multiple experiments, then they will see all of the Variants that they have been assigned from all of the Experiments. For example, a player might be taking part in one Experiment where they see stronger enemies on level one of your game, and another Experiment where they see more powerful weapons. In this case, the player will see both stronger enemies and more powerful weapons.

Whilst this answer is straightforward, the consequences are not. For example, how will you be able to measure the results of the stronger enemies Experiment if some of the players also have more powerful weapons? The answer depends on how much the Experiments overlap, how able you are to extract the relevant data from your analytics system and how well you are able to analyse it.

Another possible pitfall is if a player is in two Experiments that both alter the same variables. For example, one Experiment could make level one enemies stronger while another makes all enemies weaker. If a player is in both Experiments, what will the effect be on level one enemies? Because Variants apply their changes as absolute values rather than relative, the answer will depend on the order that the Variants are applied to the player - the player may have weaker or stronger enemies.

The order in which the changes are applied is the same order as the Experiments are defined in the Game Config - early changes in the Config will always be overwritten by later changes. As this can cause unexpected behavior, we recommend not running parallel experiments on the same values.

Because of these potential downsides, we strongly recommend that you ensure that players can only be enrolled in a single Experiment at a time unless you are confident that the Experiments do not contaminate each other. It might not always be an issue - sometimes it's OK for a player to be in more than one Experiment.

Avoiding Overlapping Experiments

There are a few ways to avoid a player being in multiple Experiments:

1. Only ever run one Experiment at a time. This is the simplest solution and, in many cases, can be a viable solution.
2. Use your segments to target different Experiments to different player groups. You need to be careful that your segments don't affect your results. For example, if you target one Experiment at players in Finland and another at players in France then you may end up unintentionally testing cultural preferences.
3. Use the "Playerbase Subset" segments to define unique, non-overlapping subsets of your player base. For example, if you created two subsets then you could put players from the first subset into one Experiment, and players from the second subset into another. This would guarantee that no players would be in both Experiments. This is the best way to control exclusivity, but it requires some additional set up. See Implementing Player Segments for how to configure this.

Technical Details of Specialized Game Config Delivery

• When a player logs in, the currently active set of Experiment memberships is evaluated on the server and a specialized/patched game config archive is built for this set.
• The player downloads the patched gameconfig (if it has changed) via the regular game config download mechanism.
• Server-side code that might depend on game config values in Experiments can only do so while there is an active session, as the Experiment membership is only evaluated at session start.
• In particular, Experiments involving initial values for a player model should populate the values in PlayerModel.GameOnInitialLogin rather than the PlayerModel constructor, as the construction happens before processing the login and building the specialized config.

INFO

Heads up: Since the server has to keep track of all active game config specializations, having many variants on the same segments can cause increased memory usage. It is therefore recommended to keep an eye on the memory usage statistics when enabling new experiments.

Limitations of Experiments

Unfortunately, not everything in the Game Configs is safe to experiment with. The current limitations include:

1. Activables (for example in-game events and offers) are currently not supported. As a workaround, we recommend creating multiple versions of your activable and using the targeting tools to separate them to parallel Audiences. In practice this creates a natural Experiment.
2. Multivariant Experiments, where two or more parallel Experiments modify the same Game Config values simultaneously, may lead to unpredictable results that contaminate both Experiments. We recommend not running multivariant Experiments.
3. In App Purchases should not be used, because removing an IAP definitions (eg: when the Experiment is concluded) may cause the player to lose the purchase. As a workaround, you could hide the IAP from the game UI if the player is not in the experiment.

Experiment Ideas

There are endless different Experiments that you could run on your game. Here are a few suggestions to show you some of the possibilities:

Name	Hypothesis	Notes	Variants
Early Game Funnel.	If we tweak the early game funnel then we can discover which settings work best to retain players because players will either prefer a harder or easier early game experience.	A simple experiment with two variants.	[33%] Control: No change. [33%] Harder: Change level one enemies to be 50% weaker. [33%] Easier: Change level one enemies to be 100% stronger.
Initial game resource tuning.	If we give new players different initial resource bundles, then we will find an option that increases player retention, because some bundles will help players get more invested in the game.	Four variants this time. You also need to set the Account Age of the Experiment's Audience targeting to New Players, because this only makes sense to apply to players when the first create an account with you.	[20%] Control: No change. [20%] HighGems: InitialGems * 2. [20%] HighGold: InitialGold * 2. [20%] LowGems: InitialGems * 0.5. [20%] LowGold: InitialGold * 0.5.
High price anchoring.	If players see a 100€ purchase in the shop, then they are more likely to purchase items, due to the high priced item making the cheaper items look like better value.	A sneaky psychological hypothesis! But will it work? Running an Experiment is the perfect way to find out.	[50%] Control: No change. [50%] HighPrice: Make a super high priced IAP item visible.
Alternative shop button.	If players see a different icon on the shop button, then they are more likely to purchase items from the shop, because they find the different design to be more discoverable and/or engaging.	You don't have to limit yourselves to changing the stats of your game design or economy. You can also put UI configuration into your Game Config and then you use that in your Experiments. In this case, the name of your shop icon, but you could also store buttons colours, positions, and more.	[33%] Control: No change. [33%] PiggyBankIcon: Use the pig icon in the tab bar. [33%] ArrowUpIcon: Use the arrow icon in the tab bar.
Conversion offer.	If players are presented a really good value conversion bundle, then they might spend more money later, because they are more invested in the game.	You can implement this by changing the contents of an offer in the Experiment.	[33%] Control: No change. [33%] CheapOffers: Show only the low cost conversion offers. [33%] ExpensiveOffers: Show only the high cost conversion offer.
Alternative theme in Finland.	If players in Finland see a more culturally relevant theme, then they will have increased retention, because they are more engaged.	You would define the Audience for this Experiment using a segment that targets players in Finland only. You could then tweak colours, names, icons or whatever you have available in your Game Config.	[50%] Control: No change. [50%] FinnishTheme: Alternative theme.
Alternative happy hour event.	If players see different happy hour events, then we will find out which ones are most popular, because players will engage with the ones that the find most interesting.		[20%] Control: No change. [80%] AlternativeContent: Different producer in discount.

INFO

Here to Help: Hopefully, this document has made you think of some great Experiments of your own, but maybe you're unsure about how to implement them? Talk to us - we're here to help!