Key Concepts

• An Experiment (sometimes known as an A/B Test or a Split Test) is a way to test different Variants on an Audience with the intention of finding out which Variant performs the best.
• A Variant is a patch to a game config that changes some of the config's variables. An Experiment may involve one or more Variants.
• The Audience describes a section of your player base. This is defined as a percentage of all players (e.g., "50% of all of your players") and/or using Segments (e.g., "all players from Finland").
• The Control Group is a section of your Audience who are served the base game config instead of a Variant. This gives you a baseline to compare your Variants against.
• A Tester is a player specifically designated to be able to try out an Experiment before it is enabled for all players.

Overview

Experiments allow you to test different game configs on different groups of players. Each Experiment contains one or more Variants, and each Variant contains a set of alternative values to the base game configs.

Players are enrolled in active Experiments if they meet the Audience criteria. They are assigned to either the Control Group or to one of the Variants. This assignment is random, but you can control the relative distribution of players in each group using weights.

Each player model retains a full history of the Experiments that they have participated in. Players generate analytics events when joining an Experiment, and this information is also included in all analytic events that the player generates. You can use this information in your analytics pipeline to later study the efficacy of your Variants.

Introduction to Audiences

A player is eligible to be enrolled in an Experiment if they meet the Audience criteria. There are a few different ways to define an Audience for each Experiment, and they can be used together to make a powerful filter for selecting the perfect group of players:

• Rollout Enabled determines whether players will be allowed to join the Experiment while it is active. This is set to true by default, meaning that players will join the Experiment while the rest of the rollout criteria are met. You can switch this to false to easily manually stop rollout without, for instance, entering a new rollout percentage or capacity limit.
• Rollout percentage lets you decide what portion of your player base is eligible to join the Experiment.
• A capacity limit can be set to cap the total number of players allowed to join the Experiment.
• Targeting allows you to specify which segments the players must belong to. You can reference as many segments as you like. Note that if a player leaves a segment after being enrolled, then they will not be removed from the Experiment. For more information on segments, see Implementing Player Segments.
• Account age lets you choose between all or just new players. “New players” are players who are starting your game for the first time. This is great if you want to test things like early game progression, tutorials, or balance a player's initial resource allocation without disrupting older players who have already played the game for a bit.

Introduction to the Lifecycle of Experiments

Every Experiment has a lifecycle, and normally, an Experiment will progress forward one phase at a time through this lifecycle. We'll cover this in more detail later on (including reasons why you might want to take a backward step), but for now, it helps to have a general idea of the phases that an Experiment passes through:

The lifecycle as visualized on the LiveOps Dashboard

1. Testing. The Experiment has been created but is not yet active. It is only visible to players who have been set as a Tester for this experiment. This is a chance for you to test out the Variants on a few hand-picked players (for example, your QA team) before rolling them out to the wider player base.
2. Active. The Experiment is now live! Players will be enrolled in the Experiment if they meet the audience criteria. Once they are enrolled, they will start seeing their Variants.
3. Paused. The Experiment is temporarily suspended for non-Tester players. Players assigned to this experiment do not see their Variant Game Config but instead see unmodified Game Config, as if they were in the Control Group. No new players are enrolled in this phase. Note that this phase is not shown in the diagram above, but replaces the Active phase.
4. Concluded. The Experiment is over. It's time to look at the results and draw some conclusions!

Implementation

Take a look at Implementing Experiments for a step-by-step look into how an experiment is implemented.

Experiment Ideas

There are endless different Experiments that you could run on your game. Here are a few suggestions to show you some of the possibilities:

Name	Hypothesis	Notes	Variants
Early Game Funnel.	If we tweak the early game funnel, then we can discover which settings work best to retain players because players will either prefer a harder or easier early game experience.	A simple experiment with two variants.	[33%] Control: No change. [33%] Harder: Change level one enemies to be 50% weaker. [33%] Easier: Change level one enemies to be 100% stronger.
Initial game resource tuning.	If we give new players different initial resource bundles, then we will find an option that increases player retention because some bundles will help players get more invested in the game.	Four variants this time. You also need to set the Account Age of the Experiment's Audience targeting to New Players because this only makes sense to apply to players when they first create an account with you.	[20%] Control: No change. [20%] HighGems: InitialGems * 2. [20%] HighGold: InitialGold * 2. [20%] LowGems: InitialGems * 0.5. [20%] LowGold: InitialGold * 0.5.
High price anchoring.	If players see a 100€ purchase in the shop, then they are more likely to purchase items due to the high-priced item making the cheaper items look like better value.	A sneaky psychological hypothesis! But will it work? Running an Experiment is the perfect way to find out.	[50%] Control: No change. [50%] HighPrice: Make a super high-priced IAP item visible.
Alternative shop button.	If players see a different icon on the shop button, then they are more likely to purchase items from the shop because they find the different design to be more discoverable and/or engaging.	You don't have to limit yourselves to changing the stats of your game design or economy. You can also put UI configuration into your Game Config, and then you use that in your Experiments. In this case, the name of your shop icon, but you could also store button colors, positions, and more.	[33%] Control: No change. [33%] PiggyBankIcon: Use the pig icon in the tab bar. [33%] ArrowUpIcon: Use the arrow icon in the tab bar.
Conversion offer.	If players are presented with a really good value conversion bundle, then they might spend more money later because they are more invested in the game.	You can implement this by changing the contents of an offer in the Experiment.	[33%] Control: No change. [33%] CheapOffers: Show only the low-cost conversion offers. [33%] ExpensiveOffers: Show only the high-cost conversion offer.
Alternative theme in Finland.	If players in Finland see a more culturally relevant theme, then they will have increased retention because they are more engaged.	You would define the Audience for this Experiment using a segment that targets players in Finland only. You could then tweak colors, names, icons, or whatever you have available in your Game Config.	[50%] Control: No change. [50%] FinnishTheme: Alternative theme.
Alternative happy hour event.	If players see different happy hour events, then we will find out which ones are most popular because players will engage with the ones that they find most interesting.		[20%] Control: No change. [80%] AlternativeContent: Different producer in discount.

🤝 Here to Help

Hopefully, this document has made you think of some great Experiments of your own, but maybe you're unsure about how to implement them? Take a look at Implementing Experiments. If you have further questions, talk to us - we're here to help!

Further Reading

For more information on implementing your experiments, see Implementing Experiments.

If you're an advanced user of experiments (i.e., you're planning to run multiple experiments at the same time), be sure to check out Optimizing Experiments.