DANGER

Managing customized Alertmanager alerting/recording rules is still an experimental feature. Metaplay does not guarantee feature availability at all times.

Dependencies

• Knowledge of Alertmanager rules - Before writing your own rules, you should have a basic understanding of Alertmanager recording and alerting rules.
• A Grafana instance - You should have a healthy Grafana instance for managing custom rules.
• metaplay CLI - You need metaplay CLI tool for managing rule groups.

Overview

Alertmanager alerting/recording rules are useful for continuous game server health monitoring and alerting. You can define your own rules to meet tailored needs.

Metaplay Platform supports both LogQL-based and PromQL-based rules. Rules can be divided into two types according to where the rules will persist:

• Grafana-managed rules: Rules that are created and persisted within Grafana instance, sharing the same lifecycle as the Grafana instance. While this approach is recommended for Grafana Cloud, Metaplay Platform does not enforce a specific lifecycle for the Grafana instance. As a result, Grafana pods may rotate, leading to a loss of rule persistence.
• Data source-managed rules: Rules persisted in multi-tenant Rulers. This is our recommended way for managing your own rules when working with Metaplay Platform. Please visit section How-to Guides for example usage cases.

How-to Guides

Creating rule groups

Before creating your first rule in Grafana, you need to create a rule group, to which custom rules shall be attached.

For the moment, rule groups can only be created with metaplay CLI if you follow these steps:

# choose an environment
export METAPLAY_TENANT_NAME=<target environment name>
# obtain an OAuth2 access token
export METAPLAY_ACCESS_TOKEN=$(metaplay auth show-tokens | jq -r .access_token)
# obtain the stack domain under which environment is provisioned
export METAPLAY_STACK_DOMAIN=$(metaplay get environment-info $METAPLAY_TENANT_NAME --format json | jq -r .observability.loki_endpoint | cut -d "/" -f 3)

Now you can manage rule groups with metaplay CLI. For example, creating a new rule group against Loki data source:

curl -X POST \
    -H "Authorization: Bearer $METAPLAY_ACCESS_TOKEN" \
    -H "Content-Type: application/json" \
    --data-binary @rulegroup.json \ # a rule group definition in JSON format
    https://$METAPLAY_STACK_DOMAIN/stackapi/tenant/v1/$METAPLAY_TENANT_NAME/observability/loki/loki/api/v1/rules/$METAPLAY_TENANT_NAME # proxy URL

INFO

The proxy URL can be split into three parts:

Part	URL
observability endpoint	https://$METAPLAY_STACK_DOMAIN/stackapi/tenant/v1/$METAPLAY_TENANT_NAME/observability/
component identifier	loki
loki HTTP API endpoint	loki/api/v1/rules/$METAPLAY_TENANT_NAME

You can also try other Loki HTTP API endpoints as per Loki HTTP API by simply combining a Loki HTTP endpoint with observability endpoint and component identifier, where request will be proxied to the backend Loki components if a proxy rule presents, example:

# List all rule groups in the environment
curl -X GET \
    -H "Authorization: Bearer $METAPLAY_ACCESS_TOKEN" \
    https://$METAPLAY_STACK_DOMAIN/stackapi/tenant/v1/$METAPLAY_TENANT_NAME/observability/loki/loki/api/v1/rules/

Likewise, if you want to create rule groups for Prometheus data source, you can combine Mimir Ruler HTTP endpoints with the same observability endpoint and a new component identifier alertmanager.

Creating rules

INFO

When creating rules, please ensure that you are creating data source-managed rules, which will be persisted. For any rules that are Grafana-managed, they share the same lifecycle with Grafana instance, hence such rules may get lost if Grafana instance reboots or rotates.

With a custom rule group in place, you can now create alerting/recording rules in your Grafana instance. Please check the page of Create data source-managed rule for more details.