Target Audience

This page is primarily intended for users of the Self Hosting (Private Cloud) tier package. If you are using any of the Metaplay SaaS plans, information from this page may not be directly relevant to your needs.

Overview

Game server administrators need to secure not only the game server itself but also the infrastructure that it runs on. Our default infrastructure modules attempt to follow reasonable practices in securing the different components, but it is worthwhile to understand the main vectors of attack and ensure that the environments are secured accordingly.

On a high level, we can roughly identify the following main components from the stack, which should be addressed when considering security:

• AWS account: The actual Amazon Web Services account that the game server and infrastructure run inside.
• VPC networking: Practically all components of the infrastructure stack run within an AWS Virtual Private Cloud network.
• Kubernetes control plane: The actual control plane which can be used the administrate the Kubernetes cluster, which runs game servers and the supporting services.
• Kubernetes nodes: The machines which make up the Kubernetes cluster.
• Databases: The source for persisted game server state, run as AWS RDS clusters.

Securing the Infrastructure Provisioning

We advocate applying good infrastructure-as-code practices to managing infrastructure, and the tool we suggest for that is HashiCorp Terraform. Adhering to the IaC practices allows you to more easily track how your infrastructure evolves over time as well as to ensure that at every infrastructure deployment you bring your infrastructure to a known, idempotent state.

Along with the benefits of IaC, you should also be mindful of the security related to the provisioning:

• Because infrastructure provisioning requires typically a fairly privileged level of access to the underlying cloud platform APIs, it is important to protect these access keys. If you run Terraform locally on your own machines, it is important for you to protect your AWS API keys and rotate them frequently. If you run Terraform as part of an automated CI/CD pipeline, the same applies, as well as protecting your CI/CD system and not allowing unnecessary access to the system or log outputs.
  • When teams are small and new, it is understandable that individual infrastructure engineers may run Terraform from their local machines due to the easiness and convenience. As your organizations mature, it is advisable to move towards automated infrastructure deployments through CI/CD tools as this will also enhance security and auditability by allowing the CI/CD system to keep track of all deployments and their associated logs.
• To function safely and keep track of the known state, Terraform preserves the state of infrastructure into state files. For continuity purposes, it is often advisable to configure Terraform to store these state files in e.g. S3 buckets. These are known as remote states. Terraform state files may, and often do contain sensitive information, so protecting these remote state files from access by others is critically important.
  • If storing the states in S3 buckets, make sure that the bucket is configured to be encrypted at rest, that it isn't publicly accessible, and that the bucket access controls limit access to only those individuals or systems who are expected to run Terraform.
• Never store secrets within your IaC repositories. Terraform makes it possible to pull data from other systems at execution time, so consider storing sensitive values that you may need at deployment time in other systems, such as AWS Secrets Manager or HashiCorp Vault. These dedicated secrets managers offer features such as audit logging and fine-grained access control to secrets, making sure that they are kept safe. It also makes rotating secrets easier when you know where they are kept and that changing them will allow for the changes to trickle down as part of regular deployment flows.
  • This same applies to CI/CD systems; they should ideally pull secrets as needed from dedicated secrets managers, as opposed to having those secrets persisted in the CI/CD system itself.

Securing Individual Levels of the Infrastructure

In this section, we give pointers and guidance on how to approach securing each level of the infrastructure stack.

AWS Account

Your AWS account is arguably one of the most important items to secure. A malicious party who gains control over your account can cause immense damage to not only your game but also potentially to any backups and other items that may be stored in the account.

• It is very advisable to run your production infrastructure in a separate AWS account from your other work. This limits blast radius if e.g. your development account gets compromised. Ideally, your production account should only run the game server infrastructure and other AWS accounts should be used for other systems.
  • We suggest typically a structure where the master AWS account is retained for consolidated billing and child AWS accounts are created under the AWS Organizations feature. A good baseline is to establish a production AWS account for all production infrastructure, a development AWS account for all development and testing activities, as well as a fully separate AWS account for placing AWS IAM users into (and then allowing those users to use IAM roles to get into different accounts).
  • AWS provides good documentation on master account management and child account management.
    • For further suggestions on architectural decisions of AWS account setups, please refer to Organizing Your AWS Environment Using Multiple Accounts.
• Maintain good practices in user management; make sure that a minimal amount of required individuals have access to AWS accounts. For those individuals that do, make sure they have secured their user accounts with multi-factor authentication and adhere to a periodic rotation of any AWS API keys which might be associated with their accounts. Leaked AWS API keys are a major source of security issues in the world and one of the easiest attack vectors that attackers can use.
• Use AWS IAM roles to allow individuals to access accounts instead of provisioning standalone IAM users for the individuals. It's much easier to maintain a secure user account if you only have to maintain one and not a whole load!
• Periodically review your accounts and remove unneeded access and in general maintain a level of hygiene. It is much too easy to forget this and combined with lax policies with e.g. key rotation, you are inviting problems!
• A good list of practical steps to secure AWS IAM is available at Security best practices in IAM, written by Amazon Web Services.

Networking

If you are deploying your infrastructure via the environments/aws-region module from our infra-modules GitHub repository, you deploy a two-tiered network setup by default, where the VPC network is split into public and private subnets. By design, the infra-modules place all possible infrastructure resources into the private subnets, and only uses the public subnets for hosting public-facing load balancers for the game servers and related tooling.

If you provision additional infrastructure into the VPC network, or if you adjust e.g. the Kubernetes cluster nodes to have public IPs, you should consider the following things.

• In general minimize the amount of infrastructure in the public subnets. By placing resources in the public subnets, they are associated a public IP and may be accessible from the public internet. By placing resources in the private subnets, we can guarantee that no direct access from the public internet can occur.
• For resources that you place in public subnets, make sure that those resources are protected by appropriate security groups and limit the IP addresses allowed to connect to those resources. Also, limit access to specific ports you need by using the security groups. Liberally configured security groups can be convenient for your users and allow them to access systems from home, but it also makes those same endpoints easily accessible to malicious users.
• While in general it is important to keep software and systems patched up and up-to-date, it is even more critical to keep publicly accessible systems secured.
• Amazon Web Services also provides a document named Security best practices for your VPC.

Kubernetes Control Plane

Most services, as well as the game servers, that are run are run as containers on Kubernetes. Kubernetes is a fairly complex and extremely configurable and extensible system for container orchestration. The flip side of this is that the complexity makes it important for you to secure it properly.

• Minimize the amount of people who have access to the Kubernetes control plane and for those who do need access, attempt to have them use e.g. AWS IAM roles to access the control plane (as opposed to static accounts with static secrets). The principle of limiting privilege also applies here.
  • Kubernetes API Access Control contains documentation on how to configure authorization and authentication for Kubernetes in general.
  • As we specifically use the AWS EKS variant of Kubernetes, cluster authentication can be handled with AWS IAM identities. AWS has prepared a document on Cluster authentication.
  • If you are using our Terraform modules to manage the Kubernetes cluster, you can use the module parameters to configure additional cluster users based on AWS IAM identities. The module Github readme provides examples of how to do this. Ask us for additional help!
• We understand that getting stuff to work is a first priority and as such by default our Terraform module provisions the AWS EKS control plane to be accessible publicly. We realize that this is not ideal for the long term, and ideally, you will have some form of secure connectivity to the VPC network (via e.g. a VPN or by running Terraform from a secure host within the VPC), at which point you should move the AWS EKS control plane to be purely private!
  • We have made this compromise as even if the AWS EKS control plane is publicly accessible, it is still considered secure as it requires all API requests to be authenticated by IAM and then authorized by Kubernetes RBAC.
  • A good compromise of having a public Kubernetes endpoint is to add additional CIDR whitelisting for the control plane and only allowing access to it from known addresses.
• By default we enable Kubernetes control plane logging to AWS CloudWatch. It is important for you to periodically keep an eye on these logs as they will contain the logs of what is going on within the control plane and if any malicious actions are being carried out. Because the logs exist in AWS CloudWatch, it is important for you to ensure that your AWS account is sufficiently secured (see above) so as to minimize the risk of allowing an attack to tamper your logs.
• Amazon Web Services has a section named Amazon EKS Best Practices for Security under their EKS Best Practices Guides document, which outlines other good practices for Kubernetes security in AWS EKS.

Kubernetes Nodes

Another attack vector on Kubernetes are the actual underlying compute nodes on which containers are executed. In our base design we run Kubernetes nodes as EC2 instances. We do this to allow for more control over the execution environment (as opposed to e.g. running Kubernetes pods on AWS Fargate). The flip side is that the usual EC2 security precautions should be taken.

• As mentioned in the network section, run Kubernetes nodes in the private subnets of the VPC. This will minimize the external footprint and not expose the individual nodes to be accessible directly from the internet. If you do run Kubernetes nodes in the public subnets, double-check to make sure how the security groups are configured and limit access as much as feasible.
• Because EC2 instances are plain vanilla virtual machines, it is important to keep them up-to-date. This means ensuring that you use current AWS AMI images and up-to-date operating system versions.
  • In our design, we manage the EC2 instances through auto-scaling groups and by default always use the latest available images during the time they are Terraformed. However, when updating auto-scaling groups and their launch templates, you will need to destroy and allow the recreation of the EC2 instances after AMI images are changed. This should be done during a scheduled service window as it will often run the risk of causing downtime as the instances are swapped.
• By default, the EC2 instances do not have any keys baked into them. This is by design as no one should have a need for accessing the EC2 instances directly. There are ways of injecting keys or allowing administrator access to the nodes, which may sometimes be needed for debugging purposes. These techniques are not discussed here; suffice to say that if you need to carry out such operations, make sure to remove any such keys or changes from your infrastructure afterward to minimize attack vectors.
• As with any virtual machines and other pieces of infrastructure, periodic reviews of logs should be carried out to make sure nothing unintended is happening. In our base setup, we deploy through the metaplay-services Helm chart log collectors onto each Kubernetes node. Along with collecting container logs, these log collectors will also collect logs from the underlying EC2 nodes (e.g. /var/log/messages). These logs can be accessed by default via Grafana.

Databases

The databases are critical to the functioning of your game and as such are critical pieces of infrastructure to protect.

• We always provision database clusters in the private subnets. While this can slightly complicate accessing the database for debugging purposes, we feel that this is an important tradeoff and that databases should never be directly accessible from the public internet. Additionally, live manual operations against production databases should be avoided. If you need to interact with the data, it is almost always more advisable to use database snapshots to seed a new debugging database cluster and interact against that, as opposed to the production databases!
• At present, game servers require database credentials to be able to access the database. In the future we intend to transition these to AWS IAM role-based access to the database, but for the time being, it means that you will need to safeguard these credentials. They are stored for the game servers to access as Kubernetes secrets within the Kubernetes namespace that the game server is deployed in, and as such you should limit access to Kubernetes resources within those namespaces (you should in general limit access to Kubernetes...).
  • Because those credentials are static, you should periodically consider rotating these credentials. The credentials managed by Terraform and rotating the game server password can be done by tainting the old password, for example, terraform taint -target='module.infra.module.deployments.random_string.db_password["deployment-name"]' and re-running Terraform. This creates a new random password for the deployment and updates the game server database user with it.
• Alongside protecting the live database clusters, it is also important to safeguard the backups as well. By default, we configure production clusters to automatically take snapshots on a daily basis, but in addition to this, it is important for you to consider configuring e.g. AWS Backup or similar backup routines to take out-of-band, longer-term backups. It is advisable to also store these backups in a separate AWS account so as to minimize the risk of a malicious party being able to tamper or destroy the backups.
  • The infra-modules environments/aws-region module allows you to enable AWS Backups of game server infrastructure resources. It is also possible to pass in a separate AWS Backup vault this way and have backups taken there.

Securing Game Servers

Game servers, due to their role, have to be externally accessible for them to be useful. This does, however, mean that the publicly accessible endpoints become possible targets for attacks. As always, one of the safest things to do is to make sure that you periodically update your game server and the underlying software with the latest versions of software and libraries to maintain an up-to-date posture with security patches. That said, there are additional steps that can be taken to increase security.

Game Server Administrative Endpoint

All Metaplay game servers have an admin dashboard component shipped along with them. The dashboard is a dynamic web application and interacts with backend server APIs to allow administrators and LiveOps managers to control the game. This is also one of the bigger possible attack vectors and a malicious user can wreak significant havoc.

• Ensure that you configure authentication and authorization for your game server admin dashboard. This is done by configuring the game server's options with appropriate roles and permissions (see Working with the LiveOps Dashboard HTTP API) as well as setting the authentication options to correct values. This will tell the game server to require a valid identity and role to be presented before it will allow access to the dashboard. If you're unsure of how to do all this, ask us!
• We expose the admin dashboard to the outside world via a Kubernetes Ingress resource, and in our default design, this Ingress resource is implemented using nginx. This means that you have the full flexibility of nginx to implement additional security in front of your admin dashboard.
  • When deploying your game server via the metaplay-gameserver Helm chart, you can pass annotations to nginx via the admin.annotations map in Helm values. The ingress-nginx user guide on annotations gives a good starting point for what you can easily tackle with annotations to the Ingress resource.
  • Some simple configurations that can greatly increase your security are to use rate limiting to protect from DDoS attacks, as well as to use whitelist source ranges to allow only known and trusted IP ranges to access the admin dashboard.