In this workshop, you will build out a simplified analytics pipeline similar to the Game Analytics Pipeline discussed in the previous section. You will build out the following architecture:
The services used in this lab include: Amazon Cognito, Amazon API Gateway, AWS Lambda, Amazon Kinesis, Amazon S3, AWS Glue, Amazon Athena, Amazon QuickSight, and Amazon CloudWatch.
Amazon Cognito lets you add user sign-up, sign-in, and access control to your game so that you can easily authenticate your players and authorize them to allow them to have access to backend resources. For example, you can use Cognito to authenticate requests made to an API or a load balancer.
Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. API Gateway will act as the “front door” for your game client to access your backend services. It is best practice to have an extra layer of separation between the player and your backend resources.
AWS Lambda is serverless compute capacity in the cloud, which lets you run code without provisioning or managing servers. Lambda will be used as an orchestration tool in this pipeline to execute code that can send data from your game to your AWS analytics pipeline.
Amazon Kinesis makes it easy to collect, process, and analyze real-time streaming data so you can get timely insights and react quickly to new information. You can use Kinesis for real-time delivery of in-game data collected from game servers and clients to be stored in your S3 data lake. There are multiple different flavors of Amazon Kinesis, but two will be covered in detail in this lab: Kinesis Firehose, which allows you to easily deliver streaming data to 4 built-in destinations on AWS including Amazon S3, and Kinesis Analytics, which allows you to run SQL queries on your streaming data in real-time.
Amazon S3 provides durable object storage in the AWS cloud. It makes for a great data lake solution because of its virtually unlimited scalability. As your data grows, you can increase storage from gigabytes to petabytes of data. It allows you to decouple storage and compute so that you can scale both independently as needed. This can act as a central data store for all of your game data.
AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy to prepare and load data for analytics. You can use AWS Glue to discover the data in your S3 data lake to make it searchable, queryable, and available for ETL.
Once your data has been discovered using AWS Glue, you can use Amazon Athena to query your data using standard SQL. With Athena, you can query exactly the data you want to analyze.
Once you query the data you are interested in analyzing, you can use Amazon QuickSight as a business intelligence service to discover insights about your game data. QuickSight works really well for batch analytics with data that has been collected over a time frame of a couple hours, to days, months, even years. You can create and publish interactive dashboards and visualizations. You can even discover hidden trends and do forecasting using machine learning. With QuickSight, you can answer questions about your game - is it too hard? Is it too easy? Are your players engaged and going to stick around?
Amazon CloudWatch is a monitoring and observability service that provides you with data and actionable insights to monitor your applications, respond to system-wide performance changes, optimize resource utilization, and more. You can instrument your pipeline to send custom metrics to CloudWatch and view these metrics on an automatic dashboard that CloudWatch provides. This service works well for visualizing data in real-time.