Rec Room: From Workflow Chaos to Orchestrated Bliss

An explosion in users

Online community platforms have grown dramatically in the last 5 years, as people of all ages come together in virtual communities, finding and engaging with likeminded people. One such online platform is Rec Room: a massively multiplayer online game platform, where users of all ages and backgrounds come together to create and play “games within a game”.

Organic communities have sprouted out of Rec Room, spanning video games, movies, TV, music, and even cocktails. With support for VR, consoles, mobile, and PC, Rec Room has grown to 85M+ lifetime users.

With this explosion in their player base has come increasing demands on the Rec Room data team. While they had built many integrations to the variety of services that they used, those connections were brittle, prone to breaking, and featured little visibility across platforms.

The Rec Room team needed an observable, reliable data platform to achieve 3 things:

• Time: Get back the hours they spent chasing broken data pipelines
• Reliability: Gain the trust of their stakeholders, giving them confidence that the data they were using was reliable
• Revenue: Give everyday users a reliable data platform, a tool they could use to make revenue-impacting decisions around ranking, recommendation, and personalization for their player base

The state of the Rec Room data organization

Analytics Engineering, Data Science & ML teams

These teams are users & consumers of the data platform at Rec Room. They are responsible for building models to answer business questions. From basic aggregation and linear regression to complex forecasting algorithms, attribution analysis, and more. This team has a broad set of skills, favors Python, and has limited infrastructure experience.

Server Team

Handles data infrastructure, supporting analytics engineers, data scientists, and machine learning teams. This team is responsible for making sure that data is accessible and reliable. Led by Emerson Franks, Principal Engineering Lead at Rec Room and the star of our story today.

The Rec Room Stack

The Rec Room data team uses Azure as their cloud infrastructure provider of choice, with the following key tools:

Container Orchestration: Azure Kubernetes Service

Data Science & Model Training: Databricks, where teams primarily write Python

Data lake: Azure Data Lake Service Gen2

Data movement: Fivetran, pulling from a variety of sources

Data sources: Google Sheets, Azure SQL, various APIs and other systems

Transformation: dbt

Alerting: PagerDuty & Slack

Challenges before Prefect

Unreliable data

When Emerson first joined Rec Room in mid-2022, the Rec Room stack was growing quickly to support their explosion in users. They had nearly all of the fundamental pieces in place: tons of data movement, transformation, models being trained, and ad-hoc analysis.

According to Emerson:

We had an immature stack in terms of hardening and observability. We had done a lot of great work to cobble things together and make them work, but one thing that stood out to me was our lack of an orchestrator.

It had become clear to Emerson that while they had built a powerful data practice, it was hard to identify when something was broken. Building new pipelines or troubleshooting existing ones took significant time out of each workday.

(We needed) a single source of truth that is accurate and reliable so the business could make decisions on outcomes. Especially as you start getting into machine learning, and generating rankings and recommendations for players…in real time, (we want) a data platform that is solid behind those business-impacting decisions.

Infrastructure expertise

The Rec Room data users have a variable set of skills, and their focus is on business decisions instead of technical architecture.

“Analytics team” for us is a very broad term: it’s everything from business analytics (monetization, retention, token spend) to player behavior and player journey.

Rec Room analysts and data scientists didn’t have infrastructure or DevOps experience, but were using a stack where they would need to know how to provision a Databricks cluster, a VM, or serverless infrastructure on Azure. This limited the output of some users significantly.

From a data stack, to a reliable data platform

Rec Room implemented Prefect in 2022 and hasn’t looked back. With an orchestration platform that lives across their complicated stack, they have built confidence among stakeholders while empowering users to make revenue-impacting decisions.

While their stack of a data lake with Fivetran+dbt+Databricks is a common and effective one, Rec Room knows that they need an orchestrator that can help break down silos and give them visibility into all of the tools that they use. With Prefect, Rec Room has the power to build critical data pipelines faster, and the confidence that any workflow failure will be addressed and fixed right away.

Data that Rec Room can trust

One of the core things that you need…is a single source of truth…that is accurate and reliable. That’s where we are now (with Prefect).

With Prefect, the Rec Room team now has observability across their entire data platform. This has given them insight when things go wrong, and the ability to act fast and correct any issues.

We’re now in a place with our data where we understand what exactly is happening, and that has given us the confidence to extend into personalization.

Having a unified view of workflows, and the confidence to know when they are broken, is a key value proposition for any data team that wants to build trust with their stakeholder teams.

Empowering analysts, data scientists, and ML engineers

Emerson and team have built a hardened, observable data platform with Prefect that gives others the ability to multiply their own output.

We have built template files that power Prefect deployments, so an analytics engineer or data scientist can just add new entries to a config file and it will deploy all of their code (on a schedule) for them.

From simple transformations powered by dbt to complex workflows involving Fivetran, dbt, and Databricks, the Rec Room team can now get their workflow into production in about five minutes. Before adding orchestration, this end-to-end deployment of a workflow would not have been possible.

Let’s say we have a new moderation database that comes into existence. We can prepare that for analysis and get it scheduled with dependencies (on a 15 minute interval) in about 5 minutes.

Editor’s note: When I asked Emerson if this was possible before they used Prefect, he said “no”.

What’s next

Workflow orchestration isn’t only about enterprise scheduling. Prefect is a powerful framework that has given Rec Room the ability to build a data platform, empowering others to level-up their work without having to be infrastructure, DevOps, or orchestration experts.

With the trust and confidence of their stakeholders earned, the Rec Room team has their sights set on further hardening their data platform, building advanced features for their users, and turning towards streaming and event-based workflows.

We’re listening to streams of data, and could potentially do some interesting things based on those…and have them constantly running. This was a differentiator for us, when choosing Prefect.

Prefect makes complex workflows simpler, not harder. Try Prefect Cloud for free for yourself, download our open source package, join our Slack community, or talk to one of our engineers to learn more.