How Barstool Sports Scaled Media Data Operations with Prefect

Meet Barstool Sports

For media companies, attribution isn't negotiable. When an advertiser buys a media plan, they expect proof of performance across every channel in that plan.

Barstool Sports knows this challenge well. The company produces podcasts like Pardon My Take, which held the number one sports podcast spot globally for years. They create original video content, run an e-commerce business, and manage social media presences across thousands of accounts, including the Barstool Viceroy network, where basically every university in the country has a Barstool-affiliated online presence posting sports content on Twitter, Instagram, and TikTok. All of that content generates data that needs to flow reliably into dashboards and reports for a business that sells advertising based on reach.

Before Prefect: TypeScript Lambdas Without Observability

Before Barstool had a dedicated data engineering team, the broader engineering group handled data orchestration with the tools they knew best. "When they started to have data orchestration needs, they essentially did what they knew, and that was write TypeScript scripts," explains Jason Damiani, now the company's senior data engineer. Those scripts were deployed using the serverless framework as AWS Lambdas with cron schedules attached. A harness function would fire off messages to an SQS queue, other Lambdas would pick them up and process them, and logs went to Datadog. That was the extent of the orchestration.

The approach had almost no observability. Each job was a loosely coupled chain: a top-level function kicked off work, individual Lambda functions picked up messages from SQS queues and processed them independently, and nothing tied the pieces together. "There's absolutely no observability there to figure out on the given run which tasks succeeded, why did they fail, could they have been retried," Jason says. "All that was just non-existent."

The problems compounded at Barstool's scale. Pulling social media data from hundreds if not thousands of accounts meant depending on third-party vendor APIs that would intermittently fail. "These third-party vendors would have issues where their API would just stop working or not send back data for whatever reason and we would never know what the problem was. You wouldn't really know until somebody went to analyze data or build a report and saw that there was missing data."

Rate limiting was nearly impossible to manage. The team would hit 429 errors without knowing it, and those requests would never retry. "Retries generally looked like somebody a day or two later would say, I think there's missing data. Okay, let's just try to rerun the job and see what happens." Critical syncs between BigQuery exports and Snowflake would fail with no one noticing. "The data wasn't there for whatever reason. There was no alerting. There was no Slack channel that a job was posting to." There were no functional dependencies between jobs either. Nothing enforced the logic of "when this thing completes, run that thing; if it fails, don't." It was all imperative, with no dependency graph to lean on.

For a company whose advertising revenue depends on proving content reach across platforms, unreliable data infrastructure directly affects the bottom line.

Investing in Data Engineering

Barstool's decision to hire a dedicated data engineer was an acknowledgment that the TypeScript-and-Lambda approach had hit its limits. "They gave me the keys and said, 'We're tired of managing this. We've pushed the boundaries of what we know as TypeScript backend developers. Do whatever you think is necessary,'" Jason recalls.

Jason already had deep experience with Prefect. He was the engineer who brought in Prefect at Cisco years earlier, back when the platform was still coming out of stealth. "I had a lot of the same issues and gripes with Airflow that Jeremiah and Chris set out to solve," he says.

That prior experience meant Barstool didn't need a lengthy evaluation process.

The Rebuild

Barstool's current setup runs on Prefect Cloud using the workerless concept, where the Prefect platform has an IAM role that launches ECS tasks in AWS on their behalf. Everything lives in a monorepo containing flow definitions and supporting code like API clients. They use Pulumi for all infrastructure management: building Docker containers, pushing them to ECR, defining ECS clusters and task definitions. A prefect.yaml file triggers deployment of all flows on every merge to the main branch.

Over about a year and a half, the data engineering team progressively rewrote every TypeScript Lambda job as a Prefect flow. The migration also let them move away from unreliable third-party vendors and build direct API integrations with social platforms. The team also started orchestrating dbt Cloud jobs with Prefect, integrating transformation workflows into the overall pipeline. Joseph, who joined as a data engineer five months ago and has been working primarily on the dbt and data warehousing side, is now getting onboarded to writing Prefect flows as well.

"One dedicated data engineering resource replaced three or four other TypeScript engineers that were doing this work part-time," Jason says. "And then not only did I rewrite all of their stuff incrementally over the course of a year and a half, I also added maybe three to four times the number of ETL flows that existed. I wouldn't have been able to do that unless I had a platform like Prefect."

The shift was visible to leadership. At the one-year mark, Jason's manager told him: "It's amazing that I don't have to think about this stuff anymore. We have such few data issues whereas before we were literally just putting out fires all the time because it was so hard to figure out what was actually going on in the infrastructure."

Proving Reach to Advertisers

With reliable data infrastructure in place, Barstool can now do what the business actually needs: prove to advertisers that their media plans delivered. Those plans might include podcast live reads, display ads on the blog, dynamically inserted ads on back catalog downloads, and sponsored social media posts. Each channel generates its own metrics, and Barstool needs to tie them all together.

"A large part of it is just acquiring that data so we're able to prove to advertisers that we met the goals that were on the media plan," Jason explains. And when one channel underperforms, the data often reveals unexpected wins elsewhere. "We'll go back to the advertiser and say, look at how many impressions you were actually getting that you didn't technically pay for. It's almost like a make good to that advertiser."

This attribution capability proved its value during Black Friday and Cyber Monday, when management wanted real-time visibility into whether talent were promoting sales. Prefect flows ran continuously, pulling recent posts from social platforms and providing "a real-time, maybe 30-minute latency view of all of the posting that we've done in the past two days." With that data flowing, the team could quickly identify which talent needed reminders to post and which were already promoting the sale.

What's Next

Barstool is exploring ML to improve how they attribute content across platforms, using audio and video similarity detection to automatically identify when the same content appears in different formats.

That kind of evolving scope is part of why deployment flexibility matters for a team like Barstool's. "I think it's really cool how much effort Prefect's put into being able to run your workflows in a production way across such heterogeneous hardware, where you could go literally down to production orchestration with something that's running in a Jupyter notebook somewhere," Jason says. "That makes Prefect so much more approachable than other tools that would require a tremendous amount of infrastructure setup to get going."

For a small data engineering team supporting a media company's data needs across podcasts, e-commerce, social media, and advertising, the difference between constant firefighting and reliable orchestration came down to having infrastructure that could grow with the work rather than against it.

Try Prefect Today

Curious how companies like Barstool Sports, WHOOP, Cash App, and others build resilient data platforms with Prefect? Get started for free and experience it firsthand.