Cash App Gains Flexibility in Machine Learning Workflows with Prefect

“Fraud is a high-stakes game that takes constant work to stay ahead of bad actors. Working on it at Cash App means constantly striving to be at the cutting edge of the field.” - Isaac Tamblyn, Cash App

Cash App employs teams of financial experts that are constantly on the lookout for new transaction patterns from bad actors, but identifying this behavior is just step one in preventing fraud. These experts collaborate with Machine Learning Engineers on state-of-the-art infrastructure to deploy new models that combat new fraud patterns as quickly as they arrive.

Wendy Tang is part of the ML Tools and & Training Team at Cash App responsible for building and maintaining this infrastructure. Her team started their orchestration journey a few years ago with Airflow to construct ETL pipelines. They found Airflow to be a capable option for running SQL queries that moved data from one place to another or performed transformations. But limitations soon appeared. The flexibility and scalability needed to get up and running quickly with new fraud prevention models simply wasn't there. As Wendy mentioned in her Prefect Summit session,

"Airflow was no longer a viable option for Machine Learning workflows.”

Moving beyond ETL to orchestrate Machine Learning workflows

Whether or not your data team is responsible for mitigating fraud, it is likely the demands of your data team are increasing. We have a unique point of view into the changing landscape of data infrastructure here at Prefect. Just a few years ago, the data warehouse was the center of a data team’s universe. Now many see this field becoming commoditized. As ETL pipelines become less of a differentiator and source of value, more is needed from an orchestration tool to address new areas of value. The Cash App platform team wanted more than Airflow could offer, and found limitations began appearing in the following areas:

Cash App’s ML workflow platform needs

☁️ Heterogeneous compute

Compute needs can vary significantly between Machine Learning models and at various stages of a model’s life. A one-worker-fits-all approach to orchestration can result in very large and very small tasks using the same compute. Prefect’s deployments can run across configurable work pools in a wide variety of container, AI, and even Prefect-managed infrastructure options.

🎨 Custom Python

Private and custom Python packages can be critical to ML model development. They provide a clean and easy way to deploy and reuse modular, organized, and production-ready code. Task-specific package management is difficult with some orchestrators, as they can compete with the orchestrator’s default set of packages. Prefect's dependencies and environment are entirely separate from every flow it manages and the flow’s dependencies and environment, and each flow can be deployed to a unique environment. Wendy’s team at Cash App implements flow-specific environments via Access Control Lists that enable custom packages for users across clouds.

🤝 Exchanging data between tasks

Machine Learning workflows will often reuse assets. For instance, a developer at Cash App may want to create a variety of candidate models, each as its own workflow. Performance data on each model then passes to another workflow that promotes the best one into production. Airflow wasn’t very good at exchanging data between DAGs. Prefect flows have a simple and Pythonic way to share data between flows.

Cash App created a three-fold list of requirements that led them to adopt Prefect: flexibility for Machine Learning workflows, easy adoption by Machine Learning practitioners, and, most important of all, a high level of data security. Prefect’s wide array of deployment options and lightweight Pythonic developer experience have kept the team on the platform ever since.

Cash App’s secure data assembly lines

At Cash App, the ML Tools and Platform Team supports teams working with sensitive data across different development and production environments. Wendy likes to think of their ML workflows as an assembly line. Prefect offers a fast way to construct and iterate on components piece-by-piece before bringing them into production with Deployments.

As an Admin, she gives users deployment options aligned to the data privacy and compute needs of their workflows. Those options span local, Google Cloud, AWS, and Databricks environments. She isolates each user’s view of the ML platform by managing Access Control Lists for each Deployment. The result is better data security and compute efficiency.

Cash App’s future with Prefect

Workflows are continuing to evolve at Cash App. As the team looks to move beyond tree-based models into new and larger models, they are experimenting with ControlFlow. It is an open-source framework built on top of Prefect for resilient agentic AI workflows. The team wants to upgrade their compute instances to address the horizontal and vertical scaling needs of complex model types as well. We think our integrations with distributed computing frameworks such as Ray and Dask will be critical here. They keep the Pythonic experience their users already know.

Since moving to Prefect, the team has noticed that the added flexibility “has generated higher interest among internal customers as the platform offers more flexibility in workflows.” We are excited to keep partnering with Cash App, especially given the announcements made at Prefect Summit!

Discover the future of data at Prefect Summit

Organizations like Cash App are looking to build their next generation of data tools to take advantage of innovation without compromising the trust of their users. You can find more stories on how Prefect is enabling this trust with transactional, flexible, and portable orchestration on our customer stories page.