Gus Buchele leads a team of software engineers responsible for orchestrating workflows across a Python-based, cloud-native platform made up of hundreds of microservices. These services handle a variety of tasks and rely heavily on APIs and transactional databases.
To manage this complexity, the team adopted Prefect 1.0 early on. It gave them a Pythonic way to build, schedule, and monitor workflows without being locked into a rigid structure. Prefect 1.0 served them well, helping the team move quickly and bring order to a fast-growing system. But as their orchestration needs became more dynamic and infrastructure demands increased, they began planning a move to Prefect 2.0.
As their system matured, the team needed orchestration that could keep pace with increasingly dynamic workloads, higher task concurrency, and stricter reliability expectations. The patterns and infrastructure that had worked well at first began to show strain under new demands.
The team faced limitations in three key areas: concurrency, observability, and resilience. Workflows could easily overwhelm downstream systems, particularly under heavy load. Observability was minimal. Identifying bottlenecks often meant chasing logs and task IDs for hours, which Gus described as “like finding a needle in a haystack.”
Debugging and testing workflows became more difficult as the system grew. “Our monolithic workflows were prone to cascading failures,” Gus said. “Testing and release windows were huge due to the deployment delays.”
Deployments were manual, often taking several days to roll out across services. Without CI/CD pipelines, each release carried risk. And as data volumes grew, memory constraints and performance bottlenecks became more frequent, especially when workflows had to adapt to dynamic runtime conditions.
The team needed more than incremental improvements. They were ready for a more modular, observable, and scalable orchestration system. One that could evolve alongside their platform.
The new architecture gave Gus’s team greater flexibility and observability, with better support for the scale and complexity their workflows had grown into.
They moved from Prefect agents to Kubernetes workers, deploying the system on Google Kubernetes Engine (GKE). Each worker was configured with machine types tailored to specific workflows, and GKE node pools allowed the infrastructure to scale automatically based on real-time demand. “GKE node pools allowed us to dynamically scale our infrastructure,” Gus said. “Our workers triggered scaling based on workflow demand, using our predetermined machine types for optimal efficiency.”
Prefect 2.0 also introduced a more flexible task runner model, giving the team fine-grained control over how tasks were executed within flows. Combined with Prefect’s modular design, this enabled them to break workflows into smaller, more maintainable units. “Prefect 2.0 allowed us to manage and debug workflows more effectively,” Gus explained. “Most importantly, it provided robust concurrency management, letting us control the number of simultaneous tasks and flows.”
The real test, though, was whether this architecture could hold up under pressure.
One of the team’s most demanding services handles deeply nested JSON payloads and inserts the parsed data into transactional databases. This process previously pushed the limits of their infrastructure and risked overwhelming downstream systems. After migrating to Prefect 2.0, the team was able to run this service confidently, even under significant pressure. “One example is a service that handles large API payloads, processing up to 100 megabytes of nested JSON. By leveraging Prefect Flows, we parse these payloads and insert the data into our transactional databases. Concurrency controls are in place to prevent overwhelming the database, even during high-demand operations such as scheduling hundreds of thousands of flow runs to reprocess three months of data.”
The transformation delivered measurable results. “Prefect reduced our deployment times from several hours to just 20–30 minutes and significantly increased our system throughput,” Gus said. This new level of efficiency enabled the team to tackle large-scale data initiatives with confidence.
Cascading failures became rare, and workflows were easier to debug and evolve. Reflecting on the outcome, Gus added, “With Prefect, we’ve enhanced maintainability, reduced cascading failures, and improved our ability to manage large-scale workflows confidently.” These improvements were made possible by the team’s new orchestration architecture.
With their Prefect 2.0 environment fully deployed, the team is now preparing to upgrade to Prefect 3.0, which offers deeper observability and more advanced scaling features. They’re also continuing to fine-tune their use of Kubernetes workers and GKE node pools to ensure their infrastructure stays aligned with evolving demand.
As Gus explained, “We’ve built a framework with Prefect that not only solves today’s problems but prepares us for tomorrow’s growth.”
Curious how companies like Cisco, Cash App, and others build resilient data platforms with Prefect? Get started for free and experience it firsthand.
