How Foundry Cut Workflow Deployment Time by 80% Using Prefect
Customer: Foundry
Industry: Cryptocurrency infrastructure
Use Case: Data pipeline orchestration
Key Outcomes:
- Engineers migrated pipelines from GitLab CI to Prefect within a single sprint
- Detection time for failures dropped by 95-99 percent, from up to 5 days to under 15 minutes
- Monthly pipeline downtime reduced by up to 85%, from 30 hours to under 5
Meet Foundry
Foundry supports large-scale institutional Bitcoin mining. Its flagship product, the Foundry USA Pool™, helps clients consolidate hash rate and increase revenue from mined blocks. Foundry also builds software products that support hardware fleet management and mining firmware, as well as provides site operations services to institutional miners.
Behind the scenes, a lean but critical data team powers internal reporting and internal customer dashboards. Siva Nadesan, Director of Data, leads the platform and analytics efforts. Josh Caskie, a data engineer on the team, helped lead the evaluation and implementation of Prefect.
“We’re focused on institutional clients only. These are major operations, not hobbyists, with huge fleets of miners and serious operational needs.” — Siva Nadesan, Director of Data
Growing Pains with GitLab CI
For years, Foundry's data team relied on GitLab CI for scheduling and orchestration. While effective at allowing the team to get basic pipelines running with low development time, it lacked reliable scheduling and success/failure monitoring. This left the team firefighting issues manually in a disconnected system where each pipeline lived in its own repo, further reducing visibility into jobs.
“At the time, we lacked basic retry logic. If a job failed, you had to go run it manually. It was ad hoc every time.” — Josh Caskie, Data Engineer
Schedules defined in GitLab CI weren’t reliable due to shared runner contention and job start times drifted. More importantly, the system wasn’t built to handle parameterized runs or recover from failures with context.
“The data quality issues were real. And because we didn’t have the right tooling, the team couldn’t be accountable. We couldn’t say confidently what failed and when.” — Siva
As usage grew and expectations rose, the cost of broken workflows and invisible failures became too high to ignore. They needed orchestration that could keep up, not just with their pipelines, but with the business they were supporting.
Evaluating Prefect Against Competition
Siva had experience with orchestration tools going back to the mainframe era. He’d previously used CA7, AutoSys, Control-M, Airflow, and Astronomer, but didn’t want to automatically default to old tools. He wanted a modern solution that fit the team’s Python-based workflows and wouldn’t force a rebuild.
Josh ran a structured evaluation of Prefect against other leading orchestrator platforms. He ran POCs, reviewed architecture, and compared costs.
“We needed something easy for engineers to pick up, where we wouldn’t be locked into a whole new paradigm. We didn’t have time to train the whole team for six months just to get workflows running, which ruled out many options.” — Josh
It was soon discovered that other tools wouldn’t be a fit. Their opinionated modeling approach would have required the team to rewrite their existing pipelines, which wasn’t feasible. Pricing models based on cloud credits made it difficult to forecast costs reliably, and, most critically, the team didn’t have time for a long ramp-up period. They needed something their engineers could use productively right away, not a tool that would take months to adopt.
“The more we dug into other tools, the more it became clear that they weren’t the right fit. For our team specifically, we were looking at around six months at least to get every repo migrated, which didn’t work for us, even if we ended up loving the end result.” — Josh
In contrast, Prefect offered a better fit across the board.
- Python-native development that fit their existing workflows “We needed a tool that was flexible, easy to use, and Python-first. That’s what Prefect gave us.” — Siva
- A flat, predictable pricing model “We really liked Prefect’s pricing model. The flat fee made it easier to plan compared to cloud credit-based models.” — Josh
- A clear migration path from GitLab pipelines “Josh documented the process really well. Anyone who started the migration from GitLab to Prefect was able to convert all their pipelines within a sprint.”— Siva
- Support for Terraform and infrastructure as code “We didn’t want to hand-build infrastructure. We manage everything with Terraform, including Prefect clusters and resources.” — Siva
- Flexibility to scale across teams and use cases “It’s general enough that we could onboard other teams if they need orchestration. There’s a lot of extensibility there.” — Josh
How Prefect Powers Their Data Platform
Foundry runs three Prefect workspaces: development, stage, and production. Everything is managed through Terraform, including workspaces, work pools, and blocks. Most flows are batch jobs moving data between internal systems, S3, and Snowflake. They also use Prefect to trigger dbt models and generate reports.
One of the most critical pipelines Prefect supports is the Operational Dashboard, a key operational tool used by Foundry’s business development and product teams. The dashboard consolidates performance metrics, helping to streamline information the team relies on for daily conversations and decision-making.
“We ingest and transform data for our dashboards using Prefect and dbt. If someone has a call coming up, they check the Operational Dashboard for the most accurate look at the data.” — Siva
Before Prefect, failures in this pipeline were often hard to detect. Now, alerts are immediate, and the data team can address issues before they affect the business.
From Fire Drills to Focus
Before Prefect, the team spent too much time firefighting silent issues. Now, issues are surfaced immediately, teams are alerted through PagerDuty, and incidents are resolved quickly.
“Now it’s part of our on-call process. Our daily standups include checking Prefect. Everything’s visible.” — Josh
Prefect didn’t just improve observability, it changed the team’s posture. Engineers now have the tools and accountability they need. That shift restored internal trust in data and enabled better business decisions.
“We didn’t have the right tools to make ourselves accountable. Prefect brought that in and helped us continue to build trust with stakeholders.” — Siva
That trust is critical. Foundry’s dashboards inform customer conversations and product strategy. If a stakeholder is preparing for a call, they need to know the data is right. Prefect helps make that possible.
The Measurable Payoff of Switching to Prefect
Prefect didn’t just modernize Foundry’s orchestration. It delivered clear, measurable improvements through reliability, speed, and team efficiency:
- Detection time dropped (MTTD) 95–99% “With Prefect, issues that used to take days to detect now surface in under 15 minutes. That kind of visibility changes how the whole team operates.” — Josh
- Recovery time improved (MTTR) 80–90%“Before Prefect, we had very little visibility into shortcomings. Now we catch 100% of issues right away. It’s night and day.” — Siva
- Missed data refresh SLAs dropped 80–90% “Prefect gave us the observability to stay ahead of issues.” — Josh
- Pipeline failure visibility: Up to 100% “Before Prefect, we had very little visibility into shortcomings. Now we catch 100% of issues right away. It’s night and day.”— Siva
- Pipeline downtime reduced 75–85% “We used to lose time on other initiatives because downtimes meant manual fixes. Now we rarely see issues, and if we do, a quick retry usually solves it.” — Josh
- Workflow deployment time cut by ~80%“Prefect helped us get new workflows into production much faster. It took days before, now it’s hours.” — Josh
- Engineer onboarding sped up 50–70% “Before Prefect, it could take two or three weeks to get a new engineer fully onboarded. Now it’s usually less than a week.” — Siva
These changes gave the team back hours of manual effort, improved engineering velocity, and helped restore trust in the data powering internal reports and customer conversations.
A Foundation for Reliable Data and a Reliable Business
Foundry didn’t choose Prefect just to modernize their pipelines, they chose it to build trust, eliminate manual effort, and ensure their platform could scale with the business. Better orchestration gave the data team time back and gave the business data it could count on.
“With Prefect in place, we don’t have to think about orchestration as a problem every day. The platform just works.” — Siva
“We could get our jobs running in production quickly. Prefect wasn’t trying to reinvent how we work. It just fit.” — Josh
Related Content
