At one level Prefect ticked the basic requirement of a workflow orchestrator (like Airflow) but I could also see how Prefect was a technology that could help us scale by improving observability to our team and encouraging adherence to data engineering best practices (testing, functional paradigms, etc).

I was very impressed with the quality of the technology as well as the overall energy of the project. Since we were building business-critical systems, we really wanted a partner, not just a technology.

At present the main cost saving for engineering is the improved observability we have into our data processes. It has allowed us to debug issues more quickly and understand complex behaviors in ways we couldn't before.

By reliably importing the data several times a day, our analysts can then make use that data by performing analyses, writing reports, and sharing those results with our front office, coaching staff, and players.

We have been able stay on top of the data flows we've moved to Prefect easily. Seeing failures, successes, and outages in a timely and clear fashion has let us inform stakeholders what's up with the data flows.

Prefect Core provides a really nice, clean set of easy to use features for constructing data pipelines (Tasks, Flows, States, Results) and it’s easy to model the dependencies between tasks. There was a very short learning curve for getting started, and the framework provided by Prefect core was already way better than our crude implementation of try/catch statements.

We call Prefect our secret sauce. We can now enter other industries at an unlimited capacity.

Prefect, for us, has changed how we position our company and our roadmap. We shifted hard from a dashboard company (with a relentless cycle of never being done) to a modern, clean engineering company.

Prefect Cloud is worth 3 full-time engineers.

We context-switched to Prefect, got into Prefect Cloud as early adopters. We have definitely gotten the return on our investment. All aspects of our data organization use Prefect for scheduling data workflows. Prefect is now how we write these data applications and people expect to start there for anything new.

Saving us days on DAG design vs. Airflow

We were able to use AWS ECS Fargate as our agent and flow execution environment to limit our total cost of ownership on the infrastructure side. It was a huge success and has really enabled our data user to do much more than just waiting on Data Engineering, which was the ultimate goal.

I used the parallelized hyper-parameter tuning (incredible) to run about 350 experiments in 30 minutes, which normally would have taken 2 days.

By cutting our machine learning experiment time from 2 days to 30 minutes we had a huge gain in the number of experiments we could run each day. By allowing us to iterate faster, we could quickly experiment with more machine learning approaches, which ultimately got us to production ready models very quickly.

As a vendor to covered entities under HIPAA, security and compliance are huge issues for us. Prefect’s hybrid execution model was key to being able to run with Cloud while none of our client data left our infrastructure.