On March 24, 2017, we started writing a new kind of workflow tool.

Today, on its two-year anniversary, we’re excited to announce that Prefect has launched as an open-source project with an Apache 2.0 license.

Click here to get started, or dive right into our repo on GitHub!

Our team has spent years building tools for data engineers and data scientists, including as a PMC of Apache Airflow. Prefect is the culmination of those lessons. Our design is informed by hundreds of data engineering use cases that have allowed us to identify core frictions in the modern data stack. We’ve worked with over a hundred Lighthouse Partners to improve our software over the last six months. And we’ve designed every piece — from the execution engine to the logging to the data handling — to be modular and easily customizable.

As a result, at launch, Prefect is already the easiest, most powerful, and best-tested open-source workflow engine.

Here are a few things Prefect is doing right now:

• Processing terabytes of climate data

• …and

   

• Powering a quantitative investment strategy

• …and turning on a coffee machine every morning

• Orchestrating the backend of our Cloud execution platform

• …and

   

• Parameterizing machine learning models

• …and lots and lots (and lots) of

   

If you can code it, Prefect can automate it.

What will you build?

Welcome to Prefect!

Prefect is a new kind of workflow management system.

We started with a simple premise:

Your code probably works. But sometimes it doesn’t.

When your code works, you might not even need a workflow system. We call that positive engineering. It’s only when things go wrong that a system like Prefect starts to be valuable. That’s negative engineering: all the little details that guarantee your code either achieves its goal, or fails successfully. In this way, workflow systems are actually risk management tools, like insurance: there when you need them, invisible when you don’t.

And yet, we don’t see a single tool designed that way. Other workflow systems seem to believe that they’re actually positive engineering tools, somehow enabling users to do things they couldn’t do otherwise. As a result, they feel no shame in asking users to generate yet another config file, or contort code into a convoluted DAG structure. Prefect already knows you can write incredible code; it just wants to make sure it works.

Prefect takes your code and transforms it into a robust, distributed pipeline. You can continue to use your existing tools, languages, infrastructure, and scripts. Prefect builds a rich DAG structure, but in a way that respects positive engineering and doesn’t inhibit it. You can use Prefect’s functional API to transform scripts with minimal hooks; or you can access the deferred computational graph directly; or any combination thereof. It’s up to you.

The most common thing we hear about negative engineering is: “This should be easy!”

Prefect is the first step toward making that true.

The Prefect Platform

Open-sourcing the Prefect Core engine is a major milestone and completes the first stage of the Prefect platform rollout.

Soon, we will expand that platform with Prefect Cloud. In both its free and paid versions, Prefect Cloud will automatically extend the Core engine with:

• a full GraphQL API

• a complete UI for flows and jobs

• remote execution clusters

• automatic scheduling

• permissions and authorization

• projects for flow organization

• secure runtime secrets and parameters

• …and a few things we’re not ready to talk about yet!

One of our goals is that Prefect runs on Prefect and indeed, Prefect Cloud is already starting to power Prefect HQ. We’re working with our Lighthouse Partners to get everything ready for a wider release. Please get in touch to apply for access.

We can’t wait to see what you build.

Happy engineering!

- The Prefect Team