How Prefect Works

From the simplest pipeline to the most complex workflow, see how Prefect's structure supports your use case.

Your Environment
Prefect Cloud

With Prefect, you can turn Python code into a flow — a schedulable, observable, reactive process — just by adding an @flow decorator to the main function.
from prefect import flow, task
def hello(name: str):
print(f"Hello, {name}")
def say_hello ():
	for name in ["Marvin", "Arthur", "Ford"]:

You can deploy a flow so that it can run on remote infrastructure.

$ prefect deploy

Your code and the data it handles only reside in your environment, but Prefect knows:

    What code should be retrieved

    Where it should run

    When it should run, based on a schedule or event

Prefect creates flow runs — jobs during which the flow code is executed — in advance for flows with a schedule, or ad hoc when a flow's trigger event happens.

Flow runs have a state — their current status. Flow runs are created with a Scheduled state.

A worker — a process running in your environment — regularly checks with Prefect for any flow runs that are scheduled to start.

When it's time for a flow run to start, Prefect provides the worker the information it needs:

  • Where the code can be retrieved
  • What type of infrastructure it required
  • What requirement it has

When the worker receives the information, it gathers the necessary assets, prepares the infrastructure, and starts the flow run.

The flow run sends logs and state change updates to Prefect.

06:39:52 AM Worker submitting flow run06:40:12 AM Creating job...06:40:31 AM Job 'grinning-ferret-9l4zf': Pod has status 'Pending'06:40:46 AM Completed submission of flow run '65428d93cc39'06:41:07 AM Job 'grinning-ferret-9l4zf': Pod has status 'Running'

@flow (retries=3)
def say_hello():
	for name in ["Marvin", "...

Prefect tracks and controls this state. For example, when a flow run fails, Prefect can direct it to retry.

A flow can be divided into tasks — independently observed and controlled parts — by adding an @task decorator to each function.
from prefect import flow, task
def hello(name:str):
	print(f"Hello, {name}")

def say_hello():
	for name in ["Marvin", "Arthur", "Ford"]:

Task runs also have state, so they can be tracked and retried independently. The dependencies between task runs are also tracked.