How Prefect Works
From the simplest pipeline to the most complex workflow, see how Prefect's structure supports your use case.
With Prefect, you can turn Python code into a flow — a schedulable, observable, reactive process — just by adding an @flow decorator to the main function.
from prefect import flow, task
def hello(name: str):
print(f"Hello, {name}")
@flow
def say_hello ():
for name in ["Marvin", "Arthur", "Ford"]:
hello(name)
You can deploy a flow so that it can run on remote infrastructure.
$ prefect deploy my_flow.py:say_helloYour code and the data it handles only reside in your environment, but Prefect knows:
What code should be retrieved
Where it should run
When it should run, based on a schedule or event
Prefect creates flow runs — jobs during which the flow code is executed — in advance for flows with a schedule, or ad hoc when a flow's trigger event happens.
Flow runs have a state — their current status. Flow runs are created with a Scheduled state.
A worker — a process running in your environment — regularly checks with Prefect for any flow runs that are scheduled to start.
When it's time for a flow run to start, Prefect provides the worker the information it needs:
- Where the code can be retrieved
- What type of infrastructure it required
- What requirement it has
When the worker receives the information, it gathers the necessary assets, prepares the infrastructure, and starts the flow run.
The flow run sends logs and state change updates to Prefect.
@flow (retries=3)
def say_hello():
for name in ["Marvin", "...Prefect tracks and controls this state. For example, when a flow run fails, Prefect can direct it to retry.
A flow can be divided into tasks — independently observed and controlled parts — by adding an @task decorator to each function.
from prefect import flow, task
@task(retries=3)
def hello(name:str):
print(f"Hello, {name}")
@flow(retries=3)
def say_hello():
for name in ["Marvin", "Arthur", "Ford"]:
hello(name)
Task runs also have state, so they can be tracked and retried independently. The dependencies between task runs are also tracked.
With Prefect, you can turn Python code into a flow — a schedulable, observable, reactive process — just by adding an @flow decorator to the main function.
from prefect import flow, task
def hello(name: str):
print(f"Hello, {name}")
@flow
def say_hello ():
for name in ["Marvin", "Arthur", "Ford"]:
hello(name)
You can deploy a flow so that it can run on remote infrastructure.
$ prefect deploy my_flow.py:say_helloA worker — a process running in your environment — regularly checks with Prefect for any flow runs that are scheduled to start.
When the worker receives the information, it gathers the necessary assets, prepares the infrastructure, and starts the flow run.
The flow run sends logs and state change updates to Prefect.
@flow (retries=3)
def say_hello():
for name in ["Marvin", "...A flow can be divided into tasks — independently observed and controlled parts — by adding an @task decorator to each function.
from prefect import flow, task
@task(retries=3)
def hello(name:str):
print(f"Hello, {name}")
@flow(retries=3)
def say_hello():
for name in ["Marvin", "Arthur", "Ford"]:
hello(name)
Your code and the data it handles only reside in your environment, but Prefect knows:
What code should be retrieved
Where it should run
When it should run, based on a schedule or event
Prefect creates flow runs — jobs during which the flow code is executed — in advance for flows with a schedule, or ad hoc when a flow's trigger event happens.
Flow runs have a state — their current status. Flow runs are created with a Scheduled state.
When it's time for a flow run to start, Prefect provides the worker the information it needs:
- Where the code can be retrieved
- What type of infrastructure it required
- What requirement it has
Prefect tracks and controls this state. For example, when a flow run fails, Prefect can direct it to retry.
Task runs also have state, so they can be tracked and retried independently. The dependencies between task runs are also tracked.