tldr

Get realtime logs, a live execution graph, and data lineage for your dbt projects for free in seconds.

To see it work, create a Prefect Cloud account, install uv, and run

1uvx prefect-cloud deploy flow.py:dbt_flow \\
2--name dbt_deployment \\
3--from PrefectHQ/jaffle_shop_duckdb_prefect/tree/duckdb \\
4--with-requirements requirements.txt

in your terminal to deploy our sample jaffle_shop_duckdb_prefect dbt project to Prefect Cloud.

Run the deployment immediately with

1uvx prefect-cloud run dbt_flow/dbt_deployment

or use

1uvx prefect-cloud schedule run-dbt/run-dbt "0 12 * * *"

to schedule the deployment to run daily at noon. Want to see how to do this for your own dbt projects?

Keep reading.

Introduction

Running dbt manually gets you started quickly, but eventually you need scheduled runs and proper monitoring. The usual path involves weeks of infrastructure setup, scheduler configuration, and monitoring dashboards.

This tutorial takes a different approach. We'll show you how to deploy your existing dbt project as a scheduled, monitored pipeline using nothing but a simple Python script and GitHub. The entire process takes under 5 minutes - we timed our product marketing manager doing it from scratch.

You'll learn how to wrap your dbt commands in a Prefect flow, deploy directly from your repository, and get immediate visibility into your pipeline runs.

Getting started

Turning a local dbt project into a reliable, scheduled pipeline is a breeze with Prefect Cloud. To demonstrate, let’s start from scratch.

Prerequisites

• A free Prefect Cloud account
• A free GitHub account
• git installed on your system
• gh (the GitHub CLI) installed on your system
• uv installed on your system

Create your dbt project

Open your terminal in a location you typically clone or create git repositories.

1uv venv dbt-example --python 3.12

1source dbt-example/bin/activate

1uv pip install "dbt-core==1.10.2" "dbt-duckdb>=1.9.4,<2.0.0"

1dbt init --profiles-dir . my_prefect_dbt_project

Select duckdb as your database to complete the dbt project setup.

Note that in its default state, the example dbt project will fail when run. If you’d like to see how Prefect handles dbt failures, leave things as they are. Otherwise uncomment the line

1-- where id is not null

in models/example/my_first_dbt_model.sql and save the file before proceeding to have a successful first run.

Add the Prefect parts

Now, open the folder containing your dbt project in your favorite IDE or code editor.

Create a file called requirements.txt in the root of your repository with the following contents:

1prefect-dbt>=0.7.0
2dbt-duckdb>=1.9.4,<2.0.0

Next, create a new Python script called flow.py and add the following code:

1from prefect import flow
2from prefect.runtime.flow_run import get_run_count
3from prefect_dbt import PrefectDbtRunner
4
5@flow(
6    description=(
7        "Runs commands `dbt deps` then `dbt build` by default. "
8        "Runs `dbt retry` if the flow is retrying."
9    ),
10    retries=2,
11)
12def dbt_flow(commands: list[str] | None = None):
13    if commands is None:
14        commands = ["deps", "build"]
15
16    runner = PrefectDbtRunner(
17        include_compiled_code=True,
18    )
19
20    if get_run_count() == 1:
21        for command in commands:
22            runner.invoke(command.split(" "))
23    else:
24        runner.invoke(["retry"])
25
26if __name__ == "__main__":
27    dbt_flow()
28

This script contains a Prefect flow. Using the PrefectDbtRunner, you can invoke the dbt CLI in Python against your project with any valid dbt command. This script locates the dbt project and dbt profile in its current directory, then runs dbt deps followed by dbt build. If any of the dbt steps fail, the flow run will retry and run dbt retry.

The contents of your dbt project folder should now look like this:

1├── analyses
2├── macros
3├── models
4│   └── example
5│       ├── my_first_dbt_model.sql
6│       ├── my_second_dbt_model.sql
7│       └── schema.yml
8├── seeds
9├── snapshots
10├── tests
11├── dbt_project.yml
12├── flow.py
13├── profiles.yml
14├── README.md
15└── requirements.txt

Push your local files to the repository

Use gh to log in to GitHub from your terminal.

1gh auth login

Open a terminal in your dbt project folder and run the following commands to create a GitHub repository and push the contents of your dbt project to it:

1git init

1git add .

1git commit -m "Initial commit: prefect dbt project created with dbt init"

1gh repo create my_prefect_dbt_project --public --source=. --remote=origin --push

Deploy and run

Finally, let’s turn this project into a schedulable pipeline that runs in the cloud.

Log in to Prefect Cloud from your terminal.

1uvx prefect-cloud login

Then, deploy your flow from the GitHub repository you created. Be sure to replace <your-github-username> with your GitHub username.

1uvx prefect-cloud deploy flow.py:dbt_flow \\
2--name dbt_deployment \\
3--from <your-github-username>/my_prefect_dbt_project \\
4--with-requirements requirements.txt

Run the deployment immediately with

1uvx prefect-cloud run dbt_flow/dbt_deployment

or use

1uvx prefect-cloud schedule run-dbt/run-dbt "0 12 * * *"

to schedule the deployment to run daily at noon.

What’s next?

Any further changes you make to your dbt project and push to your repository will be automatically picked up when you run your Prefect deployment. No need to deploy again!

From the Prefect UI, you can run any number of dbt commands against this project sequentially as a single, ad-hoc flow run.

What you've built

You now have a dbt pipeline running in the cloud. It runs on schedule, retries when things fail, and you can see what's happening in real-time. When something breaks, you'll know why and when you need to backfill data, just trigger a run from the UI.

This setup scales from prototype to production without changing your workflow. Make changes to your models, push to GitHub, and your pipeline stays current.

Get started with a free Prefect Cloud account and create your first dbt pipeline running in the next few minutes.