Managing Prefect Deployments with Terraform

What You Will Learn

In this tutorial, you'll learn how to use the Prefect Terraform Provider to manage Prefect deployments. Specifically, you will:

• Define and configure the Prefect provider for authenticating with Prefect Cloud.
• Set up a work pool to manage workflow execution.
• Retrieve flow code from GitHub or private repositories.
• Set up schedules for automatic workflow execution.
• Understand the benefits of using Terraform to manage deployments at scale.

If you've worked with Prefect to orchestrate workflows, you've likely created deployments using the prefect.yaml config via CLI, Python scripts, or directly through the API. These methods work great for developers who are building and testing workflows, but if you’re an infrastructure or platform engineer, whose priorities are to configure a broader organization’s platform in a consistent and repeatable way, you might need a more consistent, automated approach

Using Terraform along with the Prefect Terraform Provider can help bring that consistency and scalability to your workflow deployments.

In this tutorial, we’ll show how to use the Prefect Terraform Provider to manage Prefect deployments. You’ll learn why this approach is beneficial, how to define and configure deployments, and how to apply this pattern to your own workflows.

What Is a Deployment in Prefect?

Before we dive into Terraform, let's clarify what a deployment is in Prefect.

A deployment is a server-side representation of your workflow. It's how you tell Prefect when, where, and how to run a flow. This includes:

• Where to find your flow code (e.g. a Git repo or Docker image)
• When to run it (through schedules or triggers)
• How to run it (infrastructure settings like work pool, environment variables, etc.)
• What parameters to use

Deployments don't store the code itself, they just reference it. This separation makes deployments perfect for infrastructure-as-code, where your configurations are stored as code and versioned for consistency.

What Is the Prefect Terraform Provider?

The Prefect Terraform provider lets you manage your Prefect resources through Terraform’s declarative interface and toolchain. You can use it to manage:

• Deployments / Flows
• Work pools
• Automations
• Service Accounts
• Blocks
• Webhooks
• …and more

By using Terraform tool to define these resources, you get:

• Centralized configuration: Keep all your orchestration configurations in one place, alongside your cloud infrastructure
• Seamless CI integration: Deploy flows using the same Terraform pipelines your team already trusts for cloud infrastructure
• Environment parity: Spin up identical deployments across development, staging, and production without worrying about drift

In this tutorial, we’ll focus on defining and managing deployments, the core component for running workflows in Prefect.

Use Case: Scheduling your Deployment in Terraform

Let's walk through a real-world scenario where you, as an infrastructure engineer, need to set up workflows for multiple data engineering teams. The teams write their Prefect workflows and push them to GitHub, but now it's up to you to deploy those workflows, schedule them, and ensure they run consistently.

To accomplish these tasks, you will want to:

• Define where the workflows will run (e.g., work pools, cloud infrastructure)
• Set up and manage deployments for each workflow
• Ensure the workflow code is retrieved from the right source (e.g., GitHub)

Using Terraform to define and manage these tasks helps repeatability and consistency across all environments, while also automating deployment and scheduling processes.

First, let's start by defining the Prefect provider, which will allow us to authenticate and connect Terraform to your Prefect Cloud account.

1. Define the provider

Start by defining the prefect provider block to authenticate with your Prefect Cloud account and workspace (or self-hosted server).

1terraform {
2  required_providers {
3    prefect = {
4      source = "prefecthq/prefect"
5    }
6  }
7}
8
9# Prefect Cloud
10provider "prefect" {
11  api_key      = "<your Cloud API Key>"
12  account_id   = "<your Cloud account ID>"
13  workspace_id = "<your Cloud workspace ID>"
14}
15
16# Self-hosted Prefect w/ Basic Auth
17provider "prefect" {
18  endpoint       = "<your self-hosted URL>"
19  basic_auth_key = "admin:<your-basic-auth-password>"
20}

2. Choose where your workflows will run

Next, create a work pool. In Prefect, a work pool connects your workflows with the infrastructure that will execute them.

You can use Prefect’s Managed infrastructure by selecting theprefect:managed work pool type, or you can opt for your own compute resources.

Here, we'll use Prefect’s managed infrastructure:

1resource "prefect_work_pool" "my_work_pool" {
2  name              = "my-work-pool"
3  type              = "prefect:managed"
4}

3. Define how your workflow code will be pulled

Prefect offers a variety of code-storage options.

In this scenario, your team is pushing their code to version control. The most straightforward option is to configure your deployment to pull from the source repositories:

1resource "prefect_flow" "my_flow" {
2  name = "my-flow"
3}
4
5resource "prefect_deployment" "my_deployment" {
6  name    = "my-deployment"
7  flow_id = prefect_flow.my_flow.id
8
9  work_pool_name  = prefect_work_pool.my_work_pool.name
10  work_queue_name = "default"
11
12  pull_steps = [
13    {
14      type       = "git_clone"
15      repository = "<https://github.com/org/public-repo>"
16      branch     = "main"
17    },
18  ]
19  entrypoint = "dir/file.py:flow_name"
20}

Prefect also supports pulling from private repositories. If you are managing access tokens in Terraform, you can pass a GitHub access token to a prefect_block resource, and reference it in your prefect_deployment resource’s pull_steps:

1resource "prefect_block" "github_credentials" {
2  name      = "private-repo-creds"
3  type_slug = "github-credentials"
4  data      = jsonencode({
5    token = "<your github access token>"
6  })
7}
8
9resource "prefect_flow" "my_flow" {
10  name = "my-flow"
11}
12
13resource "prefect_deployment" "my_deployment" {
14  name    = "my-deployment"
15  flow_id = prefect_flow.my_flow.id
16
17  work_pool_name  = prefect_work_pool.my_work_pool.name
18  work_queue_name = "default"
19
20  pull_steps = [
21    {
22      type       = "git_clone"
23      repository = "<https://github.com/org/private-repo>"
24      branch     = "main"
25      
26      # Note: Blocks/Variables in pull_step configs will
27      # be resolved at runtime for security, so they must
28      # be defined as a Jinja templating expression
29      # <https://docs.prefect.io/v3/deploy/infrastructure-concepts/prefect-yaml#the-pull-action>
30      credentials = "{{ prefect.blocks.github-credentials.private-repo-creds }}"
31    },
32  ]
33  entrypoint = "dir/file.py:flow_name"
34}

4. Set up a schedule

Prefect deployments can run on a defined schedule. You can easily do this by defining a prefect_deployment_schedule resource and bind it to the original deployment. Here, we'll configure the deployment to trigger every 30 seconds:

1resource "prefect_deployment_schedule" "every_30_seconds" {
2  deployment_id = prefect_deployment.my_deployment.id
3  active        = true
4  timezone      = "America/New_York"
5  interval      = 30
6}

5. Apply and deploy

Running terraform apply will prompt your Terraform CLI to provision your Prefect objects.Now, watch the magic unfold!

1➜ terraform apply --auto-approve
2
3...
4
5Plan: 5 to add, 0 to change, 0 to destroy.
6prefect_block.github_credentials: Creating...
7prefect_flow.my_flow: Creating...
8prefect_work_pool.my_work_pool: Creating...
9prefect_flow.my_flow: Creation complete after 0s [id=ae1f4598-1f44-45e6-bdf3-dd6c4f4cd21a]
10prefect_work_pool.my_work_pool: Creation complete after 0s [id=7ea7ebf5-5e06-4810-9413-8bc4b1749969]
11prefect_deployment.my_deployment: Creating...
12prefect_deployment.my_deployment: Creation complete after 0s [id=e3191b6a-49d7-4a56-b6b1-5d342a46a049]
13prefect_block.github_credentials: Creation complete after 0s [id=e2a82229-1e16-43cb-8128-49241aaad1cf]
14prefect_deployment_schedule.every_30_seconds: Creating...
15prefect_deployment_schedule.every_30_seconds: Creation complete after 1s [id=4163560f-5d57-47aa-8fcf-f7853764a9d9]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Benefits of Using Terraform With Prefect

Using Terraform for managing Prefect deployments brings several benefits:

• Consistency: Everything is defined as code, ensuring your workflows work the same way across different environments.
• Reproducibility: Easily recreate or tear down deployments using version-controlled Terraform configurations.
• Centralized management: Manage your entire stack, including workflows, infrastructure, and cloud resources, using one too.

While it does require familiarity with both Terraform and Prefect's deployment model, teams already invested in Infrastructure as Code will find the benefits far outweigh the learning curve.

Declarative Prefect at Scale

In this tutorial, we showed you how to use the Prefect Terraform Provider to manage deployments. Specifically, we covered:

• Defining the Prefect Provider and authenticating with Prefect Cloud
• Setting up a work pool for workflow execution
• Retrieving flow code from GitHub
• Configuring deployments and schedules

By treating deployments as code, we gain consistency, audibility, and the ability to scale across environments.

Log in or sign up for free to start managing your deployments with Terraform.

Resources

• Prefect Terraform provider on Terraform’s registry
• Prefect Terraform provider on GitHub
• Prefect docs on Infrastructure as Code (IaC)