Announcing the Terraform Module to Deploy the Prefect Docker Agent on AWS EC2
Mar. 24, 2022

This blog post will walk through how to deploy the Docker agent on AWS EC2 using Terraform.

Jamie Dick
Jamie DickSenior Infrastructure Engineer

The Prefect Agent is a lightweight process that orchestrates flow runs. During operation the agent process queries the Prefect API for any scheduled flow runs, and allocates resources for them on their respective deployment platforms.

This blog post will walk through how to deploy the Docker agent on AWS EC2 using Terraform. The newly released EC2 agent module deploys the agent in a resilient and fault-tolerant fashion leveraging services like an autoscaling group to ensure the agent can withstand zonal failures. It also takes into account AWS best practice configurations by applying least-privilege access through fine grained IAM policies & roles. Below I will walk through deploying the agent with this module in its simplest form, leveraging the default values and conditions defined. Skip down to more advanced configuration options where you can bring your own network or customize the agent configuration itself.

First—a prerequisite:

Log into Prefect Cloud and generate an API service account key. This allows the agent to communicate with Prefect cloud to poll for jobs, share status updates, and more.

Once generated, store the API key in AWS secrets manager via the console, or by using the following CLI command:

$ aws secretsmanager create-secret --name prefect-api-key --secret-string "{\"key\":\"API_KEY_HERE\"}"

Now that our secret's in place—let’s deploy the agent! This module was written in such a way that by default, we can provide it with zero inputs and it will build an agent with all necessary ancillary dependencies out of the box. Our Terraform file will contain the following:

If you’re new to Terraform—check this out to get Terraform installed and your AWS provider configured.

We’ll run terraform init to initialize our working directory containing Terraform configuration files.

Next, run terraform apply which will automatically create an execution plan and prompt you to approve it before deploying the infrastructure to your AWS account.

You may want to provide further customization to this deployment to fit your specific requirements and environment. For example—you may have a VPC already built that you want to deploy the agent into or you may want to apply some configuration to the agent itself, like labels. No problem! You can do so with this module by simply providing inputs to the variables you want to customize. See the readme for a complete list of exposed parameter options. We can use the below code to deploy our agent into a pre-built VPC, label the agent with an environment tag, and enable local flow logging.

And that’s it!

Check out this link for a few more examples and all the necessary Terraform code to support the deployment.

Happy engineering!

Posted on Mar 24, 2022
Blog Post
DevOps & CI/CD

Love your workflows again

Orchestrate your stack to gain confidence in your data