February Updates: A new way to run (your flow)!

Jenny Grange | 2021-02-16

We’re excited to share some new updates to Prefect for February! TL;DR: A new run flow page; flow READMEs; an air gapped version of the UI; a new collaboration with Fivetran and support for extra pip packages in the Prefect Core image! 😁

Run Flow

Following on from the introduction of RunConfigs in Core, we’ve updated the UI so you can update RunConfigs on a per run basis.  The new run page should make it easier for users to change Docker images; to send test runs to development clusters and try out a range of different deployment patterns.

It also lets you randomize your run name, generating gems like this 🦚 🦤 💃
It also lets you randomize your run name, generating gems like this 🦚 🦤 💃

You can now edit the Docker image or environment variables you use for a run when using a Docker RunConfig or specify a different Task role ARN or Execution Role ARN when running your flow with AWS ECS. The new page also includes extra help with your JSON formatting and an improved calendar that lets you select the timezone you want to schedule your flow run in. 

Who knew writing JSON could be so validating
Who knew writing JSON could be so validating

Air-Gapped UI

Prefect Server now bundles fonts and icons as part of the build to provide an air-gapped UI. Air-gapping should allow users to run the Server UI in environments that have no access to the internet with no further configuration needed.

Tom Cruise is probably checking his flows on air-gapped FBI computers
Tom Cruise is probably checking his flows on air-gapped FBI computers

Flow READMEs

Have you ever wanted to tell your team-mates (or your future self) all about your flow and how to run it?  Now you can add a readme to your flow in the UI to include all the important parameters, RunConfig, secrets (not the actual values!) information that it would be useful to know before running your flow. 

Everyone's favorite pastime: documentation!
Everyone's favorite pastime: documentation!

The FivetranSyncTask

Step 1: Prefect + Fivetran; Step 2: ???; Step 3: Profit
Step 1: Prefect + Fivetran; Step 2: ???; Step 3: Profit

Story time! The good folks at Fivetran reached out to us to discuss a possible collaboration opportunity; we promised them that integrating with Prefect was incredibly simple (as it should be!) and within 48 hours we had an open PR as a testament to that fact.

If you load data through Fivetran, then the new FivetranSyncTask will help manage your Fivetran connector sync process. The task starts and monitors a Fivetran sync job and should help make complicated data syncs less complicated!

We hope that this is just the beginning - if you're a Prefect + Fivetran user and have feedback about this task specifically or if you would like to propose an integration enhancement, please open an enhancement request and let's talk!

Extra Pip Packages

Is it Prime delivery day?
Is it Prime delivery day?

Prefect now supports an EXTRA_PIP_PACKAGES environment variable in our official Docker images. The variable takes a list of extra packages to pass to pip install on image startup, making it easier to install extra packages. So if you have a flow that requires packages such as “my-extra-package1” that aren’t included in the official images, you can add them as part of your RunConfig: 

flow.run_config = DockerRun(env={"EXTRA_PIP_PACKAGES": "my-extra-package1 my-extra-package2"})

Look out for another Prefect Changelog update in March! There's lots of exciting Prefect news to come and you can see a preview in Liftoff, our latest Prefect blog post. You can also stay up to date with Prefect updates by following @PrefectIO on Twitter; joining us on Slack; and checking the Prefect Core, Server, and UI repos!