A Prefect New Year (January 2021 Updates)

Whew! Prefect has been hard at work since our last update and we've got a ton of exciting new features across the stack. In December we focused a lot of attention on streamlining execution and Agents in Core, providing sensible defaults to make getting started as easy as possible while providing all the tools you need to tailor Agents and environments to your infrastructure. Without further ado, here are some of the features we've released in the last month!

Core

RunConfigs (Environments deprecation)

One of the trickiest parts of getting started with Prefect has been figuring out the sometimes muddled responsibilities of your flows' various parts. Concepts like environments were both too flexible and not flexible enough, depending on the flow or task. We needed to rethink this design so we could provide an interface more in line with user expectations, which would allow both deep configuration and dead-simple ease-of-use. Enter stage-left: RunConfigs. Replacing environments, run configs are objects attached to your flow that determine where and how a flow run from that flow should be executed. Each run config type corresponds to an agent type and allows you to specify things like CPU and memory. RunConfigs will be replacing Environments, which are now deprecated, but don't worry! They'll be sticking around for the foreseeable future to give you plenty of time to switch.

If you want to learn more about RunConfigs, Jim Crist-Harif wrote an excellent blog post that'll walk you through all the bells and whistles of 0.14.0!

You can also check out the RunConfig docs.

Agents

Agents received a lot of love over the past month. Further supplementing the new run configs model, agents now allow you to set resource defaults like memory and CPU for any flow runs they pick up that don't have run configs.

• Agents have been significantly streamlined, filling our vision for them to be lightweight orchestration processes. This means that starting with 0.14.0 your agents will be able to spin up the necessary infrastructure and submit your flow run jobs dramatically faster.

• Kubernetes Agents now give a lot more insight into how jobs are processed and what's happening in your K8s cluster; K8s events will now be reported back via flow run logs.

Take a look at the Agent docs for specifics on each agent and what agent configuration looks like in 0.14.0!

Flow Idempotency

This is a big development for anyone registering flows as part of CI pipelines - Michael Adkins created a new method for flows (flow.serialized_hash) which generates a deterministic hash of your flow. When passed to flow.register as the idempotency_key, flows that haven't changed will no longer increment their version. Now versions will reflect true changes to your flow's structure or code. 👌🏽

UI

Run Calendar

Ever wanted to see your flow runs from a specific day in the Prefect UI? Well now you can! Some excellent work by Jenny Grange introduces a brand-new Calendar view to the UI that lets you see all the flow runs across your team, and interact with them as you might with appointments on your work calendar. We'll be building out functionality of the Calendar in the coming months, allowing you to schedule new runs, adjust the schedules of existing runs, filter views based on an agent or project, and more... stay tuned!

Interactive API

The Interactive API was introduced last year as a way to test queries or run complicated mutations directly from the UI, without needing to worry about token management or an external IDE. We didn't expect the tool to become such an integral part of many of our workflows nor did we anticipate its impact as an introduction to GraphQL for many of our users. As such, it lacked many of the features of other GraphQL explorers... but no longer! Thanks to Allyson Lubimir , the Interactive API has received a serious upgrade based on GraphiQL, the de-facto standard for GraphQL explorers. You can now see query history, store variables for use across queries/mutations, and explore the Prefect API's GraphQL schema in more depth than ever before, among many other features, right from the Prefect UI.

Artifacts

We released a much-requested and highly-anticipated feature in early December: Artifacts! The Artifacts API provides a powerful, flexible interface to publish results, charts, statistics and more from your flows and tasks and to see them rendered natively in the Prefect UI... and the best part is all of that can be done with a single line of code. Take a look at Chris White 's blog post on Artifacts or check out the Artifacts API documentation to learn more.

Task Library Additions

Dremio

Prefect now integrates with Dremio: run Dremio queries directly from your flows and pass the results to other tasks!

Check out the DremioFetch task here

Great Expectations

Built directly on top of the Artifacts API and in partnership with the Great Expectations team, Prefect now natively integrates with the Great Expectations API.

Explore the new Great Expectations tasks here

PostgresExecuteMany

Ever found yourself looping through tons of Postgres inserts and thought there must be a better way? Well you're right, now there is! You can now use the PostgresExecuteMany task to well... execute many. Batch requests and free up valuable database connections!

Explore the PostgresExecuteMany and other Postgres tasks here

Trello

Another exciting integration: you can now create Trello cards from your Prefect flows! If your flow fails and you want to let your team know they need to investigate the failure, you can do so automatically!

Check out the Trello task API here

---

Interested in contributing? Josh Meek wrote a great blog post to help you get started with contributing to the Task Library!