Introducing: the Prefect Public Changelog

Chris White | 2020-09-28
Prefect Logo
Prefect Logo

Hello folks! As many of you know, the Prefect Team maintains a fast pace of development. You may have noticed that Prefect Core is released weekly (typically on Tuesday or Wednesday), but unless you watch closely you might have missed that both Prefect Cloud and the Prefect UI are released almost daily! While we have tried our best to keep our constantly growing community up-to-date on our latest and greatest features, many have requested a more durable location to look for updates.

With that, I'm happy to announce the Prefect Public Changelog! We will use the public changelog to describe and socialize any interesting new features or enhancements that we think you might have missed. We plan to release changelog updates roughly every two weeks.

In addition to the new changelog, there are many other places you can peruse for updates and info:

With all of that out of the way, let me catch you up on some of the new features and enhancements that we've been hard at work on!

Prefect Core, Server and UI

Now that Prefect Server and the UI are standalone repositories, I'd like to take this opportunity to explain how releases for these work and how they relate to Core releases. Both the Server and UI are versioned with a Calendar Versioning scheme.

Prefect Cloud is always built with the latest Server release as a dependency, and is released on a much faster cadence than Server (note that you can see when Cloud and the UI were last released in the left menu in the UI!).

The Prefect UI, on the other hand, is 100% open source - each tagged release of the UI is auto-deployed to the production Prefect Cloud environment.

Whenever Prefect Core is tagged for release, it will pull the latest released Docker images for both Server and the UI and retag them so that every time you run prefect server start from a specific version of Core, the appropriate images are pulled.

For those that are feeling adventurous, both Prefect Server and the UI release Docker images with the "master" tag that can be used for development. Check out Prefect's Docker Hub page for all available images and tags.

Flow Concurrency Limiting by Label

There are many situations in which users need to limit the number of flows of a certain type that are running concurrently. For example, if you have a group of flows that are all run within a tightly controlled Kubernetes cluster, you might seek to limit how many can run at the same time to avoid over utilization of resources.

Flow concurrency limiting is designed to enable this in a configurable way: you can now set concurrency limits on individual flow labels via the GraphQL API or the UI. The UI management screen for this feature can be found by navigating from the left menu to Team > Flow Concurrency. When a limit is hit, new runs associated with the limited label will simply queue up until a slot becomes available.

Flow Run Cancellation

By popular demand, Prefect now supports active cancellation of flow runs! This feature works on a best effort basis depending on your configured execution environment. The easiest way to test this feature out is by clicking the red "Cancel" button on the top right of the Flow Run UI page. Alternatively, you can call the cancel_flow_run GraphQL route with a given flow run ID. Your flow run will first enter a "Cancelling" state as its execution environment attempts to halt the execution of all currently running tasks. Please allow some time for this to occur! Once all tasks have been accounted for, your run will enter a terminal "Cancelled" state.

Please note that this feature is only available for flows running on versions 0.13.0 and above.

Flow Groups: Schedules, Parameters and Labels

The Prefect Team has been adding many new features that allow you to configure your flow natively in the UI. In order to achieve this, we need to introduce the concept of a "Flow Group". A flow group represents a versioned collection of flows and provides configurable settings that override the settings on your individual flows within the group. Moreover, these settings will persist across re-registrations until you clear them.

Currently, Flow Groups allow you to override the following properties of your flows:

  • Schedules

  • Labels

  • Default Parameter values

  • Cloud Hooks (i.e., state change notifications)

For more information on where to configure these in the UI, check out the UI documentation here.

Flow Label Visibility

In addition to the ability to edit your flow's labels from the UI, the UI team also recently added a new feature that will highlight flows and flow runs that have labels with no actively running agent associated with them. We hope this will help users debug late runs and educate new adopters on how flow label affinity works!

Stay Tuned!

Stay tuned for more updates! The Prefect team is growing and our feature list will only continue to expand.

Happy Engineering!