
May 13, 2022
Prefect 2.0b4: A slew of updates
Community contributions, flow and task runner improvements, collections, QOL, bug squashing and more.
We're really busy over here at Prefect! We've been getting great feedback from early adopters. There's a lot of work going on behind the scenes as we work on building some exciting new features that will be exclusive to Prefect 2.0, but we want to keep the enhancements flowing to you. In that spirit, there are a lot of quality-of-life improvements here!
Give our latest release a go yourself:
pip install -U "prefect>=2.0b4"
Docs to get started here. You can also join the discussion about this release on Discourse, or in our public Slack community!
While most of the development of Prefect 2.0 is still happening internally, we're incredibly excited to be getting contributions in our open source repository. Big shoutout to our contributors for this last release:
@dannysepler
@ColeMurray
@albarrentine
@mkarbo
@AlessandroLollo
Flow and task runners
Flow runners now pass all altered settings to their jobs instead of just the API key and URL
The Kubernetes flow runner supports configuration of a service account name
The subprocess flow runner streams output by default to match the other flow runners
The Dask task runner has improved display of task keys in the Dask dashboard
The Dask task runner now submits the execution graph to Dask allowing optimization by the Dask scheduler
Note that the Dask and Ray task runners will be moving out of the core Prefect library into dedicated prefect-ray
and prefect-dask
collections with the next release. This will reduce the number of dependencies we require for most use cases. Since we now have concurrent execution built in to the core library, these packages do not need to be bundled with Prefect. We're looking forward to building additional tasks and flows specific to Ray and Dask in their respective collections.
Collections
Speaking of collections, we've received our first user-contributed collection. It includes tasks for Cube.js, check it out!
The following collections have also been recently released:
You can see a list of all available collections in the Prefect Collections Catalog.
Windows compatibility
We've excited to announce that we've begun work on Windows compatibility. Our full test suite isn't passing yet, but we have core features working on Windows. We expect the majority of the edge cases to be addressed in an upcoming release.
Documentation improvements
We've added some new documentation and made lots of improvements to existing documentation and tutorials:
Added documentation for associating conda environments with separate Prefect profiles
Added storage steps and advanced examples to the Deployments tutorial
Expanded documentation of storage options
Added workspace details to the Prefect Cloud documentation
Improved schedules documentation with examples
Revised the Kubernetes tutorial to include work queue setup
Improved tutorial examples of task caching
CLI
Deployments can be deleted from the CLI
The CLI displays help by default
prefect version
is robust to server connection errorsprefect config view
shows sources by defaultprefect deployment create
exits with a non-zero exit code if one of the deployments fails to be createdprefect config set
allows setting values that contain equal signsprefect config set
validates setting types before saving themprefect profile inpect
displays settings in a profile instead of duplicating prefect config view behaviorprefect storage create
trims long descriptions
Bug squashing
We've eradicated some bugs, replacing them with good behavior:
Flow runs are now robust to log worker failure
Deployment creation is now robust to
ObjectAlreadyExists
errorsFutures from async tasks in sync flows are now marked as synchronous
Tildes (~) in user-provided paths for
PREFECT_HOME
are expandedFixed parsing of deployments defined in YAML
Deployment deletion cleans up scheduled runs
Optimizations and refactors
You might not see these fixes in your day-to-day, but we're dedicated to improving performance and maintaining our reputation as maintainers of an approachable and clean project.
The
state_name
is attached to run models for improved query performanceLifespan management for the ephemeral Orion application is now robust to deadlocks
The
hello
route has moved out of theadmin
namespace so it is available on Prefect CloudImproved readability and performance of profile management code
Improved lower-bounds dependency parsing
Tests are better isolated and will not run against a remote API
Improved representation of Prefect
Setting
objectsAdded extensive tests for
prefect config
andprefect profile
commandsMoved testing utilities and fixtures to the core library for consumption by collections