For the first ever State of Workflow Orchestration Report, Prefect partnered with Gradient Flow to survey almost 600 data practitioners from a wide variety of industries. We explored the current and future state of workflow orchestration, focusing on the priority uses cases and critical features of orchestration tools.

Here are some key highlights from the report, along with our predictions for the future of Workflow Orchestration:

Key Highlight: It should be no surprise to those following the eruption of the modern data tooling landscape that Data Engineers are in high demand. With a 57% increase in unique job postings year over year, the demand for Data Engineers is poised to overtake the demand for Data Scientists this year.

🔮 Our prediction: With the growing complexity of the modern data stack, we are seeing a rise in roles farther away from the end data product - the insights - and closer to the infrastructure and maintenance of data pipelines. With Data Engineers solidified as a mission-critical member of the data team, we predict a burgeoning need for DataOps Engineers in the near future.

Key Highlight: Orchestration tools really are the Swiss Army Knives of the Data/ML Engineers' toolbox: 91% of Data/ML Engineers said that they use orchestrators for more than a quarter of all recurring tasks. Although more Data/ML Engineers use Airflow (32%) over Prefect (17%) currently, if given the option to start a new project, the same Data/ML engineers would choose to start with Prefect (18%) over Airflow (15%).

🔮 Our prediction: Companies have already started including experience with workflow orchestrators as a required skill in job descriptions for engineering roles. However, companies should also expect discriminating candidates to be more excited about opportunities that leverage the most innovative solutions. Join our Slack community to see which modern companies are looking for engineers skilled in dataflow automation!

Key Highlight: Data Science is the primary use case for orchestration (29%). Transformation (13%) and Movement (12%) come in next as the legacy use cases for orchestration, but DataOps (12%) is identified as a new but impactful priority for data teams as well.

Key Highlight: Among all respondents, Ease of use (38%) is most commonly identified as an important feature, followed closely by caching and monitoring (37%). In particular, 51% of Data Analyst/Scientists users identified Caching as one of the most important features for an orchestrator.

🔮 Our prediction: The future of Dataflow Automation is already here. With big data comes big data challenges, and the modern orchestration tool needs to be able to flexibly work with not just processes and workflows, but also elegantly manage the valuable data being passed between tasks and jobs. Learn more about how Prefect supports Data Science/ML uses cases by providing multiple ways to cache and persist data between tasks or flows.

Access the full Report here and don't forget to share your thoughts with us on the current and future state of Workflow Orchestration!