Workflow Automation: Empowering teams through our in-house, self-service framework
SHARE
Jun. 21, 2020

What is Workflow Automation and why is it so important in mid/large-size companies where multiple teams have similar requirements but don’t necessarily communicate about their needs.

Maikel Penz
Maikel PenzData Engineering Team Lead

A Workflow consists of steps, configured to respect a predefined order and accomplish a specific business objective. They vary from something as simple as defining an IT request process in a small company to complex data transformations aimed to deliver key business insights.

Leaving complexity aside, some characteristics are common across all of them:

  • Workflows need to run somewhere;

  • Repeatedly;

  • Triggered on a schedule or ad-hoc;

  • And sadly they will not always perform as expected.

To address the above is where the concept of Workflow Automation comes in. It can be thought as a framework that seeks to standardize and facilitate the development/deployment of workflows across an organization. These frameworks are especially important in mid/large-size companies where multiple teams have similar requirements but don’t necessarily communicate about their needs.

Posted on Jun 21, 2020
Community Post
AWS
R
Python
Prefect Cloud
AWS Fargate
Amazon ECR
Scheduling
Retries
Notifications
Mapping
Logging
Error Handling
Dynamic DAGs
Debugging
Caching

Love your workflows again

Orchestrate your stack to gain confidence in your data