Prefect x Kubernetes x Ephemeral Dask: Power, without responsibility
SHARE
Jul. 19, 2021

Prefect is rapidly becoming the standard in dataflow automation. In our last blog we briefly ran through using Prefect Executors to parallelize on a single node.

George Coyne
George CoyneSenior Solutions Architect

Prefect is rapidly becoming the standard in dataflow automation. In our last blog we briefly ran through using Prefect Executors to parallelize on a single node. The question that usually follows single node parallelization is:

How do we parallelize across multiple nodes.

The goal of this blog is to provide a minimal example of how to(with as little external configuration as possible) parallelize across multiple nodes using an ephemeral Dask cluster. In this example we will be using Kubernetes for Flow scheduling, but the pattern is similar with ECS.

Posted on Jul 19, 2021
Community Post
Kubernetes
Dask
Coiled
Docker
Scheduling
DevOps & CI/CD

Love your workflows again

Orchestrate your stack to gain confidence in your data