Big Data Workflow Automation with Prefect and Coiled
community

Big Data Workflow Automation with Prefect and Coiled

When running Prefect out-of-the-box, orchestration of your workflow is done in the cloud, while the actual computation of your code is done locally on your machine. This means that your Prefect workflows are limited by the resources of your machine. You can transcend these local machine limitations by delegating compute-intensive workloads to a Dask cluster on Coiled.