Integrating Prefect & Databricks to Manage your Spark Jobs
SHARE
Nov. 3, 2020

Integrating Prefect & Databricks to Manage your Spark Jobs

Ashton Sidhu
Ashton SidhuData Scientist & ML Engineer

Prefect is a workflow management system that enables users to easily take data applications and add retries, logging, dynamic mapping, caching, failure notifications, scheduling and more - all with functional Python API. Prefect allows users to take their existing code and transform it into a DAG (Directed Acyclic Graph) with dependencies already identified.

Posted on Nov 3, 2020
Community Post
Databricks
Spark
Notebooks
Retries
Logging
Mapping
Dynamic DAGs
Caching
Notifications
Scheduling
Integrations
RELATED LINKS

Love your workflows again

Orchestrate your stack to gain confidence in your data