How to Make Your Data Pipelines More Dynamic Using Parameters in Prefect

How to pass runtime-specific parameter values to your data pipelines

January 13, 2022

What data teams can learn from finance ft. Jeremiah Lowin, CEO of Prefect

Prefect CEO Jeremiah Lowin chats with Boris Jabes, CEO of Census, on The Sequel Show podcast to discuss what data teams can learn from finance.

January 13, 2022

A modern ETL pattern using Prefect’s S3 Storage and Kubernetes runtime

This post is a tutorial on patterns for Python ETLs leveraging Prefect workflow orchestration. Prefect makes your Python jobs easy to repeat, manage, scale, and understand.

January 10, 2022

The Prefect Way to Automate Your Data Pipelines With Fivetran

Dataflow automation platform Prefect brings Fivetran connectors to its open-source orchestration project

January 10, 2022

Built In Honors Prefect in Its Esteemed 2022 Best Places To Work Awards

Built In today announced that Prefect was honored in its 2022 Best Places to Work Awards. We’ve earned a place on the list of 100 Best Remote-First Companies.

January 5, 2022

Built In Honors Prefect in Its Esteemed 2022 Best Places To Work Awards

Prefect Earns Spot on Built In’s 100 Best Remote-First Companies List

January 5, 2022

Orchestrating ELT with Prefect, dbt Cloud, and Snowflake (Part 3)

How to use Prefect and dbt Cloud with a Snowflake data warehouse

January 4, 2022

2021 Growth Report

Check out how far the Prefect community has come in our 2021 Growth Report!

December 29, 2021

Prefect is a 2021 Pipeline Academy Award Winner

The 2021 Pipeline Academy Awards (The Pipies) are brought to you by Pipeline Academy. In their words, "it is our way of praising the teams that built meaningful software solutions that serve a real purpose, the ones that are most likely here to stay to make the lives of data engineers better."

December 23, 2021

Orchestrating ELT on Kubernetes with Prefect, dbt, & Snowflake (Part 2)

This article is the second in a series of tutorials about orchestrating ELT data pipelines. The first post demonstrated how to organize and orchestrate a variety of flows written by different teams and how to trigger those in the correct order using Prefect. This post builds on that by capturing more advanced use cases and showcases how to deploy the entire project to a Kubernetes cluster on AWS.

December 21, 2021

Getting Started With Flows

This guide is here to help you step through 7 simple steps to get started running a simple flow using Prefect Cloud and your local machine. Ready to get started?

December 16, 2021

TECHBLITZ: Prefect tackles negative engineering

Prefect CEO Jeremiah Lowin sits down with TECHBLITZ, a Japanese media outlet covering startups.

December 15, 2021

Prefect’s Next Stage Ecosystem: Premier Partnership Program and Verified by Prefect

Prefect announces our new Premier Partner program, featuring newly Verified by Prefect partners Fivetran, Monte Carlo Data, dbt Labs, PagerDuty, Great Expectations, Firebolt, and Airbyte.

December 7, 2021

How to Achieve More Trustworthy Data Pipelines with the Prefect Integration for Monte Carlo

December 7, 2021

You No Longer Need Two Separate Systems for Batch Processing and Streaming

For a long time, the inflexibility of batch orchestrators and schedulers kept forcing data teams to maintain a separate set of systems to satisfy the modern event-driven, real-time data workloads. But it doesn't have to be that way. Read for a new approach.

November 30, 2021

Scaling Out With Prefect

How to set up a production workflow on Prefect.

November 28, 2021

The Nice Way To Manage Your Data Science Workflow

Learn to build an ETL pipeline with Prefect in a beginner example.

November 28, 2021

30% of DataOps Organizations Use Prefect

DatOps in Practice: A survey.

November 28, 2021

Introducing Radar

Introducing Prefect Radar: A new way of visualizing workflows that elegantly clarifies their execution at any level of scale.

November 22, 2021

Backstage + Prefect

Box presents how they integrated Backstage with Prefect during Backstage's Community Sessions.

November 17, 2021

Flow of Flows - Orchestrating ELT with Prefect and dbt

November 16, 2021

Orchestrate a Data Science Project with Prefect

Optimize Your Workflow with a Few Lines of Code

November 12, 2021

Jeremiah Lowin on the Digital Anarchist Podcast

November 8, 2021

Jeremiah Lowin returns to the Data Sharks Podcast #19

November 8, 2021

Prefect on the OSS Startup Podcast

November 8, 2021

How to Cut Your AWS ECS Costs with Fargate Spot and Prefect

Creating serverless data engineering pipelines using Python.

November 1, 2021

The Prefect Way to Automate and Orchestrate Data Pipelines

Community member Thuwarakesh Murallie, a Data Scientist at Stax, Inc., presents a great beginners guide for getting started with Prefect.

October 28, 2021

Prefect Pipeline Code for Building a Basic Data Lake

The Prefect code you need to get started building a data lake on AWS.

October 25, 2021

Building a Basic Data Lake with Prefect, S3, Glue and Athena

The story of how Smooth Sensors built a data lake on AWS with Prefect, S3, Glue and Athena.

October 25, 2021

A Mathematician Dove Into Technology Head First

Hear Chris White cover his journey into data via mathematics on the Coruzant Technologies podcast.

October 15, 2021

When to start a formal product organization?

Prefect CTO Chris White covers the steps to building a product organization at a growing company.

October 15, 2021

Livestream - Future of Prefect

The return of Laura Lorenz! Join Laura for another spooky October livestream as she takes Orion for a spin. Laura will highlight the new API, subflows and caching, as well as the killer new UI and deployment details.

October 8, 2021

Prefect Orion: Our Second-Generation Workflow Engine

Love your workflows again.

October 5, 2021

A People's History of Kubernetes

The 30th anniversary of Linux gives us a chance to reflect on the evolution of open source and how it has transformed the corporate landscape of technology-makers.

October 5, 2021

Bootstrapping In America: Jeremiah Lowin of Prefect

Prefect is a workflow management system that helps to eliminate data flow problems for companies’ data teams. They provide a new dataflow automation platform that reduces engineering time by automating pipelines and workflows of any complexity.

October 4, 2021

Prefect Acquires SLATE Data

We've acquired SLATE Data, a premier data engineering company, to expand our service and support offerings and deliver even more value to our users.

October 4, 2021

A Demo of Prefect

A simple demo of getting started with Prefect, including code snippets, from @Creative-Force

October 1, 2021

Building a Competitive Data Architecture, One Technology at a Time

It's time to rethink data architecture. The architectures that have been built over the years were suitable for on-premise, legacy environments but tended to be static, inflexible, and accessible to only a few. Today's data-driven organizations demand capabilities that adapt to the enterprise and open new paths of innovation to business users.

September 27, 2021

How to make sure your design team gives good remote feedback

Asking everyone to keep their cameras on is a good start, but leaders need to do more than focus on the technical aspects when running remote critiques.

September 27, 2021

Scaling Your Prefect Workflow to the Cloud

When running Prefect out-of-the-box, orchestration of your workflow is done in the cloud, while the actual computation of your code is done locally on your machine. This means that your Prefect workflows are...

September 22, 2021

Big Data Workflow Automation with Prefect and Coiled

When running Prefect out-of-the-box, orchestration of your workflow is done in the cloud, while the actual computation of your code is done locally on your machine. This means that your Prefect workflows are limited by the resources of your machine. You can transcend these local machine limitations by delegating compute-intensive workloads to a Dask cluster on Coiled.

September 22, 2021

Applying Practical Mathematics with Chris White of Prefect

Our guest this week has a Ph.D. in Mathematics. Chris White was the first hire and is now CTO of the rapidly growing company, Prefect.

September 20, 2021

The Simple Guide to Productionizing Data Workflows with Docker

In this walkthrough, we’ll provide an end-to-end minimal example of how to packag Python work into Docker containers for production.

September 9, 2021

Prefect: Zero to Hero

From no experience with data tooling to fully automated Prefect pipelines

August 27, 2021

Inside Prefect: Reflections on my Internship Experience

A peek behind the curtain from a software engineering intern

August 27, 2021

Prefect: Should you utilise the next generation of data pipelining software?

A mature market like data orchestration software requires something special to move a company from its current tooling. However, we believe there are several core features that showcase the benefit of using Prefect.

August 17, 2021

Prefect: How to Write and Schedule Your First ETL Pipeline with Python

July 22, 2021

Prefect x Kubernetes x Ephemeral Dask: Power, without responsibility

Prefect is rapidly becoming the standard in dataflow automation. In our last blog we briefly ran through using Prefect Executors to parallelize on a single node.

July 19, 2021

Workflow Orchestration with Prefect and Coiled

July 16, 2021

The Data Exchange Podcast: Chris White on building an engine that enables arbitrary proprietary workflows.

July 15, 2021

Parallel Prefect: A bit about Dask, real fast

The problem: I heard Prefect can run Python in parallel, what is the simplest way to achieve this?

July 8, 2021

Prefect 0.15.0: A New Flow Run Experience

An overhauled flow run CLI, sub-flow result passing, agentless execution, improved run observability, new auth patterns, and more!

July 1, 2021

Data Sharks: Jeremiah Lowin of Prefect

The Data Sharks discuss the future of Data Orchestration and how Prefect provides the easiest way to automate your data with incredible flexibility.

June 18, 2021

Escape Velocity: Prefect's Series B

Following our $32M Series B, we're scaling our team, building new products, and forming A More Prefect Union.

June 10, 2021

Prefect Announces $32M Series B Funding Led by Tiger Global to Deliver on Growing Demand for Dataflow Automation

"Any engineer delivering custom code for their organization can easily reap the benefits of Prefect's platform, creating a massive addressable market for the company to capture." Official press release

June 10, 2021

Data flow automation engine Prefect raises $32M

"Prefect’s success aligns with broader industry trends in digital transformation...data flows have raised the global GDP by at least 10% over a decade. The value totaled $7.8 trillion in 2014 alone, contributing to economic growth primarily by raising productivity." Coverage by VentureBeat

June 10, 2021

Prefect raises $32M for dataflow automation technology

"We see the data stack as a whole just exploding in terms of attention and the need for Prefect is growing exponentially." Coverage by TechTarget

June 10, 2021

Prefect Secures $32 Million for Its Dataflow Automation Platform

“Our mission is to help data engineers and data scientists eliminate negative engineering by delivering a world-class orchestration tool for today’s modern data stack" Coverage by StartUp Savant

June 10, 2021

D.C. startup raises tens of millions of dollars in second venture round of 2021

"The new capital will let Prefect hire up, grow its commercial business and further develop its platform" Coverage by Washington Business Journal

June 10, 2021

DC’s Prefect already raised two funding rounds in 2021. Here’s what’s next for the dataflow automation company

"We are chasing our own growth." Coverage by Technical.ly

June 10, 2021

Prefect Raises $32M To Be The ‘Toothpick’ In Companies’ Data Stack

“Suitors will always come, but Prefect has all of the elements to be an enduring business that forges its own path,” said Ethan Kurzweil of BVP. Coverage by Crunchbase News

June 10, 2021

Introducing: The Prefect KV Store

A flexible building block for workflow metadata

June 3, 2021

MLOps: Task and Workflow Orchestration Tools on Kubernetes

If you need a simple and clear answer―use Prefect.

May 28, 2021

Orchestrating Image Preprocessing Workflows for Label Studio with Prefect

Kevin Kho, Open Source Community Engineer at Prefect, showcases Prefect workflows with Label Studio

May 26, 2021

Prefect + Microsoft: Advancing Dataflow Automation Together

Unveiling the first details of our deepening relationship with Microsoft.

May 21, 2021

Enhance positive engineering with Prefect and Microsoft for Startups

We are delighted to welcome Prefect into the Microsoft for Startups portfolio. Working with Prefect will help our joint customers easily deploy on trusted infrastructure with the convenience of Prefect Cloud.

May 21, 2021

If you give a data engineer a job...

May 11, 2021

Don’t Panic! The Prefect Guide to Building a High-Performance Team

Building the startup is just as important as building the product. Here's how we establish a high-performance and collaborative culture.

April 28, 2021

Introducing a Dataflow Management System Backed Up by Prefect, AWS, and Github Actions

Take your local Prefect workflows to the next level by provisioning AWS & Prefect Cloud infrastructure, with Github Actions playing an important role.

April 27, 2021

Best Workflow and Pipeline Orchestration Tools – Machine Learning Guide

"I believe that [Prefect] is one of the best automated workflow management tools out there."

April 20, 2021

Data Engineering & Business Intelligence – An Exciting Quest

Prefect. dbt. Redshift. BI tools. Learn about Cloud Academy Staff Engineer Giuliano Sofi's (literally) epic journey to redefine and transform their data and reporting!

April 16, 2021

Building a Healthy Open Source Community with Jeremiah Lowin, Founder and CEO of Prefect

Join Jeremiah Lowin and Anu Sharma for a deep dive into open-source community building from 0 to 5,000.

April 9, 2021

Simple, Success-Based Pricing

Eliminate negative engineering with Prefect Cloud’s new success-based pricing, a generous free tier, and automatic discounts.

March 24, 2021

Liftoff: Prefect's Series A

Welcoming a new cohort of strategic investors.

February 25, 2021

GitHub CEO Joins Prefect’s $11.5 Million Funding Round for Dataflow Automation

Watch Jeremiah Lowin's interview about our Series A, future plans, and upcoming announcements!

February 17, 2021

Prefect Raises $11.5M in Series A Funding to Eliminate Negative Engineering in Dataflow Automation

February 9, 2021

Liftoff

A preview of our upcoming announcements, including our Series A, Microsoft partnership, updated product vision, and new community programs.

February 9, 2021

Serverless Data Pipelines Made Easy with Prefect and AWS ECS Fargate

Even though there are so many workflow orchestration solutions and cloud services for building data workloads, it’s hard to find one which is actually pleasant to use and allows you to get started quickly. One of my favorite tools for building data pipelines in Python is Prefect — a workflow management platform with a hybrid agent-based execution model.

February 8, 2021

Prefect Workflow Automation with Azure DevOps and AKS | Infinite Lambda

February 5, 2021

Prefect Workflow Automation with Azure DevOps and AKS

The team at Infinite Lambda shares how to automate Prefect flow deployments on Azure.

February 5, 2021

Understanding Great Expectations and How to Use It

Use Prefect and Great Expectations to load valid data into Snowflake.

February 4, 2021

Getting Started with Parallel Computation

Prefect provides many simple and expressive building blocks that users can snap together to add advanced features to their workflows. This post shows how easily Prefect tasks can be executed in parallel without requiring deep knowledge of distributed computing. In fact, it only requires two extra lines of code!

January 20, 2021

What We Love About Prefect

It’s open source, plays nicely with Kubernetes, and there’s a great community! We love how easy it was to set this up, and the clean UI and dashboards make it a pleasure to use.

January 15, 2021

Livestream - 0.14.0: your deploy just got easier

December 18, 2020

Prefect 0.14.0: Improved Flow Run Configuration

We’re excited to announce the release of Prefect 0.14.0

December 17, 2020

Introducing: The Artifacts API

Create native Prefect UI integrations with one line of code

December 8, 2020

Getting started with operationalizing your Python code

In the first part of this blog series we saw how quickly Prefect can help you run parallelized Python code from a fresh OS install

December 2, 2020

Scheduling & Optimizing RAPIDS Workflows with Dask and Prefect

Scheduling and optimizing accelerated data science workflows is really easy with Prefect, RAPIDS, and Dask. Get started with this tutorial.

November 9, 2020

Integrating Prefect & Databricks to Manage your Spark Jobs

This tutorial explains how to run Databricks notebooks and Spark jobs in Prefect flows.

November 3, 2020

Accelerating modelling with Frea: Paidy's in-house feature store with Prefect and PostgreSQL.

We’ve recently partnered with Prefect to make it easy for us to build scalable solutions for data science at Paidy that make sense for our team and ways of working. This blog post talks about the in-house Feature Store known as Frea which is built on top of the Prefect platform, and which reduced the model training pipeline time from weeks to hours.

October 26, 2020

How to Automate Python Workflows in Prefect (Step-by-Step Guide)

There was quite a steep learning curve for me to pick up Airflow. Thankfully, in the midst of my struggles, I discovered a library that would make the entire process ridiculously simple. Enter Prefect.

October 21, 2020

The Prefect Task Library

The Prefect community is a welcoming place for developers of all experience levels, and our task library is an excellent way to get involved with contributing to an open-source project.

October 8, 2020

How Prefect Kept my Family Cool in the Pool this Summer

The data workflow platform Prefect is used commercially for serious data engineering and machine learning pipelines. It's being rapidly adopted in industries like financial services, e-commerce, healthcare, and many others.

September 22, 2020

Slate Data Transforms its Business Model with Prefect Cloud

Slate Data provides data engineering services to reinvigorate data-driven business decisions by modernizing data tech stacks and empowering industry-specific data scientists.

September 17, 2020

Managing dependencies between data pipelines in Apache Airflow & Prefect

A simple approach to managing dependencies between your workflows.

September 4, 2020

Distributed data pipelines made easy with AWS EKS and Prefect

Anna shows you how you can obtain a highly available, scalable, distributed system that will make the orchestration of your data pipelines for ETL & ML much more enjoyable

August 25, 2020

The All-New Prefect Server and UI

Two key Prefect products are becoming open-source projects in our biggest release ever.

August 6, 2020

Workflow Automation: Empowering teams through our in-house, self-service framework

See how the Journal team used Prefect to build a self-serve automation framework for their data teams

July 20, 2020

Introducing: Prefect Universal Deploy

We’re excited to announce our best release yet: Prefect 0.8.0

December 11, 2019

Snowflake: the details of our first Data Warehousing project in the Cloud

Learn how Trade Me adopted Prefect to build a workflow for Snowflake.

July 28, 2019

Toward Dataflow Automation

Prefect Cloud is coming

June 10, 2019

Why Not Airflow?

An overview of the Prefect engine for Airflow users

May 1, 2019

Pipeline Pitfalls

Data engineers are often responsible for the planning, building and maintaining of data pipelines. During that process, they may face the challenging decision to either create a custom system or use an existing framework.

May 1, 2019

Prefect is Open-Source! 🎉

Today, on its two-year anniversary, we’re excited to announce that Prefect has launched as an open-source project with an Apache 2.0 license.

March 24, 2019

Prefect Runs on Prefect

Every morning, Marvin runs standup for our team. That’s not unusual for a tech startup, except that Marvin is actually a Prefect workflow.

January 17, 2019

The Golden Spike

A Prefect Update

January 9, 2019

Positive and Negative Engineering

Don't panic.

October 1, 2018