Prefect’s Next Stage Ecosystem: Premier Partnership Program and Verified by Prefect
Dec. 7, 2021

Prefect announces our new Premier Partner program, featuring newly Verified by Prefect partners Fivetran, Monte Carlo Data, dbt Labs, PagerDuty, Great Expectations, Firebolt, and Airbyte.

Andrew Black
Andrew BlackHead of Partnerships

As a dataflow automation platform, Prefect’s power lies in connecting tools across data stacks so that data can be made available easily and reliably. This means that integrations aren’t just important to Prefect, they are fundamental to what Prefect does. Together with these products, Prefect is able to help data engineering teams deliver timely and accurate data for the reporting, analytics, and ML use cases organizations need to make fast, data-informed decisions.

Prefect has over 50 integrations today supporting an array of use cases. To date, these have all been contributed and maintained by our community of 11,000+ members. An average of three new integrations are submitted every month.

As impressive as this is, the need for more and deeper integrations continues to grow as our customer base and use cases expand. Plus, our enterprise customers need guarantees that every release of Prefect will continue to be compatible with their data stacks.

To address these needs, today we’re announcing a new technology partner program that will level up our integrations and working relationships with a select group of platforms our users value most today and are excited about for the future. These tasks will be officially maintained and supported by Prefect (in conjunction with the partner). In addition, new communication channels for sales and support have been built, so buying, implementing, and maintaining our products together will be better coordinated for a seamless experience. Data engineers can also look for more joint thought leadership with these companies to advance the practices and knowledge in our communities.

In taking this approach, our goal is to deliver a better joint product experience that helps data engineers deliver accurate, timely data to business users, and have more fun doing it.

  • Integrations with Premier Partners will now be “Verified,” meaning that Prefect and the partner have inspected the code and tested for compatibility. These tasks will be designated with a “Verified” tag in the Prefect Docs.

    • Importantly, community members will still be encouraged to contribute tasks. In fact, we’ll be better equipped to offer them technical help and coordination between the parties.

  • Better joint experience: Support, sales, and customer success will all be easier, with faster issue/bug resolution.

  • Aligned roadmaps: We’ll regularly examine each others’ roadmaps to see how we can continually open up more functionality and make our products easier to use together.

This program will continue to grow and evolve as we learn together with our partners, communities and customers. Our goal is to ensure every data stack component and tool can be orchestrated and monitored by Prefect.

Without further ado, our launch Verified program partners are: dbt, Fivetran, Monte Carlo, Firebolt, Great Expectations, PagerDuty, Soda, and Airbyte. These are companies and products we believe in and, frankly, we just really like working with.

What can you expect from these integrations? As a baseline, every one of these platforms can be orchestrated by Prefect as part of a flow, and they will return status updates and other information as enabled by the API. In many cases, data about the task will be represented as an artifact in the Prefect UI. And we’ll also be pushing metadata such as flow run ID, times, transformations applied, and sources for those tools where it makes sense, such as data observability, cataloging, and lineage.

And as always, please provide your ideas and feedback!

Note: Not every integration is available today, but they all will be very soon, as well as others.

  • Airbyte - Airbyte is the fastest-growing open-source data integration platform, helping organizations sync data from APIs, databases, and files to data warehouses, lakes, and other destinations. With more than 8,000 organizations adopting Airbyte in just its first 16 months of existence, its momentum is impressive—and so is the enthusiasm for it in our community. Using Prefect and Airbyte together, data engineers will be able to build and manage ETL pipelines even faster and more easily. See the Airbyte integration here.

  • dbt Labs - dbt is among Prefect’s most used integrations. We’ve updated and validated the open-source task, and added one for dbt’s cloud offering. Importantly, this task will not just enable you to trigger a dbt task, but can also write artifacts to Prefect to record all of the associated metadata about when and how the dbt job ran, and its outcome. We can’t wait to see what our joint users do with this new capability. See the dbt integration here.

  • Firebolt - Firebolt has come hot out of the gates with a truly innovative data warehouse that is not only super fast, but also aimed at data engineering use cases. Our joint vision of making data engineers’ jobs easier and more enjoyable could not be better aligned. Look for a release in January 2022.

  • Fivetran - Fivetran is a newer partner in our ecosystem, and we’re finding tremendous momentum in our enterprise customer base using our platforms together. Our joint users love Fivetran’s simplicity and dependability where predefined app-to-app ELT is what’s called for. When more customized dataflows are needed, they can be built with Prefect. And now users can orchestrate Fivetran with Prefect as well, for a single view of the entire organization’s dataflows. See the Fivetran integration here.

  • Great Expectations - Another integration that’s heavily used with Prefect, Great Expectations helps data teams identify data that is not in line with, well, expectations. Our users love the openness and flexibility of Great Expectations, and we have a lot of similar DNA with our open-source roots. We’ve updated and validated the Prefect task to be compatible with Great Expectation’s latest version, and added an integration with their emerging Cloud service. See the Great Expectations integration here.

  • Monte Carlo - We were introduced to Monte Carlo through one of our SI partners, who described Monte Carlo as “magic.” When we tried it, we instantly agreed. Monte Carlo’s data observability platform makes detecting and addressing errors in data a point-and-click exercise. Using Prefect and Monte Carlo together, data teams can ensure that their pipelines and the data itself are delivered on time and accurately. (GA of the Monte Carlo integration is coming Jan 2022. Let us know in our Slack community if you’d like to try a pre-release version.)

  • PagerDuty - Speeding issue resolution time is a really important benefit Prefect can offer data teams, and PagerDuty has set the standard in this space. While previously Prefect could offer customizable notifications through Slack or email, this partnership allows our joint customers to send notifications directly through PagerDuty for sophisticated routing, logging, and updates. The integration will also make it easier for data engineers to go the other way, that is, work from issue up through Prefect to identify and resolve the dataflow issue at the source. See the PagerDuty integration here.

We’re really excited to offer this program for our users, as well as learn from these amazing companies. Our early users have already built some amazing solutions with these platforms, and we’ll be sharing their stories throughout 2022.

As excited as we are for this milestone, this is just the beginning of our journey of making the Prefect ecosystem the most comprehensive, capable, and valuable in datadom. Happy engineering!

Andrew Black

Head of Partnerships

You can reach me @AndrewBlack in our Slack community

Posted on Dec 7, 2021
Blog Post
Monte Carlo
dbt Labs
Pager Duty
Great Expectations

Love your workflows again

Orchestrate your stack to gain confidence in your data