It’s SNOWing in DC: Prefect and Snowflake Partner for a Better Modern Data Stack
blog

It’s SNOWing in DC: Prefect and Snowflake Partner for a Better Modern Data Stack

Andrew Black | Head of Partnerships

April 28, 2022

Snowflake needs no introduction. It has become the standard for cloud data storage for thousands of data-driven organizations around the world. It’s also among the most popular – and loved – data warehouses in our developer community. So as shared in a press release today, we’re super excited to announce we have joined as official partners to make our products and experiences even better together.
Orchestration + Data Warehousing: A Match Made in the Modern Data Stack
For those who are not familiar with data warehouses or data workflow orchestration, it’s worth explaining why this pairing is so important and impactful. No matter what their data stack, organizations have to have a way to move data around and perform actions on it (data workflow orchestration), and a place to store and retrieve it (data warehouse). As data stacks have become more complex with more platforms and tools, more use cases, and of course a lot more data, both workflow orchestration and data warehouses have become more important. With best-of-breed orchestration paired with super fast and reliable data access, data organizations can build and iterate faster and deliver insights more reliably with greater confidence. While that sounds like a lot of marketing gobbledigook, it’s exactly what our users are telling us. In fact 95% of our users report significant productivity gains by adopting Prefect.
How do we do all this together? As the new standard in dataflow automation, Prefect enables data engineers to automate every data task and dataflow in their processes – most of which have some connection to the data warehouse. Built-in error handling means any issues are detected and handled, and our Cloud UI makes it much easier to diagnose issues. Prefect offers performance capabilities far above other orchestrators, and Snowflake has no speed limit, so it can more than keep up.
Some might ask why Snowflake customers need Prefect if Snowflake already has workflow orchestration. The answer is, well, some might not. If you have relatively straightforward pipelines that operate entirely within the Snowflake sphere, Snowflake orchestration might be right for many use cases. But the vast majority of pipelines today involve dozens of steps across multiple platforms, and Prefect is best positioned to handle those.
But perhaps even more importantly, Prefect offers visibility into your data workflows, even those within Snowflake. This means that once you trigger a dataflow with Prefect, you know exactly the state of it, get warnings if there are errors, and easily restart in code or in the UI. It’s this comprehensive visibility that customers such as Alex Welch, VP of Enterprise Data and Analytics at TastyTrade, really set Prefect apart. “Prefect is the glue of our data stack. It gives us visibility into our critical pipelines that we never had before.”
In summary, using Prefect and Snowflake together, data teams can get more out of each product, and more out of their data. 
Our Partnership
undefined
Partnership announcements can sometimes seem bland, but deepening our relationship with Snowflake is really a big deal. We've been recognized by Snowflake as a "Select" partner, meaning we have passed technical, customer and use case validations. This means our users can be assured that Prefect has been validated to work with Snowflake, and continually updated to reach the latest versions. We’ll also have more access to each others’ technical resources to support customers. And for those customers new to adopting our products together, the experience will be streamlined – everything from technical validation through deployment.
What’s Next
First up, we’ll be sponsoring Snowflake Summit, Snowflake’s annual conference on all things data. This will be the first of many events together to educate and support our customers.
We’re also already working on new and improved integrations for Prefect 2.0 to make our connections easier to set up and maintain by wrapping all available functions into a Collection.
We’re also looking forward to learning more from our joint users. The funny thing about an open source product is that we actually don’t know who all our users are and how they’re using Prefect. As we work together with Snowflake, we’ll be able to see that many more use cases, impacts and needs directly so we can make the platforms even more powerful together.
Ready to Dive In?
If you’re a Prefect user and haven’t yet tried Snowflake, give it a shot. You can sign up for a free trial here, and get up and running quickly with our library of pre-built tasks. You can also, of course, roll your own if you prefer. I’d also recommending reading Anna Geller’s stellar blog series on using Prefect with Snowflake and other modern data stack tools, such as dbt.
If your organization is standardized on Snowflake but aren’t using Prefect, give it a try. Because it’s just Python, Prefect is super easy for any data engineer to pick up, and you can develop and test straight from your laptop. Our docs will walk you through it, or you can join our Slack with thousands of Prefect enthusiasts ready to assist and share ideas.
Happy Engineering!