Why Dyvenia Adopted Prefect

The Stakes

dyvenia is a data consultancy that helps companies across industries to modernize and democratize their usage of data. dyvenia works with a variety of stakeholders, including marketing, supply chain and finance professionals in large and mid-size companies to solve their data engineering, data science and analytics challenges.

Technical Problem

As a data consultancy, dyvenia is faced with a variety of different data stacks across many different customer situations. They are working with constantly changing data sources, from legacy vendors (Oracle, SAP) to modern tools (Snowflake, dbt, Databricks).

Airdropping into a client situation where millions of dollars are on the line can be tense, and dyvenia must move quickly to assess and start making an impact immediately. As a true data partner, dyvenia knows that any solution that they create must also be repeatable and long lasting.

Previously, dyvenia was using a custom orchestration tool. They not only struggled with inefficient development and lack of visibility, but they also found it difficult to manage. They had previously considered Airflow to configure & run their clients’ data pipelines.

Airflow is complicated - for our partners we want the technology to be easy to maintain. Airflow is unnecessarily complex and hard to learn, which is at odds with our principle of "learn and adapt fast". - Alessio Civitillo, CEO of dyvenia

How Prefect Helped

Adopting Prefect has allowed dyvenia to speed up their POCs and time to production significantly: what previously took weeks now only takes days with Prefect. dyvenia is able to simply sprinkle Prefect on top of their existing Python code, which allows them to productionalize their processes quickly.

Deployment is also another large reason for efficiency gains. Compared to their custom orchestration tool, Prefect’s Hybrid Model is a key reason for this improvement.

The ease of using just a central Prefect server compared with deploying a new server each time for orchestration is much easier - Alessio Civitillo, CEO of dyvenia

Prefect also has given dyvenia visibility and reliability where they once were handcuffed.

It takes significant time and effort to orchestrate and observe the scale of complex pipelines we manage Prefect makes sure all of our ETL piplines are reliable, trustworthy. - Alessio Civitillo, CEO of dyvenia

Identifying pipeline failures once required engineers to investigate issues within every individual tool with a scheduling capability. Prefect’s coordination plane is a trusted source that integrates with any tool a client has, giving observability and orchestration capabilities across their entire stack.

Adopting Prefect has allow dyvenia to identify failures and react quickly, increasing their client satisfaction and resulting in stronger relationships and improved trust.

Business Impact

dyvenia’s clients measure their success in terms of ROI - achieving the desired outcomes, while minimizing cost. When considering an orchestration tool, dyvenia considered Time to POC and Time to Production as their two key metrics for achieving these goals.

Time to POC has been reduced by over 90% from a week to just 2 hours. We reduce time to production from up to 6 months down to just 3 months. - Alessio Civitillo, CEO of dyvenia

Adopting Prefect hasn’t only improved dyvenia’s client engagements: it has enabled dyvenia to build an entirely new product.

New Business Model

With Prefect, dyvenia has unlocked a totally new business model for organizations previously using SAP’s data warehousing solution.

Prefect allows us to reliably pull data out of SAP and start building a strong alternative to SAP BW - a very expensve, very common data warehousing solution for enterprises - Alessio Civitillo, CEO of dyvenia

To build this solution, dyvenia is utilizing Prefect, dbt and DataHub for a full lineage of transformations, totally searchable in a catalog, and completely automated and reliable. This removes lock-in for dyvenia’s end customers and gives them the ability to schedule and integrate data from anywhere.