TL;DR: We’re Prefect, we orchestrate a thousand years of Python runtime every month. We built a cool feature called assets. Assets let you track decision and data lineage for your agents or workflows by showing you what they produce and how they relate. Check out the documentation or scroll to the bottom for some helpful resources our team made for this feature. We hope you’ll try it out and share what you think.
Prefect’s asset-oriented API lets us model lineage the same way we write functional Python. With @materialize we immediately get a clear, end-to-end view of our data assets.
-David DeStefano, Sr. ML Engineer at Adonis
Every day at Prefect starts the same: a 40-person huddle where we all repeat, in unison, “Let people write Python the way they already do and stay out of the way.” Or at least we should. After all, this mantra’s gotten us pretty far. Customers shipped stuff and mostly forgot we existed, which is success in our book.
But there was always a tiny rock in our shoe: “Okay, but what does this code actually produce?” Data workflows create outputs. Trained models, analytics dashboards, TABLES — “assets”. When a dashboard throws garbage numbers at quarter-end, someone has to follow the breadcrumbs back through tasks, tables, and stray CSVs. When the warehouse bill arrives, someone else asks, “Is anyone still using this table, or can we finally nuke it?” That’s asset lineage in the real world.
We’re, by far, not the first ones to tackle this problem. If anything, we’re late to the party. Some orchestrators have made these "data assets" the center of everything - you contort your entire workflow around asset definitions.
Months ago, we took a swing and missed. Our original design required so much boilerplate that the juice simply wasn't worth the squeeze. What’s that Russian proverb? The first pancake is always lumpy. Well, our first attempt was lumpy.
So we reconvened in our 40-person huddle and got to work. Sure enough, somewhere between the 90th and 100th recitation of our morning mantra we realized we could build asset lineage the Prefect way, no workflow contortions required. When you replace @task with @materialize, Prefect shows you how your data outputs connect, without the cruft.
Assets let you track what your workflows produce. You've got a task that writes a file to S3? Tell Prefect about it:
1# Before
2@task
3def write_data():
4# writes s3://bucket/important-data.csv
5 pass1# After
2@materialize("s3://bucket/important-data.csv")
3def write_data():
4# writes s3://bucket/important-data.csv
5 pass
6
When this runs, Prefect records that s3://bucket/important-data.csv was created. Do this across your workflows, and you get a global graph of what your system actually produces and how it all connects.
Assets aren't just for engineers. This is the strongest invitation we've ever had for people who don't write code to care about what lives in Prefect. Data analysts, product managers, business stakeholders can open the assets view and immediately understand what's happening. No need to decipher task graphs or flow runs.
They see the actual data outputs that matter to the business, with clear connections showing how everything relates. When someone asks "where does our customer segmentation data come from?" you can pull up the asset graph and trace the entire lineage in seconds. See when assets were last updated, identify potential issues, and understand the blast radius of any changes.
Our goal was simple: let users model their actual data as it exists in their world. Tables in databases, files in S3, models in registries. Assets use URI-style keys that match where your data actually lives, and the graph reflects what actually ran..
Think of it like SimCity (the city-building game). You could switch between different views of your city. Traffic view, power view, water view. Same city, different lens. That's what assets are for you. A different view of the activity you’re already tracking.
Most tools force you to define your entire asset graph upfront in their DSL (domain-specific language). Others make you explicitly declare dependencies even when your code already shows the relationships.
Dagster built their DSL around assets. You write asset-producing functions and define dependencies between assets. Their approach centers on thinking about workflows in terms of the data assets they create. This is true asset orchestration and requires users to adopt their asset-based mental model.
Airflow 3.0 rebranded their 'datasets' feature from Airflow 2.0 to 'assets', and added a new @asset decorator as syntactic sugar. Like Prefect, their @asset decorated functions act as factory for materialization events. Unlike Prefect, they don't support dynamically materializing assets or creating assets imperatively.
Prefect builds the asset graph from what actually runs. If your workflow dynamically creates different outputs based on data conditions, your asset graph reflects that reality. The graph changes because your data changed, all from one simple replacement. Swap @task with @materialize for the functions that produce data, and Prefect maps the relationships automatically.
Assets just shipped and we're already thinking about ways to make it better.
Here's what we're exploring next:
If you have suggestions or requests for what you'd like to see, check out our GitHub discussions. Our Community Slack is also a great place to connect with others and get help with your workflows.
Assets are available today for all Prefect Cloud users.
