Data pipelines are growing more interconnected every day. But with this scale comes complexity: a single workflow might query multiple databases, call several APIs, and write to various storage systems - all while other workflows compete for the same resources.
This scale has revealed a critical truth: when issues arise, understanding the impact isn't just about checking if a workflow succeeded. You need to know how it affected - and was affected by - every system it touched.
Today, we’re rolling out a suite of features that fundamentally change how teams monitor, debug, and scale their data systems. These additions make the invisible connections between workflows and external systems explicit and controllable through observability.
When customers report consistently late results, or teams notice rising failure rates, manually combing through thousands of logs isn't just inefficient—it poses a serious risk to the business. For teams who orchestrate thousands of runs daily, knowing what broke isn't enough - you need to understand patterns across deployments, spot potential bottlenecks before they impact customers, and quickly identify which workflows need attention. To address this problem, we built Enhanced Operational Dashboards that help teams to quickly identify data patterns and take action.
Our Enhanced Operational Dashboards enable teams to move from reactive troubleshooting to proactive system management by grouping workflow analytics by deployments, work pools, or custom tags. These dashboards introduce powerful analytics and grouping capabilities that let teams:
Availability: Enhanced Operational Dashboards are in beta and are currently available to Prefect Cloud users. Not a Prefect Cloud user? See our Enhanced Operational Dashboards in action by booking a demo.
Modern data systems don't exist in isolation. When something breaks in production, the real challenge isn't just finding what failed - it's understanding the ripple effects through your entire system. Did bad data propagate to your warehouse? Did it impact model training? Which customers are affected?
In light of this, we've built Resources and Run Tracing, debugging tools designed to provide both the high-level map of your system and the granular details you need to resolve issues confidently.
Rather than viewing workflows as isolated units of compute, Resources reveals the interconnected nature of your data platform - from S3 buckets and databases to API endpoints and model artifacts. Teams have a unified way to track, visualize, and control these interactions, allowing them to:
Availability: Resources is currently in beta and is available to Prefect Cloud users who opt into beta testing. If you are interested in joining the beta, reach out to your CSM. Not a Prefect Cloud user? See Resources in action by scheduling a demo.
While Resources gives you the big picture, Run Tracing provides the execution-level details. It transforms how teams debug by offering clear visibility into dependencies and event-based execution paths, allowing teams to:
Availability: Run Tracing is currently available to Prefect Cloud users. Not a Prefect Cloud user? View our Enhanced Operational Dashboards in action by booking a demo.
Building reliable data systems isn't just about handling today's workload—it's about being ready for tomorrow's growth. As organizations scale, workflows increasingly compete for shared resources which risks the possibility of overloading systems like databases or APIs. This forces teams to choose between potential failure, overprovisioning infrastructure, or building complex systems as work-arounds to limitations. To ease these growing pains, we built Deployment Concurrency to give teams precise control over workflow execution.
With Deployment Concurrency, instead of building complex throttling systems or overprovisioning infrastructure, teams can now:
Availability: This feature is available to all Prefect OSS and Cloud users. Learn more by viewing the docs.
Modern data platforms demand more than just reliable execution. As organizations scale, they need to monitor thousands of workflow runs, debug complex interdependencies, and manage growing infrastructure demands - all while maintaining system reliability and efficiency.
That's exactly what this new suite of features delivers. Enhanced Operational Dashboards transform overwhelming data into actionable intelligence. When issues arise, Resources and Run Tracing provide both system-wide visibility and granular debugging capabilities. And with Deployment Concurrency, teams can scale confidently while protecting their critical resources.
These features represent more than just technical improvements - they embody our vision for the future of workflow orchestration. By providing tools that enable teams to understand and manage their workflows at a system-wide level, we're helping organizations build more resilient, efficient, and trustworthy data platforms.
Our commitment to simplifying complex infrastructure doesn’t stop there. We are always building and iterating to continually add value to our Prefect community. We're actively working on:
Want a personal walkthrough of Prefect’s observability suite? Contact our team for a customized demo of these new features.
