The All-New Prefect Server and UI

The All-New Prefect Server and UI

Two key Prefect products are becoming open-source projects in our biggest release ever.

Jeremiah Lowin | CEO

August 6, 2020

Today is the largest release we’ve ever done across the Prefect platform, including Prefect Core 0.13, Prefect Cloud, and two completely-new open-source projects: Prefect Server and the Prefect UI.
Early versions of Server and the UI were previously available inside Prefect Core. Graduating them allows us to finally deliver tons of new, highly-requested features; four months of bugfixes and performance improvements; new dedicated open-source development hubs; enhanced deployment options; better maintenance and support — and much, much more.
True to our commitment to making software that’s easy to use, our open-source users still only need to remember one command to launch the entire platform: prefect server start.
In this post, I will describe the highlights of this coordinated release and how it exemplifies our true open-core business model. For a more technical look, please see our CTO Chris White’s accompanying post, which will be posted shortly.
The Initial Release of Prefect Server
A few months ago, we released an initial open-source version of our workflow orchestration platform, called Prefect Server. As we wrote in March:
Open-sourcing Cloud is a massive undertaking that would normally require months to plan, coordinate, and execute. However, we will be ready to release it next week. By operating at this pace (typical at Prefect!), our software can become available to any business that needs it, at exactly the moment businesses are demanding efficiency the most. Allying ourselves with our partners at the earliest possible moment — for free — is the ultimate expression of our commercial directive: deliver value, don’t extract it.
In order to release Server on that schedule, we elected to fork our proprietary Cloud codebase. What this gained us in velocity, it sacrificed in maintainability: the amount of code our small team was responsible for instantly doubled. To cope, we rallied around a classic startup cliché: “If so many people use it that we’re strained, that would be a good problem to have!”
Narrator: It was not a good problem to have.
The public response to Server’s release turned out to be extraordinary: it instantly became the most popular thing we’ve ever built. Our open-source community and customers both doubled last quarter and show no signs of slowing down.
We have always made a major commitment to supporting our users in near-real time, as any of our 1200 Slack members can attest. However, the extreme popularity of Prefect Server’s forked codebase has made it challenging for our lean team to deliver that standard. Server and its UI have not received the attention they deserve: community PRs lack visibility and we haven’t found enough bandwidth to port the enhancements we make in Prefect Cloud.
Therefore, we’ve spent the last quarter doing what we said we didn’t have time for in March: planning, coordinating, and executing a “proper” open-source product launch. Our entire team was involved in this effort — from gathering feedback, to studying user requirements, to writing code, to reviewing docs, polishing our presentation, and more. Today’s release of a brand-new Prefect Server and Prefect UI is the culmination of that effort and a rededication of our commitment to producing a new standard in workflow automation software.
And it’s just the beginning.
Two New Open-Source Projects
The clear highlight of today’s release are the two new repos for Prefect Server and the Prefect UI, which were previously only available deep inside the Prefect Core source code.
For most users, these repos may only exist as a curiosity — there is no need to install or even look at them to run our software. For our growing development community, however, they will provide an excellent experience that we were not able to offer before in a single-repo format.
Prefect Server
The new version of Prefect Server is no longer a fork; it’s a shared codebase with Prefect Cloud. This means that improvements our team makes to our flagship product will automatically and immediately benefit our open-source community. Conversely, our community can help shape the roadmap for Prefect Cloud through a dedicated development hub. Cloud will continue to deliver additional features, performance, and infrastructure to our paying customers, but its core will remain fully available. Earlier this week, we addressed a Cloud customer comment and improved their user experience by making a change to the open-source Server repo. It was an exciting moment (we’re nerds, sue us) and perfectly captured our objective of building an aligned open-core business.
The new Server gains a few popular Cloud features that the old one did not include:
  • Flow groups, which allow users to change flow settings like schedules, environments, and more — right from the UI and without modifying any Prefect Core code
  • Maintenance services like the Zombie Killer (which searches for distressed tasks and retries or fails them) and Lazarus (which searches for distressed flows and reschedules them)
…and countless other improvements, many of them invisible, that have been added to Cloud over the past four months.
Prefect UI
The Prefect UI also has its own repo, but unlike Prefect Cloud its source code is fully available. The UI in this repo IS the Prefect Cloud UI. This means that starting today, users of Prefect Server can immediately benefit from months of enhancements that have been added to Prefect Cloud.
One of the new UI’s most innovative features is that it can instantly be pointed at any Prefect API, whether Cloud or Server, by changing a setting in the UI itself. This means that Prefect Cloud users can self-host the UI — they can even fork and completely customize it — while still running against the managed Cloud service. In addition, Prefect Server users who do not want to host a UI can log in to Prefect Cloud and point the Cloud UI at their local Server instance! (Note that no Server data is sent to Cloud when you do this.)
We work with many companies that build internal dashboards on top of the Prefect Cloud API; now they can bring their custom analytics directly into the UI. We are not aware of any product that has taken this approach before, which puts it hand-in-hand with our Hybrid Execution Model as an example of how Prefect has even innovated the open-core business model.
New Product Lineup
With the release of Prefect Server and the Prefect UI, we are dramatically expanding our product lineup to meet the needs of our users. Prefect Cloud remains our flagship offering, leveraging our Hybrid Model to offer on-prem guarantees through a SaaS model. We have invested heavily in its infrastructure, performance, and security. Prefect Server and the Prefect UI allow users to run parts of Prefect Cloud locally. This is appropriate for smaller, single-node deployments or complete customization. And Prefect Core is still the very best tool for building, testing, and executing data workflows.
With this lineup, we are able to significantly advance our mission to eliminate negative engineering. We are beginning to publish case studies that show the incredible gains companies can make by adopting Prefect software — such as the 99% decrease in model development time experienced by SymphonyRM.
In addition, having aligned these codebases, our team is now able to offer true commercial support (in addition to our wonderful community channels). We are grateful to all of the partners across our open-source and Cloud communities who helped us reach this milestone.
We’re Growing!
Over the next couple quarters, we’ll be growing a lot. Open positions are always posted on our website — there are three as I write this — and we will do our best to circulate the job descriptions as widely as possible. Please help us reach groups that are under-represented in the tech world!
Stay Tuned
It’s hard to believe after all the incredible work that went into making today’s release possible, but this isn’t even the biggest news we have to share. The best is yet to come…
Please continue reaching out to us with your questions and feedback — we appreciate the opportunity to work with all of you!
Happy Engineering!
— The Prefect Team