For Jeremiah Lowin, founder and CEO of DC-based data orchestration company, Prefect, this journey was no exception. His story isn't just about creating a successful startup; it's about a lifelong quest to understand and improve the systems around him.
Growing up near New York City, Jeremiah was known for his relentless questioning and deep interest in the mechanics of things rather than their outcomes. Jeremiah's inquisitive nature found a significant outlet during his sophomore year at Harvard, when a simple contradiction sparked a defining moment. He encountered a claim in his introductory statistics class – the stock market is normally distributed – that was at odds with what he had just read in a book, The Misbehavior of Markets. This discrepancy wasn't just a matter of academic interest – it was a puzzle that needed solving. Jeremiah's drive to reconcile these conflicting ideas led him to conduct his own study, a process that not only provided answers but also shaped his approach to problem-solving.
After a few weeks of learning to use statistical tools, Jeremiah proved to himself that the stock market was, in fact, not normally distributed. He confidently took that conclusion to his statistics professor who gave him a response that has stuck with him forever:
"It’s true, stock markets aren’t normally distributed. But they’re normally distributed enough for Stats 101."
This statement, that a model can be “good enough,” became a central point for Jeremiah. He came to realize that the goal of acquiring a quantitative toolkit wasn’t to build something perfect, but to build something useful. (Unsurprisingly, today he often quotes George Box’s famous claim that “all models are wrong – but some are useful.”) Jeremiah became obsessed with ways of building “useful” models that could take messy, real world data and produce actionable insight. He spent the next few years pursuing this idea, completing a master’s degree in statistics while still an undergraduate and writing his thesis on a new class of models for measuring the dependencies between stock prices. The thesis was certainly “good enough,” winning top prizes from both his department and the university.
After graduating, Jeremiah went to work on Wall Street, joining a large hedge fund called King Street Capital in 2007 as the first member of the firm’s new risk team. There, he found himself almost instantly in a hot seat as the firm navigated the 2008 financial crisis. Jeremiah eventually oversaw market risk and was responsible for building tools for a huge number of strategies and stakeholders. The need to rapidly iterate over so many different outcomes reinforced his belief that what makes a tool “good enough” is that it delivers value to its user; effort beyond that was often wasted. Looking back, Jeremiah is deeply grateful for the enormous trust and agency that the firm gave him so early in his career. Later, he would try to recapture the best aspects of that environment at Prefect.
In 2011, Jeremiah became captivated by the then-nascent field of machine learning. For the next two years, he made his living as an ML consultant, building software and tools for a variety of clients in many industries. Ultimately, he would go to work for one of them, becoming Director of Risk for a large investment firm. His mandate was to build technology and tools that made the inner workings of the team’s decision-making observable. He found himself constantly managing an extraordinary number of stakeholders who, once they learned what Jeremiah could build, all wanted his help solving various problems.
Not because he actually founded a company – that was still years away – but because his success depended on whether he could build a set of reusable solutions that would solve each stakeholder’s problem, even before he learned what their problem was. The philosophy of “good enough” became incredibly important as he pursued practical solutions that weren’t overfit to any one use-case. The ability to create adaptable and user-friendly tools became a hallmark of his approach.
The one thing Jeremiah didn’t have was enough time. His increased responsibilities and the need to support all of his tools left less and less space for the interesting analytical work that he loved doing. He became interested in automation software as a solution to this problem and ended up becoming one of the first developers to join the Apache Airflow team. Jeremiah became one of the top contributors to Airflow and began using it extensively for his regularly scheduled daily tasks, like refreshing databases and sending reports. He gained an appreciation for automation and orchestration that has never subsided.
But when Jeremiah tried to automate his analytical work, he ran into trouble. His data science tools, such as Dask (of which he was an early user), could instantly spin up thousands of concurrent, dependent tasks. His automation tool, Airflow, could only kick off a new task every few seconds. It was orders of magnitude too slow and too inflexible to capture his data science activity. Jeremiah tried to convince the Airflow team to modernize the product for a data-native world, but failed.
So, as the origin story for many founders goes, he built a tool to solve his own problem. It was called Tin Man because its job was to perform repeated, mechanical tasks. (Jeremiah recalls being on a Wizard of Oz kick; his machine learning library was called Scarecrow.)
Jeremiah had no intent on open-sourcing Tin Man, much less selling it. However, two chance conversations would change that. The head of engineering at one of his company’s largest counterparties, as well as the head of data science at another investment firm, both approached him in the fall of 2017 with a common complaint: their workflow automation software couldn’t scale to meet the needs of their data science teams. Jeremiah gave them each a copy of Tin Man, and the response was instant: “If this was a product, I’d buy it.”
Recognizing the impact and potential of what he’d built, Jeremiah began to see a path forward that he hadn't considered before. This tool, born out of necessity and personal frustration, had struck a chord with others facing similar challenges. It was clear that there was a gap in the market for a tool that could efficiently bridge the worlds of data science and workflow automation. Jeremiah started making phone calls to ask his network for advice, and over and over the calls ended with “You know, I think my teams might be having this problem, too.”
The premise was not that automation was a new problem; workflow orchestration has been around for as long as workflows have run on computers.
What was new was that the world was becoming increasingly data-rich and dependent. Data science tooling had kept up, but data engineering lagged behind. Most automation frameworks ignored data entirely, making them incompatible with modern workflows.
Prefect was founded to modernize data engineering best practices in a way that data science teams would recognize and use. But there was a twist, informed by Jeremiah’s background in risk management: Prefect was obsessed with failure. In Jeremiah’s words, “It’s easy to see that automation leads to productivity. But when an automated workflow fails, it can be catastrophic. The time it takes to discover, triage, and repair a broken workflow can more than erase all the gains from automating it in the first place.”
Prefect’s goal was to be maximally helpful when code had crashed – something so difficult that most automation systems don’t even attempt to elevate failure notifications beyond a red light on the dashboard. Jeremiah believed that an orchestrator didn’t have to be able to prevent failure – it just had to be good enough to alert users so they could start fixing it immediately.
Before starting operations, Jeremiah was introduced to Chris White by a mutual friend. Chris had been working on a similar set of problems at Capital One, blending data science and engineering, and he became Prefect’s first employee. Today, Chris is the company’s CTO and responsible for many of its technical innovations.
In March, they released Prefect as an open-source project, a major milestone for the young company. This strategic decision wasn't just about sharing code; it was about building a community and fostering collaboration. The response was beyond encouraging. Jeremiah recalls that they thought 100 GitHub stars would represent a great launch; the project gained over 500 in its first few days. Acknowledging that GitHub Stars can be a “vanity metric,” Jeremiah said that nonetheless the team felt validated. That feeling—that people actually want, like, and will use their product—fueled them.
The company was unusually hands-on with those early users. Jeremiah and Chris treated the first ~20 heavy users as if they were customers rather than open-source community members. They flew to their offices, gave them tutorials, and took them out to get feedback. The team also did its best to maintain a 15-minute response time in the company’s Slack. Jeremiah notes that all young startups have a credibility problem – how can you prove you’ll be around in the future? – and Prefect’s solution was to “control what they could control” (one of Jeremiah’s often-repeated mantras) by building a reputation for high-quality software and service. This is something that they still believe represents an advantage today.
In the summer of 2019, Prefect was approached by a team at a Fortune 100 company. They loved the open-source product and wanted to buy a license. Prefect’s commercial product was in (very) early development at the time, but Jeremiah and the team flew to meet the interested team at their offices, and gave them a demo. Jeremiah remembers thinking that no one would buy the product in that state; it barely had a login screen or full UI. But within a few months, the enterprise became Prefect’s first customer and proved that even in that rough state, the product was already good enough.
By this time, Prefect had more than 10 employees and started focusing intently on people and culture. Jeremiah prized the environments he worked in early in his career, where titles and hierarchy vanished in favor of impact and outcome. Sarah Moses joined the company to lead operations (today she is the COO). Jeremiah notes that his “unlimited optimism” is a great match for Sarah’s “ruthless accountability” and allowed them to lay the foundation for the high-agency, high-performance culture that grounds the company today.
A few weeks later, the company was close to completing its Series A when Covid hit and the deal fell apart. Surviving became the company’s main focus. The company adjusted its strategy and decided to go for mindshare over market share, As a result, Prefect open-sourced most of its commercial product, including its proprietary UI. To the team’s surprise, this led to a surge of interest as teams tried the product, and growth took off. The company finally raised its Series A from Positive Sum, a new VC firm started by Patrick O’Shaughnessy, who had backed Prefect from day one. Valor Equity Partners, Atreides, and a huge number of high-profile angels joined the round.
This was the year the company would really “grow up.” They hired an SVP of People and once again revisited their structure, systemization, and operations in order to scale past 25 people. By 2022, the company’s headcount had swelled and Prefect 2 had been released. The product was designed around its ability to handle any code, not just code that had been specifically written to run on Prefect. This dynamism let the Prefect team quickly adapt the product to any new use case with minimal burden on their users.
In 2023, it built a true enterprise version of its product in order to meet the needs of its largest customers. Jeremiah notes that this year was all about efficiency and profitable growth. “2023 was a hard year to be happy - everything had to be fought for and we had to really lean into the processes we had established with grit, hard work, and trust.”
Looking ahead, Prefect isn't just growing; it's constantly reinventing itself. For Jeremiah and the team, it's never been about just riding the wave of their current success. It's about what's next, about staying agile and innovative in a field that never stands still. Jeremiah says, "Our biggest win isn't what we've done up to now, but how we find ways to compound it forward. We need to be ready to outdo ourselves."
This philosophy is what's kept Prefect at the front of the pack. It's not about basking in past glories; it’s about being ready for tomorrow's challenges. "Innovation isn't just about coming up with cool ideas," Jeremiah points out. "It’s about evolving those ideas so they stay relevant and, most importantly, deliver actual value to users."
For Jeremiah, Prefect is more than a company. It's a commitment to continuous improvement, to pushing boundaries, and to a culture that values adaptability and creativity. Looking forward, he sees Prefect not just as a tool for today but as a way for companies to gain confidence in the operations in the future. Prefect’s story is about the spirit of curiosity, the drive to always do better, and the counterintuitive belief in “good enough” as a strategy for real-world excellence.
Jeremiah's Lessons Learned
My Takeaways
