Kepler cuts pipeline delivery from weeks to days.

Meet Kepler

A single hallucinated number can ruin a financial analysis. Kepler builds enterprise intelligence for high-stakes industries, starting with investment firms, to tackle that problem directly. On Kepler's platform, code extracts and computes every figure and cites it back to its source, while language models interpret the question and write the answer. Kepler's engineers deploy on each firm's own data and build an ontology on top of it.

That approach starts with bringing in and preparing the data. Kepler's pipelines ingest public SEC filings and private documents from sources such as SharePoint. Preprocessing and organizing that data makes query results more relevant and supports the research tools and workflows Kepler builds on top of it. Prefect orchestrates the ingestion and processing. Every figure Kepler returns traces back to a source document, so that document has to be ingested, current, and processed before anyone asks the question.

The work behind the pipelines

Kepler initially ran its data pipelines on a homegrown execution framework. As the team hired more developers, it needed an approach that could scale and that newcomers could pick up quickly. Engineers were fixing orchestration problems about twice a week, including stuck runs, schedules that silently stopped, and dead-lettered queues. Documentation struggled to keep pace with new features.

Sean, a founding engineer at Kepler who focused on the data pipelines, expected to spend his time teaching new hires how to deploy and maintain the custom system. He questioned whether that was a good use of engineering time when the team could be focused on Kepler's core technology, and why Kepler was spending that time diagnosing problems in its own orchestration system at all. Eddie, who leads strategy and operations, wanted to free up engineers to work on the product they were building for customers rather than on internal tooling.

Kepler's previous stack paired the custom framework with Temporal for run history and recovery when compute failed mid-process. The framework around it still required engineers to troubleshoot unexpected problems during container rollouts and to navigate changes that were difficult to roll back. Kepler wanted to stop maintaining that framework and adopt an orchestration tool with public documentation that new developers could use to get up to speed.

Why Kepler chose Prefect

Kepler evaluated how its existing pipelines would translate to a new orchestration platform, comparing APIs, hosting options, and core functionality across the options on the table.

Public documentation was an important criterion. Kepler was shipping features faster than it could document them, and Sean wanted new developers to have an existing resource to learn from.

“When we looked at what we had today and how it would transfer over to these other services and what they offer in terms of APIs and hosting and core functionalities, Prefect just had it all.”

Kepler also weighed capabilities it expected to reach for later, like creating tasks dynamically in code or triggering runs with custom parameters, even where the team did not need them yet. Having that flexibility alongside the standard DAG-based representations of its workflows made Prefect the right fit.

“[Prefect has] really good API offerings. So if we need DAG-based representations or visualizations, we already have our own dashboards we're building against [Prefect's] API. That capability is a big leg up for us, in my opinion.”

Developing with Prefect

The migration took about four weeks. Kepler's first pipelines were live in 10 days, and the team retired the old orchestration stack on day 28.

Pipeline changes that used to take roughly one to two weeks to ship confidently now take one to two days. Engineers can also create isolated copies of the pipeline for development using Prefect's primitives.

“We've leveraged Prefect's primitives to make development a breeze. Every engineer can also spin up their own isolated copy of the whole pipeline with a simple command and confidently develop without concerns about cross-pollination with other environments.”

How Kepler uses Prefect

Kepler uses Prefect for continuous data ingestion and for batch jobs that run once or a few times a day. Those pipelines prepare structured and unstructured data for customers to query across.

Kepler currently runs 32 production deployments across six pipelines, with the deployment count steadily growing. It runs roughly 170,000 flow runs a week, and over 200,000 during peak weeks. More than 20 million documents have been processed through Prefect, including a full reprocessing of the roughly 18.8 million-filing SEC corpus.

To bring in new filings, Kepler repeatedly polls EDGAR's RSS feed. Its continuous operations run as microbatches, with tasks lasting a few hundred milliseconds or a few seconds and triggering themselves again while the compute remains running. Table metrics that need less frequent updates are aggregated once or a few times a day.

Kepler ingests PDFs, Word documents, PowerPoint presentations, spreadsheets, and HTML, and processing branches by format from there.

Connecting customer data

As Kepler worked with more customers, Eddie found that bringing their data sources together was becoming a larger part of the work. The customers Kepler engaged with cared about data infrastructure but had not fully addressed it. Kepler wanted to make those sources available together so customers could get answers that drew on all their data at once.

Several customers' file systems hold their own proprietary financial analysis. Kepler built connectors to bring those documents into its platform, and the synchronization and processing logic runs in a Prefect pipeline. The pipeline continually checks for new files, changes, and deletions, then downloads and processes the updates. Kepler structures and pre-processes that data so every query and workflow gets reliable, consistent results.

Eddie wanted more customer integrations and workflows to become reusable parts of Kepler's core product, and expected customization to increasingly focus on the data each customer had, where it lived, and the preparation it required. Sean described accommodating those differences as the last mile of the work.

Less orchestration maintenance

Before the migration, the team fixed orchestration problems about twice a week. Since moving to Prefect, Kepler reports that maintenance on the orchestration layer has been close to zero. The team's work now focuses on tuning its own throughput rather than fixing orchestration problems, which is the trade Kepler set out to make when it left the homegrown framework behind.