SpareBank 1 builds a shared data platform for its banks with Prefect.

SpareBank 1 is an alliance of Norwegian banks that serve individuals and businesses with mortgages, car loans and business lending. The banks compete for customers while sharing technology, infrastructure and branding. Historically the banks had each developed and maintained their data platforms separately. The alliance decided to bring that work together and build a modern data platform as a shared service.

Arv Breistøl works on the data platform team in the alliance’s shared-services company. His team provides the tools and infrastructure used by central ingestion teams and engineers in the member banks. Those banks receive centrally supplied datasets, but they also write pipelines for their own data. The platform has to support engineers doing both kinds of work.

Why Prefect fit the shared platform

The alliance chose Snowflake for storage, AWS for infrastructure and, at the time, dbt for modeling. For ingestion, it wanted to write pipelines in Python. Orchestration needed to fit that approach and become part of the platform service the banks would receive.

The evaluation predated Arv’s arrival. The team documented its use cases and scored tools including Prefect, Airflow and Dagster. Scheduling, dependencies and logging mattered to the engineers running pipelines. For the platform team, infrastructure as code was particularly important because it needed to automate setup and provision resources for other teams.

“When we did the scoring, Prefect came out on top.”

Many engineers in the alliance already knew Airflow. Arv’s impression was that the evaluators preferred the development experience in Prefect, particularly how jobs were declared and how orchestration followed the Python code. “[Prefect] was super impressive technically,” he recalls.

The pipelines required little Prefect-specific code, according to the architecture decision records Arv reviewed. Most of the implementation remained in Python pipelines the engineers would have written anyway, with Prefect supplying the orchestration instructions.

Arv also points to the Norwegian and EU regulations the banks operate under. Before adding Prefect to the stack, the alliance assessed IT risk, data privacy, outsourcing and the vendor. That included how information would be stored and used, making execution on their own AWS workers (in the EU-based region) an important part of the fit. Prefect could coordinate the pipelines while execution stayed within the banks’ infrastructure.

The overnight load other teams depend on

Around midnight, a core banking provider makes customer and transaction data available. The central ingestion team starts processing it as soon as it arrives because downstream teams have their own jobs waiting on that data. Scheduled Prefect runs start the work while the engineers are asleep.

Magnus Brathovde works on the ingestion team in the same shared-services company as Arv. By his estimate, the team loads data from roughly 4,000 source tables every day. The overnight core banking load is one part of that responsibility, alongside other external sources and legacy systems being modernized.

Modernization does not remove the need for data from those older systems. The team lifts and shifts it into the new platform so the banks can continue using it. Most ingestion is batch based, across sources that range from large and complex to smaller and specialized. Some remain stable while others change, creating the need to investigate failures and get the affected pipelines running again.

Magnus has set up Slack logging for errors. When engineers start work in the morning, they check for messages, each linking to the relevant Prefect run. For Magnus, a morning with no error messages means the overnight work has run successfully.

“We can easily see what has failed, then go straight to the stack trace in Prefect.”

If an unexpected failure requires a code change, the engineers reproduce it in their nonproduction environment and work on a fix. They deploy the change, then manually retry the run in Prefect.

That work supports a platform used mainly for analytics and reporting, but also for modelling and inference. As Arv explains, missing data could leave an advisor with outdated customer information or a portfolio risk assessment without a complete set of data. The ingestion team makes the source data available so other teams can combine it, model it and use it in reports and applications.

A bank’s own pipelines, on infrastructure already provided

For Fredrik Rosenberg, who works at one of the alliance banks, the shared service provides both those central datasets and the tools to build on them. His bank receives a Prefect account with work pools and AWS execution infrastructure already set up by the platform team.

Fredrik’s engineers use that account to run ingestion pipelines they write themselves. They use the Python package dlt to load data into the bank’s own Snowflake account, with Prefect orchestrating the work.

“We are using [Prefect] to orchestrate the data pipelines into our own Snowflake account.”

The alliance subsequently adopted dbt Cloud. For selected ELT pipelines, Fredrik’s team uses Prefect to ingest data into Snowflake and the prefect-dbt package to trigger the dbt Cloud jobs that follow. Prefect coordinates the loading and transformation steps in those pipelines. The team also uses Prefect to push data into the bank’s CRM system.

Across the wider data platform, teams combine and model data for financial reporting, anti-money-laundering work and fraud detection, among the uses Fredrik describes. For regulatory reporting, Arv explains, the banks report their portfolio holdings and risk to governing institutions in Norway. The platform’s data also feeds lending decisions and dashboards that help advisors understand a customer before a mortgage, pension or insurance conversation.

About 100 people across the alliance use Prefect more or less daily. Within that wider service, Fredrik’s team has 93 deployments in its Prefect account. Its work extends from bringing data into Snowflake to triggering selected transformations and sending data to CRM. With the work pools and AWS infrastructure already provisioned, the bank’s engineers can build and run those pipelines on the platform the alliance provides.