How Japan's Leading BNPL Company, Paidy, Transformed Their Data Operations with Prefect

Paidy is Japan's leading "Buy Now, Pay Later" (BNPL) service that allows consumers to make online purchases without needing a credit card or pre-registration. Instead, users can pay with just their smartphones. Purchases are consolidated and billed monthly, with options to pay via convenience store, bank transfer, or account transfer, with the option to pay in interest-free* installments.

When Cron Jobs Hit a Wall

John Ghanotakis, Senior Manager of Operations, and Jon Lamaster, Head of Platform & Infrastructure Engineering, explained that the rapid growth of the service prompted them to seek an alternative system to manage jobs more efficiently. They were looking for a system creating configurable jobs that can run ad-hoc or on a schedule, and isolating running environments to improve resiliency and scalability.

Finding the Right Tool

When they started looking at orchestration tools, Paidy had three main criteria:

• Avoid deploying new kinds of AWS services that weren't already being used in the organization, which would speed up the process of moving away from their existing infrastructure and reduce the need for approvals
• Address their core challenges as mentioned above: creating configurable jobs that can run ad-hoc or on a schedule, and isolating running environments to improve resiliency and scalability
• Various teams can easily use the tool to build their own jobs

They considered other competitors, smaller open source orchestration systems, and putting together a solution with existing AWS services.

"We looked at all the options and came to the conclusion that Prefect would be a useful tool for Paidy. We felt that putting together a solution would inevitably result in us having to do more maintenance. We looked at an industry standard, but when we looked at how people were deploying it, we realized we'd have to build more on top of it to make it work for us," - John and Jon

They were primarily focused on improving the robustness, recoverability, and monitoring of their existing jobs, and Prefect offered the right balance of capability and simplicity for their needs.

Prefect Across Paidy

"We use Prefect in our ETL pipelines that feed data from the real-time payment system into our data analytics platform and for scheduling and executing financial, regulatory, and transaction reports," John and Jon

It's become a key piece of infrastructure to make sure their data gets to stakeholders on time.

The tool is used to automate several business processes throughout the company, but it's mostly used by Data Engineering, Data Science, and Data Analyst teams for data transformation and reporting.

"Prefect is the backbone for scheduling reports for key merchants." - John

One feature that's been particularly helpful is the ability to have multiple work pools - EKS versus ECS - which lets them decide which infrastructure should run certain flows.

"The ability to decide which infrastructure will run a given flow with Prefect has an advantage considering the different infrastructure provided by Paidy," - Jon

They handle both real-time ad-hoc requests and scheduled batch jobs.

Prefect's Impact

The biggest improvement since adopting Prefect has been aligning tools within the company to satisfy multiple use cases.

"Without Prefect, we would have lacked synergies among the teams and potentially would have had to have multiple tools for multiple use cases," John and Jon

Prefect is a key driver for their merchants to understand transactions within the Paidy platform. They've been able to utilize existing knowledge on Prefect to more quickly onboard teams for critical reporting.

Advice for Other Teams

John and Jon's advice for other teams in similar situations is to get buy-in from the whole organization to ensure support and synergies before implementing a solution. They emphasize, "We needed something that could address our needs without bells and whistles but be 100% reliable. That is what Prefect is for us."

John and Jon describe Prefect simply:

"Prefect is an innovative job scheduler with the ability to select the appropriate infrastructure. For us, it is a tool to run reports in a timely manner. It is reliable and supports our requirements."

* Interest-free installments apply only to payments made via bank transfer or auto-debit.

To learn more about Prefect:

• visit our website
• visit us on GitHub to open issues and pull requests*
• follow us on LinkedIn for regular updates
• join our active Slack community