PaidyPaidy
SHARE
99%

Reduction in Debug Time

1

Click Rerun

700k

Stores Connected

Jun. 9, 2022

Paidy ditches their manual debugging practices and operationalizes their data practices with Prefect.

Industry

Financial Services

Company Size

50-199

Key Use Cases
  • Data Science

  • Monitoring and Alerting

Products Used
  • Prefect 1.0

  • Cloud 1.0

The Stakes

Paidy, a subsidiary of PayPal, is a Japan-based provider of buy now, pay later (BNPL) services. BNPL services like Paidy allow consumers to complete a transaction (typically online) without requiring a credit card. Accepted natively at 700,000 e-commerce sites including Amazon, Apple and Shopify, Paidy is the dominant force in Japanese deferred payment - the 3rd largest eCommerce market in the world.

Deferred payment plans can often feel like a gamble for both consumers and merchants. Paidy’s goal is to remove the risk for vendors and empower purchases for consumers - in other words, defeating the negative engineering of buying online without a credit card.

Technical Problem

Paidy is a data-centric organization, with a large team of over a dozen data scientists whose goals are to help both the customer and merchant. The data science team is responsible for:

  • Understanding the customer journey from acquisition to selection to checkout

  • Managing risk, ensuring that customers aren’t overburdened with debt

  • Delivering value-add product recommendations for both customer and merchant

Before Prefect, Paidy’s legacy data science environment left them struggling to move quickly and accomplish their goals. From ETL to ML models, code was typically bundled in a container, scheduled with cron and executed on AWS ECS on Fargate. Whenever anything failed, nobody had insight into what was causing the problem.

“We had no way to control job execution, or to run jobs with custom parameters or configuration.” - Jacobo Blanco, Data Science Manager

Debugging was a labor intensive process, requiring code to be manually downloaded, with an average time to debugging of 2 hours for each issue. Paidy was looking for a way to operationalize their dataflows and solve their recovery and remediation problem when their jobs failed.

“We had a look at Airflow and from our experience elsewhere realized we'd need to do a lot more work to have it fit our use-cases. ” - Jacobo Blanco, Data Science Manager

How Prefect Helped

According to Paidy, migrating to Prefect from their existing environment was a breeze. The quality, simplicity and elegance of Prefect’s API made all the difference for a team that was tired of managing failure with complex, homegrown solutions.

“The rich cloud API was pivotal in creating a functioning deployment model. We also set up automated alerts to Slack at deploy-time.” - Jacobo Blanco, Data Science Manager

Business Impact

Upon adopting Prefect, the Paidy team was able to solve their pain on failure. Where it previously took them 2+ hours to identify and manually solve their failed jobs, they are now able to identify the problem in minutes and react accordingly.

At scale, this has resulted in thousands of man-hours saved in the last year alone. From manually debugging to single-click reruns, the Paidy team has accelerated their time to recovery by orders of magnitude.

The Paidy team has also achieved their goal of empowering their data scientists to work independently, from idea to ETL to production.

Conclusion

Paidy operationalized their dataflows with Prefect, removing pain on failure and allowing them to democratize their entire analytics and data science pipeline. If jobs fail, Paidy knows exactly what to do and can do it in minutes.

With data scientists taking projects from idea to production on their own, Paidy has dramatically increased their data science output without dramatically growing the team.

Posted on Jun 9, 2022
Case Study
GitHub
CircleCI
Terraform
AWS ECR
AWS EC2
DataDog
PostgresSQL
AWS Athena
Looker
Spark/S3/Glue/Parquet
Debugging
Error Handling
Dynamic DAGs
Scheduling
Monitoring
Notifications
Retries

Love your workflows again

Orchestrate your stack to gain confidence in your data