At Prefect Summit, infrastructure veteran Chris Riccomini shared insights about four emerging trends that are transforming how we build and operate modern systems. As someone who has spent two decades in Silicon Valley engineering infrastructure at companies like PayPal and LinkedIn, Chris brings a unique perspective to these developments. Here's my analysis of the key trends he highlighted and their implications for the future of infrastructure.
One of the most significant shifts in modern system design is the growing adoption of object storage solutions like Amazon S3. Traditionally, systems managed their own persistence, leading to complex challenges around data replication and consistency. Chris outlined three emerging approaches to incorporating object storage:
The implications are profound: simpler operational processes, easier system integration, and greater flexibility in balancing cost, latency, and durability. Perhaps most intriguingly, this shift could fundamentally change how we think about data integration, potentially reducing our reliance on message queues and traditional ETL processes.
The second trend Chris identified is the fascinating decomposition of traditionally monolithic databases into separate, reusable components. Modern projects are breaking apart core database functionalities – query parsing, optimization, and execution – into shared libraries that can be used across different database implementations.
This decomposition, enabled by projects like Apache Data Fusion and Meta's Velox, is creating three distinct opportunities:
The trade-off question Chris raised is particularly interesting: Will this standardization commoditize databases, or will it enable innovation in other areas?
The exponential growth of PostgreSQL adoption caught my attention. Chris attributes this surge to several factors:
This extensibility has created a powerful flywheel effect: as more developers adopt Postgres, infrastructure engineers are more motivated to build on top of it, which in turn attracts more users. We're seeing this play out in two ways:
The result? A smaller operational footprint as Postgres increasingly serves multiple use cases, from OLAP workloads to vector search.
The final trend Chris discussed – durable execution – might be the most complex but also the most promising. As systems become more distributed, ensuring exactly-once semantics across multiple services becomes increasingly challenging. Durable execution frameworks attempt to solve this by logging all operations and managing replays after failures.
While powerful, current implementations often require deep understanding of concepts like determinism and careful management of code deployments. Chris highlighted three emerging approaches that might make durable execution more accessible:
The evolution of these approaches, particularly in tools like Prefect 3, suggests we're moving toward more practical and maintainable solutions for ensuring reliable execution in distributed systems.
These trends point to a future where infrastructure becomes both more powerful and more accessible. The decomposition of monolithic systems, whether databases or storage layers, combined with new paradigms for ensuring reliability, suggests we're entering an era of more flexible and maintainable infrastructure.
For developers and architects, understanding these trends is crucial for making informed decisions about system design and technology choices. As Chris's insights reveal, we're not just seeing incremental improvements but fundamental shifts in how we build and operate modern systems.
Chris Riccomini is a software engineer, startup investor, and advisor, and author with more than 15 years of experience at major tech companies such as PayPal, LinkedIn, and WePay. He has been involved in open source throughout his career and is the co-author of Apache Samza and SlateDB. He's co-author of The Missing README: A Guide for the New Software Engineer, and writes Materialized View, a weekly infrastructure newsletter.
