Introducing Luna, an in-memory SQL cache
2025-10-01
Introducing another side-project of mine: Luna. Luna is part of my ongoing familiarization attempts with Rust. In my previous blog entries, I’ve mentioned that I’ve decided to choose Rust as the additional non-GC systems programming language to compliment our use of Go at work. Ever since, there were several attempts to write some of our critical systems in Rust but due to insufficient know-how, it didn’t materialize yet. Recently however, one of our core data processing engine, Sapphire, has been nearing its limits in terms of scale. Being mainly a data ingestion and processing engine, one of its bottlenecks is how fast it can query data from BigQuery. Now, BigQuery is actually quite performant as long as you’re on top of its scaling pecularities. It is a bit pricey though. So we started to explore the idea of caching some of the heavy ones instead of upgrading BigQuery’s compute capabilities across the board; sort of Redis for columnar SQL data.
Duckdb · Duckdb · Oss · Oss · Sql · Sql
2 minutes