Introducing another side-project of mine: Luna.
Luna is part of my ongoing familiarization attempts with Rust. In my previous blog entries, I’ve mentioned that I’ve decided to choose Rust as the additional non-GC systems programming language to compliment our use of Go at work. Ever since, there were several attempts to write some of our critical systems in Rust but due to insufficient know-how, it didn’t materialize yet.
Recently however, one of our core data processing engine, Sapphire, has been nearing its limits in terms of scale. Being mainly a data ingestion and processing engine, one of its bottlenecks is how fast it can query data from BigQuery. Now, BigQuery is actually quite performant as long as you’re on top of its scaling pecularities. It is a bit pricey though. So we started to explore the idea of caching some of the heavy ones instead of upgrading BigQuery’s compute capabilities across the board; sort of Redis for columnar SQL data.
That’s where Luna comes in. Although, specifically, it is really DuckDB’s in-memory SQL capabilities that is being leveraged here. Luna is just the host process for an embedded, in-memory DuckDB instance. But that makes Luna a fully-pledge SQL database server. We just have to add the must-haves and nice-to-haves expected of cache servers such as an auth mechanism, distributed capabilities, scale, and some degree of high availability. And I think Rust is a good choice here: precise control of memory, with strong security defaults.
I’ve decided to make it open source as well. I think this is a good software to be open-sourced; certainly not the first (there are already several out there), but I think the difference will be in terms of governance and roadmap.
So, if this is right up your alley, check it out on GitHub. And contribute, if you can!