Fighting Rust’s borrow checker
2025-03-28
This is a common phrase I come across in Rust forums and IRC rooms. And I’ve got the taste of what it is after I finished porting hedge to Rust (called hedge-rs, obviously). Coming from C/C++/Go/Zig, working with Rust’s borrow checker needs some getting used to. But just to get the thing working, at least initially, I had to do some bits of circumvention, which feels like “dirty” tricks to me at the moment. I’m sure I’ll learn some of the more idiomatic ways of doing these in the (near) future as I write more Rust code. Here are some of those observations.
Programming · Programming · Rust · Rust · Rustlang
4 minutes
2025-03-18
Last year, I explored the possibility of adding a complementary systems programming language to our stack. And I mentioned that I was leaning more into Zig as it resonates with me and my biases. Well, I spent a bit of my time last week porting spindle to Rust (called spindle-rs). This blog is me sharing some of my initial impressions. The first thing is, I quite like it. Which, for me, is a very important criteria. Mind you, as a piece of software, spindle-rs is not really that complicated, so I’m barely touching the surface of what it’s going to be like writing system software fit for Rust. However, from that experience, here are some of my impressions.
Programming · Programming · Rust · Rust · Rustlang
6 minutes
Considerations on deploying LLM-based workflows
2025-02-25
Let me start with a disclaimer. These thoughts come from a startup perspective where funds and personnel are a bit scarce. There’s a lot of generalizations and assumptions made here as well; take them with a pinch of salt. In just a short span of time, the speed of improvements to LLMs, especially the mainstream ones, are short of astonishing. The massive ones are getting smarter, faster, and more accurate. And even better, the open ones, such as Llama, DeepSeek, Gemma, Qwen, etc. are also catching up, which is a good thing, as I’m more interested in them. And for enterprises who are looking into integrating LLMs into their internal workflows, or even products, the options available now are so many it’s quite confusing where to even start. I hope this blog will shed some light on some of these confusions.
AI · Deployment · Genai · Llm · Programming · Software · Systems · Tech
6 minutes
Static-linked CGO binaries using musl and Zig
2025-02-15
The ability to produce statically-linked binaries by default in Go is one of the many good reasons why I appreciate and use it. Being able to copy or move only a single file around across all sorts of locations without worrying too much about missing libraries or runtimes has definitely saved me a lot of annoyances many times over. However, working on hedge-cb, and therefore, CGO, for the past few days, reminded me once again how much of a good thing static binaries are. So I tried checking out whether I could still do static linking with CGO.
Cgo · Golang · Libc · Musl · Programming · Software · Systems · Tech · Zig
3 minutes
Cluster membership management on AWS
2025-02-07
In continuation with my previous post, I have now finished porting hedge to AWS. It’s a trimmed-down version for now; only the features directly related to cluster membership are ported. I decided to make a separate repo, called hedge-cb (in keeping with the -cb theme), instead of updating hedge directly. And it’s mainly due to CGO. I didn’t really fancy the idea of introducing CGO to hedge as it could break a lot of the CI builds at work. I had to extract the shared protobuf definitions to a separate repo however, which is a breaking change to hedge. But at least it’s only a version change (v2) as opposed to adding CGO.
Aws · Cgo · Clockbound · Cluster · Distributed-Systems · Ffi · Go · Golang · Hedge · Leader-Election · Memberlist · Programming · Software · Systems · Tech · Timesync · True-Time
2 minutes
Distributed locking on AWS (ClockBound)
2025-02-02
After some testing time, I now have a working port of spindle in AWS. There were some slight changes from the original library to account for some of the differences between Cloud Spanner and PostgreSQL, but not really by much. It’s called spindle-cb, if you’re interested. It’s still half the battle though; I still have to port hedge as well before I could really use it for some of the planned projects in my pipeline.
Aws · Cgo · Clockbound · Distributed-Locking · Ffi · Go · Golang · Locking · Programming · Software · Spindle · Systems · Tech · Timesync · True-Time
2 minutes
AWS ClockBound client for Go (update)
2025-01-27
A week ago I published a short blog about clockbound-client-go. After some testing, turns out there’s an issue in reading the actual time from ClockBound’s shared memory segment; it seems to provide only the elapsed time since boot. However, using the Rust client and the FFI bindings produce the correct results. Either there is a problem in the code that reads the shared memory segment, or the SHM contents are wrong. Or, the contents are actually correct, but the expectation is for the implementing client to figure out the bounded time from the available values.
Aws · Cgo · Clockbound · Ffi · Go · Golang · Programming · Software · Systems · Tech · Timesync · True-Time
1 minute
2025-01-22
I’ve written a Go client for AWS ClockBound called clockbound-client-go. It uses the newer, shared memory segment protocol instead of the older, socket-based protocol. This is a prerequisite library needed to port spindle (and maybe even hedge) to AWS (for an upcoming project). As great a tech Google’s TrueTime is, there is no available API for it. It is only through Spanner that spindle achieves its locking mechanisms with TrueTime. Currently, it’s the cheapest way, surprisingly, to do distributed locking that I’ve tried so far, compared to the likes of Redis, Zookeeper, etcd, Consul, etc. Yes, I could do VPC peering between GCP and AWS but it is quite costly at the moment.
Aws · Clockbound · Go · Golang · Programming · Software · Systems · Tech · Timesync · True-Time
1 minute
2024-12-28
I remember ages ago, a friend asked me what my tips would be to transition from a typical programmer to a “systems programmer”. And I distinctly remember answering him along the lines of “Y’know what, I don’t really know, maybe just get more experience?” And now, being an engineering leader to some extent, I get this question once in a while, but even now, I don’t really have a default answer, and a good one at that, to give. And writing this blog is, in part, me trying to remedy that.
Programming · Programming · Software · Software · Systems · Tech
6 minutes
How valuable is your source code?
2024-10-18
I had a think about this topic when I saw a recent Ars Technica news article about Winamp releasing their source code to GitHub and then deleting it after some backlash from the open source community. And we also hear about instances of leaked source codes as well, such as that of The New York Times, Okta, Intel’s UEFI source, Mercedes Benz, Microsoft, among others. Until now, I haven’t really put too much thought into this. I mean, from a business perspective, especially software companies who earn money through code, how valuable do we think our source code really is?
Programming · Programming · Software · Software · Tech
6 minutes
2024-10-01
An LLM creating technical debt with the spelling. In this post, I want to talk a little bit about technical debt. In a recent conversation I had with one of our engineers, we sort of inadvertently enumerated some of the more pressing technical debts we have off top of our heads in our current infrastructure, which, now that I think about it, is quite a lot. But what is “technical debt”, really?
Programming · Software · Tech-Debt · Technical-Debt · Zig
4 minutes
2024-09-30
In continuation with my previous post about the new systems programming languages, I mentioned I was considering Zig as a potential complementary systems language to our main one, which is Go(lang). Well, for the past month or so, in my spare time, I tried writing something more substantial in it to understand the language more. For some time now, I’ve been “itching” to write something similar to Hashicorp’s memberlist library, but in a lower-level language for performance, smaller footprint and minimal network load. Now, I’ve used memberlist before, and it is a superb piece of code, but I wanted something that supports a consistent leader across the whole fleet. It’s a requirement to a system I plan on building in the near future (more on this in a later post). My top choices were C, Rust, and Zig, and as I said, I took a liking to Zig due to its promised simplicity, so I wrote it in Zig. The project is called zgroup, and you can check it out on GitHub if you’re interested. It’s still similar to memberlist but with the added capability of electing a leader across the whole group. It uses both the SWIM Protocol, which memberlist uses, and Raft’s leader election algorithm.
Distributed-Systems · Programming · Programming · Raft · Swim · Zig · Zig · Ziglang
2 minutes
Thoughts on the newer systems programming languages
2024-08-20
I’m talking about the new crop of systems programming languages that advertise themselves as better replacements for C and/or C++: Rust, Zig, D, Odin, Nim , etc. I’m using the word “new” loosely here as Rust and Zig, for example, are almost a decade old now. The topic of systems programming has been on my radar (again) recently at work due to our attempts at improving the performance of some of the more critical parts of our stack. At Alphaus, we use Go as our main programming language and as much as I like Go, there are still areas in our infrastructure that could be served better with non-GC languages.
Programming · Programming · Systems · Zig
6 minutes