Introducing another side-project of mine: Luna.

Luna is part of my ongoing familiarization attempts with Rust. In my previous blog entries, I’ve mentioned that I’ve decided to choose Rust as the additional non-GC systems programming language to compliment our use of Go at work. Ever since, there were several attempts to write some of our critical systems in Rust but due to insufficient know-how, it didn’t materialize yet.

Recently however, one of our core data processing engine, Sapphire, has been nearing its limits in terms of scale. Being mainly a data ingestion and processing engine, one of its bottlenecks is how fast it can query data from BigQuery. Now, BigQuery is actually quite performant as long as you’re on top of its scaling pecularities. It is a bit pricey though. So we started to explore the idea of caching some of the heavy ones instead of upgrading BigQuery’s compute capabilities across the board; sort of Redis for columnar SQL data.

That’s where Luna comes in. Although, specifically, it is really DuckDB’s in-memory SQL capabilities that is being leveraged here. Luna is just the host process for an embedded, in-memory DuckDB instance. But that makes Luna a fully-pledge SQL database server. We just have to add the must-haves and nice-to-haves expected of cache servers such as an auth mechanism, distributed capabilities, scale, and some degree of high availability. And I think Rust is a good choice here: precise control of memory, with strong security defaults.

I’ve decided to make it open source as well. I think this is a good software to be open-sourced; certainly not the first (there are already several out there), but I think the difference will be in terms of governance and roadmap.

So, if this is right up your alley, check it out on GitHub. And contribute, if you can!

When building, and especially testing, Vortex, we use several cloud VMs with different kernel versions (usually LTS ones), and a dedicated GKE test cluster. The trouble with eBPF though, is that a lot of the features depend on what kernel version is running on the target system. And maintaining multiple cloud VMs with different kernel versions is a tad expensive, and really not that straightforward.

And so, QEMU to the rescue. When it comes to emulators, QEMU doesn’t really need introductions. This blog is a guide (mostly to myself) on how to build a Debian-based image with a specific kernel version to be used as an eBPF testbed.

This guide uses an Ubuntu-based system. This also works with a cloud VM with KVM enabled. First, let’s install our dependencies.

$ sudo apt update
$ sudo apt git install make gcc flex bison libncurses-dev libelf-dev libssl-dev \
  debootstrap dwarves qemu-system -y

Next, we clone the stable version of the Linux kernel. I’ll be using workdir as the working directory.

$ mkdir -p workdir/
$ cd workdir/
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git

# Checkout desired version (tag):
$ cd linux-stable/
$ git checkout -b v6.6.102 v6.6.102

After several trial and error, the following config seems to work for eBPF testing.

$ make defconfig
$ make kvm_guest.config

# Required for Debian Stretch and later:
$ ./scripts/config --set-val CONFIG_CONFIGFS_FS y
$ ./scripts/config --set-val CONFIG_SECURITYFS y

# BPF-related configs:
$ ./scripts/config --set-val CONFIG_BPF y
$ ./scripts/config --set-val CONFIG_BPF_SYSCALL y
$ ./scripts/config --set-val CONFIG_MODULES y
$ ./scripts/config --set-val CONFIG_BPF_EVENTS y
$ ./scripts/config --set-val CONFIG_PERF_EVENTS y
$ ./scripts/config --set-val CONFIG_HAVE_PERF_EVENTS y
$ ./scripts/config --set-val CONFIG_FUNCTION_TRACER y
$ ./scripts/config --set-val CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT y
$ ./scripts/config --set-val CONFIG_DEBUG_INFO y
$ ./scripts/config --set-val CONFIG_DEBUG_INFO_BTF y
$ ./scripts/config --set-val CONFIG_NET_CLS_BPF y
$ ./scripts/config --set-val CONFIG_NET_ACT_BPF y
$ ./scripts/config --set-val CONFIG_NET_SCH_INGRESS y
$ ./scripts/config --set-val CONFIG_BPF_JIT y
$ ./scripts/config --set-val CONFIG_HAVE_BPF_JIT y
$ ./scripts/config --set-val CONFIG_CGROUP_BPF y
$ ./scripts/config --set-val CONFIG_KPROBES y
$ ./scripts/config --set-val CONFIG_HAVE_KPROBES y
$ ./scripts/config --set-val CONFIG_KPROBE_EVENTS y
$ ./scripts/config --set-val CONFIG_KPROBES_ON_FTRACE y
$ ./scripts/config --set-val CONFIG_UPROBES y
$ ./scripts/config --set-val CONFIG_UPROBE_EVENTS y
$ ./scripts/config --set-val CONFIG_ARCH_SUPPORTS_UPROBES y
$ ./scripts/config --set-val CONFIG_MMU y
$ ./scripts/config --set-val CONFIG_TRACEPOINTS y
$ ./scripts/config --set-val CONFIG_HAVE_SYSCALL_TRACEPOINTS y
$ ./scripts/config --set-val CONFIG_FTRACE y
$ ./scripts/config --set-val CONFIG_FTRACE_SYSCALLS y

$ ./scripts/config --set-val CONFIG_CMDLINE_BOOL y
$ echo 'CONFIG_CMDLINE="net.ifnames=0"' >> .config

$ make olddefconfig

After setting up the build config, we can now build the kernel.

$ make -j$(nproc)

arch/x86/boot/bzImage is now our newly-built kernel. Next, let’s build a Debian-based (bullseye) image for QEMU to boot with our custom kernel.

$ cd ../
$ mkdir -p debian-bullseye/
$ cd debian-bullseye/
$ wget https://raw.githubusercontent.com/google/syzkaller/master/tools/create-image.sh
$ chmod +x create-image.sh
$ ./create-image.sh --feature full

bullseye.img is now our new Linux image. Let’s boot it using QEMU, mapping our local 10021 port to the VM’s SSH (22) port.

$ cd ../
$ qemu-system-x86_64 \
  -m 2G \
  -smp 2 \
  -kernel linux-stable/arch/x86/boot/bzImage \
  -append "console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
  -drive file=debian-bullseye/bullseye.img,format=raw \
  -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 \
  -net nic,model=e1000 \
  -enable-kvm \
  -nographic \
  -pidfile vm.pid \
  2>&1 | tee vm.log

Using a separate terminal, we can now use ssh and scp to access the VM and copy files to it. Let’s copy the vortex-agent binary to the home directory.

$ scp -i debian-bullseye/bullseye.id_rsa -P 10021 \
  -o "StrictHostKeyChecking no" \
  $VORTEX_AGENT_ROOT/bin/vortex-agent \
  root@localhost:~/

Finally, we can ssh to the VM.

$ ssh -i debian-bullseye/bullseye.id_rsa -p 10021 \
  -o "StrictHostKeyChecking no" root@localhost

# Then run the binary:
$ ./vortex-agent run --logtostderr

To close the VM, we can either do:

$ poweroff

from within the VM, or kill the process from outside:

$ kill [-9] $(cat vm.pid)

Related blogs:

1. On building Vortex
2. This blog

A quick update on what I’ve been building for the last couple of weeks.

I’ve put together a small team at Alphaus to explore eBPF with a specific goal: building a tool for AI security and monitoring. The kernel-level visibility you get with eBPF is perfect for this, but the real work is in making it useful.

Our immediate focus has been on getting visibility into encrypted network traffic. We needed a way to inspect TLS traffic without messing with certificates or acting as a man-in-the-middle. The obvious path was using eBPF uprobes to hook user-space functions directly.

We’re calling the tool Vortex (thought it sounded cool). It looks something like this:

And I’m happy to report we’ve had our first major successes. We can now reliably see plaintext traffic for apps built on Python (and other OpenSSL-based apps). We can now hook into the SSL_read[_ex] and SSL_write[_ex] functions in the underlying OpenSSL library that Python’s ssl module (and many other tools) uses. This gives us the unencrypted buffer just before it’s handed to the kernel for sending, or just after it’s been received and decrypted.

Still in progress but the same fundamental approach works for NodeJS applications, although they have their own TLS stack. Fortunately for us, they also use the same function names so we can attach to those as well, and pull the plaintext data out.

This means we can now monitor the raw traffic of a huge range of services without any code instrumentation or network configuration changes.

Why are we doing this? The goal is to feed this traffic data into our monitoring agent. From there, we can do interesting work like detecting anomalous traffic patterns to (and from) AI models, identifying potential data exfiltration over encrypted channels, getting a clear map of what your AI services are actually talking to, etc.

It’s not trivial, of course. Dealing with function offsets, different library versions, and the nuances between runtimes is a challenge, but the core mechanism is solid.

More to share soon as we build out the next pieces.

Sidenote: This project actually made me excited in programming once again as it involves tinkering with the low-level guts of the Linux kernel, which I love.

The source code is here, if you’re interested.

Related blogs:

1. This blog
2. Using QEMU for eBPF testing

Apologies for being silent for a while. Still alive here; still doing CTO work for Alphaus. But at the same time, since about the start of the year, I started another startup here in Tokyo with two other cofounders. Long story short - it didn’t really pan out, at least for me. I mean, the startup is still moving forward, but with me no longer in it. I “resigned”, just recently actually, due to founder conflict. There was a lot going for it; it got accepted into the Antler Japan accelerator program (Batch 4) this year. It also received an infra backing from Google Cloud worth about $100K. Anyway, it is what it is, and I have to move on.

To some extent, I’m actually quite relieved that it didn’t really work out for me as I was almost at the brink of burning out. I’ve spent most of my non-work time (late nights, weekends, holidays) for that startup that it affected my health and personal relationships (not surprising, really), which would have been disastrous.

Now, I’m working on another startup (again) in my spare time, and in my own pace. I’m still with Alphaus during normal work times but I’ve been building this new one slowly and steadily. It’s still bootstrapped at the moment; I don’t think it’s a good idea, nor good timing, to entertain any form of outside investments at this point given its current state. My CEO knows about it, and he’s very supportive.

I might talk about it at some point in the future but in the meantime, I’ll continue building it in stealth mode.

Just so you know.

For context, hedge-rs is a non-async, multithreaded code that runs on multiple nodes communicating with each other using an internal protocol on top of TCP.

Lots of .clone()ing

In Rust, accessing a variable from another thread isn’t as straightforward. This doesn’t compile.

fn main() {
    let args: Vec<String> = env::args().collect();

    thread::spawn(|| {
        println!("args1: {}", args[1]);
    });
}

The new thread is “borrowing” args from the calling function which it could outlive. When args goes out of scope, the thread could be accessing args that is no longer there.

We could do a “move” by doing this.

fn main() {
    let args: Vec<String> = env::args().collect();

    // Move args to the new thread.
    thread::spawn(move || {
        println!("args1: {}", args[1]);
    });
}

This will now compile (args is now moved to the new thread), but then I couldn’t do this anymore.

fn main() {
    let args: Vec<String> = env::args().collect();

    // Move args to the new thread.
    thread::spawn(move || {
        println!("args1: {}", args[1]);
    });

    // args was now moved to the other thread.
    println!("args2: {}", args[2]);
}

Even if we argue that this code is correct as we are not changing any values; just reading it, this won’t compile. We can clone (or copy) instead. This will work.

fn main() {
    let args: Vec<String> = env::args().collect();

    // Create a clone of args; move to new thread.
    let args_clone = args.clone();
    thread::spawn(move || {
        println!("args1: {}", args_clone[1]);
    });

    // args is still available to us here.
    println!("args2: {}", args[2]);
}

There are several of these in hedge-rs. I don’t really see this as a big problem for small, stack-allocated variables, but it might be for big-ish structs or arrays/vectors.

Arc<Mutex<T>> for mutable objects

To pass around mutable objects across threads, Rust provides Arc<Mutex<T>>, which is a mutex inside an “Atomically Reference Counted” pointer. This allows us to mutate variables across threads in a thread-safe manner.

#[derive(Debug)]
struct X {
    val: i32,
}

impl X {
    fn inc(&mut self) -> i32 {
        self.val += 1;
        self.val
    }
}

fn main() {
    let x = Arc::new(Mutex::new(X { val: 0 }));

    let x_clone = x.clone();
    thread::spawn(move || {
        println!("{:?}", x_clone.lock().unwrap().inc());
        // mutex unlocks when out of scope
    });

    println!("{:?}", x.lock().unwrap().inc());
    // mutex unlocks when out of scope
}

In this code, x_clone here looks like our previous example but it isn’t. For Arcs, you clone the pointer to increase its reference count, not clone the inner object itself. x_clone and x here still points to the same X object. Arc will only cleanup its inner object once the reference count goes to zero.

This is actually alright for me, albeit the weird ergonomics of .clone(), and .lock() with its implicit unlock pair when going out of scope. What is slightly unexpected is if it’s for atomics; I still have to wrap an atomic with an Arc as well. For example,

fn main() {
    let a = Arc::new(AtomicU32::new(0));

    let a_clone = a.clone();
    thread::spawn(move || {
        a_clone.store(1, Ordering::Relaxed);
        println!("{}", a_clone.load(Ordering::Acquire));
    });

    a.store(2, Ordering::Relaxed);
    println!("{}", a.load(Ordering::Acquire));
}

I feel like there’s not really a need for Arc here since it’s, you know, atomic.

Using Vec<T> for late initialization of objects

This one for me definitely feels like a cheat. Say you have a struct field in your other struct that you want to initialize later. Like this Lock inside Op.

pub struct Op {
    ...
    lock: Arc<Mutex<Lock>>,
    tx_worker: Sender<WorkerCtrl>,
    ...
}

Instantiating Op without setting up both lock and tx_worker is not that straightforward, like:

let o = Op { lock: ???, tx_worker: ??? };
...
// Initialize o.lock here.
...
// Initialize o.tx_worker here.
...

What did I do? Use vectors instead.

pub struct Op {
    ...
    lock: Vec<Arc<Mutex<Lock>>>,
    tx_worker: Vec<Sender<WorkerCtrl>>,
    ...
}

fn main() {
    // So I can do this:
    let mut o = Op { lock: vec![], tx_worker: vec![] };
    ...
    // Late initialization of lock.
    o.lock = vec![Arc::new(Mutex::new(
        LockBuilder::new()
            .db(self.db.clone())
            .table(self.table.clone())
            .name(lock_name)
            .id(self.id.clone())
            .lease_ms(lease_ms)
            .leader_tx(Some(tx_ldr))
            .build(),
    ))];
    ...
    // And late initialization of o.tx_worker.
    let (tx, rx): (Sender<WorkerCtrl>, Receiver<WorkerCtrl>) = channel();
    o.tx_worker = vec![tx.clone()];

    // I can then use lock like so:
    if o.lock.len() > 0 {
        // use o.lock[0]
    }

    // Same with tx_worker.
    if o.tx_worker.len() > 0 {
        // use o.tx_worker[0]
    }
}

It doesn’t feel right, but it works.

The first thing is, I quite like it. Which, for me, is a very important criteria. Mind you, as a piece of software, spindle-rs is not really that complicated, so I’m barely touching the surface of what it’s going to be like writing system software fit for Rust. However, from that experience, here are some of my impressions.

Tooling is pretty good. Toolchain installation is handled by rustup. It allows for easy switching between stable, nightly, and beta. The rust-analyzer, which includes the Rust LSP server, works out of the box, and works as how you’d expect it to work most of the time, albeit a tad slow at times. rustfmt is good with decent defaults. And cargo, Rust’s package manager, is excellent. Coming from C and C++, and having experienced CMake, Meson, Ninja, and vcpkg; cargo, with its tight integration with Rust, is actually a breath of fresh air, especially its ease of use. The overall development experience as far as tooling is concerned, is top notch.

As a language, well, it takes some getting used to. The biggest friction for me comes from “fighting the borrow checker” (as Rustaceans term it). No surprise there as it’s what really makes Rust, Rust. My mental programming model, which translates pretty well between C, C++, Go, and Zig, doesn’t map that well with Rust’s semantics. Not yet, at least. This is not a criticism, of course. To program in Rust just means doing it “the Rust way”. I just have to learn it with experience.

Syntax-wise, nothing really stood out to me. It feels similar to Zig. If I don’t put too much thought into the flow structure, it’s easy to end up with very deep nested code due to how pattern matching works in combination with Result and Option. Something like:

pub fn run(&mut self) {
    ...
    thread::spawn(move || {
        ...
        loop {
            // First, see if already locked (could be us or somebody else).
            let (tx, rx): (Sender<DiffToken>, Receiver<DiffToken>) = channel();
            if let Err(_) = tx_main.send(ProtoCtrl::CheckLock(tx)) {
                continue;
            }

            let mut locked = false;
            match rx.recv() {
                Ok(v) => {
                    'single: loop {
                        // We are leader now.
                        if token.load(Ordering::Acquire) == v.token as u64 {
                            ...
                            break 'single;
                        }

                        // We're not leader now.
                        if v.diff > 0 {
                            ...
                            break 'single;
                        }

                        break 'single;
                    }
                }
                Err(_) => continue,
            }

            if locked {
                continue;
            }

            if initial {
                // Attempt first ever lock. The return commit timestamp will be our fencing
                // token. Only one node should be able to do this successfully.
                initial = false;
                let (tx, rx): (Sender<i128>, Receiver<i128>) = channel();
                if let Ok(_) = tx_main.send(ProtoCtrl::InitialLock(tx)) {
                    if let Ok(t) = rx.recv() {
                        if t > -1 {
                            token.store(t as u64, Ordering::Relaxed);
                        }
                    }
                }
            } else {
                // For the succeeding lock attempts.
                let (tx, rx): (Sender<Record>, Receiver<Record>) = channel();
                if let Ok(_) = tx_main.send(ProtoCtrl::CurrentToken(tx)) {
                    if let Ok(v) = rx.recv() {
                        if v.token < 0 {
                            continue;
                        }

                        // Attempt to grab the next lock. Multiple nodes could potentially
                        // do this successfully.
                        let mut update = false;
                        let mut token_up: i128 = 0;
                        let mut name = String::new();
                        write!(&mut name, "{}_{}", lock_name, v.token).unwrap();
                        let (tx, rx): (Sender<i128>, Receiver<i128>) = channel();
                        if let Ok(_) = tx_main.send(ProtoCtrl::NextLockInsert { name, tx }) {
                            if let Ok(t) = rx.recv() {
                                if t > 0 {
                                    update = true;
                                    token_up = t;
                                }
                            }
                        }

                        if update {
                            // We got the lock. Attempt to update the current token
                            // to this commit timestamp.
                            let (tx, rx): (Sender<i128>, Receiver<i128>) = channel();
                            if let Ok(_) = tx_main.send(ProtoCtrl::NextLockUpdate { token: token_up, tx }) {
                                if let Ok(t) = rx.recv() {
                                    if t > 0 {
                                        // Doesn't mean we're leader, yet.
                                        token.store(token_up as u64, Ordering::Relaxed);
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    });

    ...
}

In Zig, it’s easier to tame very deep nesting due to its support for labeled blocks; most blocks can be labeled, and you can do early returns with break :label;. There is also support for labeled blocks in Rust but, as far as I’m aware, only for loop and let blocks.

For example, in Zig, you can do something like:

label: {
    const on = (msg.proto2 & 0x8000000000000000) >> 63;
    if (on == 0) break :label; // early return

    const lmin = ((msg.proto2 & 0x7FFFFFFF80000000) >> 31) * 1000;
    const lmax = ((msg.proto2 & 0x700000007FFFFFFF)) * 1000;

    @atomicStore(
        u64,
        &self.elex_tm_min,
        lmin,
        std.builtin.AtomicOrder.seq_cst,
    );

    ...
}

And then, there’s async Rust. The only experience I have with async programming (with async/await and its viral nature) was with C#, and quite frankly, I didn’t really warm up to the idea. I understand the need for a fast, runtime-assisted, concurrent programming model, but coming from Go, async Rust feels like a totally different language. At least in Go, code for concurrent programming using goroutines and channels still looks and feels like normal Go code. Mixing concurrent with synchronous Go code doesn’t feel “weird”. Not so with async Rust. When writing spindle-rs, I had to use the google-cloud-spanner crate, which is async only. Since I didn’t want to write spindle-rs in async, I had to “bridge” the async crate with my sync code. Mixing async with sync Rust code feels “weird”. At the moment, my understanding of how Tokio’s runtime works (Tokio seems like “the runtime” to use when doing async Rust) is not deep enough for me to know whether it’s even a good idea to mix the two. Now, with that said, this is, again, not a criticism to async Rust. I can even see myself using it when doing embedded programming.

The only real complaint I have at the moment is the slow compile times. I know that this is being worked on at the moment, so I won’t take it against Rust too much for now. With all the criticisms hurled towards Go, they’ve done it right in terms of compilation times. Even Zig has much better compilation times despite its age. But I’m sure it’ll get better in time.

So, going back to my choices: personally, I enjoy writing in Zig more so I will continue using it for personal projects, but I can’t, in good conscience, push it to the company; not until it becomes 1.0+ with a relatively mature ecosystem of libraries. Then again, we’ve probably established our Rust usage by that time anyway, so it doesn’t make sense to “replace” it by the time Zig becomes viable. So there you go. Rust it is. And I believe it’s objectively the better choice.

In just a short span of time, the speed of improvements to LLMs, especially the mainstream ones, are short of astonishing. The massive ones are getting smarter, faster, and more accurate. And even better, the open ones, such as Llama, DeepSeek, Gemma, Qwen, etc. are also catching up, which is a good thing, as I’m more interested in them. And for enterprises who are looking into integrating LLMs into their internal workflows, or even products, the options available now are so many it’s quite confusing where to even start. I hope this blog will shed some light on some of these confusions.

One of the first things to consider is whether to go AIaaS (AI as a service, or external) or deploy internally. While I don’t really have a problem with using Gemini, ChatGPT, or Sonnet for explorations, I think when internal data or knowledge-base are involved, hybrid deployment is the way to go. Hybrid in this sense means hosting LLMs internally for more critical information (for obvious reasons), and using external LLMs for the rest. Ultimately, the decision of what to use, whether external, hybrid, or internal, will depend on your company’s data privacy policies, governance, and regulations. I won’t be digging further into going external as it’s more about tracking and controlling what information are being included in the prompts than deployment, so we treat it as we do with any other API-based vendor integrations. Internal deployment, on the other hand, we generally have two options; a) fine-tuning a closed LLM, and b) using open models.

Fine-tuning a closed LLM really depends on whether that feature is provided by the vendor. For example, OpenAI’s ChatGPT, and Google’s Gemini can be fine-tuned with your own data, and you can use the fine-tuned version as your new LLM; the vendor will still host that LLM for you, you use it as you would with any other LLMs, which is through its API. Now, you might argue that it’s technically not an internal deployment, and you might be right. I only categorized it as such since in doing so, there’s almost always the provision from the vendor effectively promising to not use your fine-tuning data as training data for their next LLM versions, so there’s an assumption of privacy there; whether you trust them or not is up to you.

The second one; using open models, is really what you want. The idea is that, for enterprise use, you don’t really need, say, ChatGPT’s vast, generic knowledge of the world. You want an LLM that understands your hiring and onboarding policies instead of knowing the historical weather data of the Atlantic for the past decade; you want its inferencing (or “thinking”) capabilities applied to your internal data instead. In this route, you can either fine-tune an open model with your data, all hosted internally, or use an open model as is, and use RAG to augment, and ground it with your internal data, also hosted internally.

So, RAG? Or fine-tuning? I think doing both is the way to go. The general rule of thumb (advise I got from the Gemma Japan team) is fine-tune for static (or near-static) data, and RAG for more dynamic, always-changing data.

Considerations for fine-tuning are expertise, and costs. To fine-tune an LLM, you need quality training/fine-tuning data. And since enterprise data are usually all over the place, and often, not centralized, data preparation is actually one of the biggest hurdles. You probably need a team of data engineers, data scientists, and infra/ops personnel to pull this off. And the costs will involve the upfront costs of the actual fine-tuning (you need compute, both CPU and GPU/TPU), and the ongoing inferencing (or serving) costs, which will involve compute (CPU/GPU/TPU), storage, and networking.

Considerations for RAG are also expertise, and costs. Setting up RAG-based workflows involves some important components: LLM routers, embeddings generators, and vector databases. You will most definitely be using multiple LLMs for multiple purposes; one for text summarization or generation, one for reasoning, another for research, or coding, and so on. Each LLM will be deployed separately; could be on a big VM, or a cluster of VMs. And since these clusters need GPUs, you’d probably want a serverless, scale-to-zero environment, as GPUs will bear most of the costs in this layer. You might be able to get away with traditional auto-scaling clusters (with additional checks in place to scale down during idle time) but environments like Kubernetes with, say, Keda for scale-to-zero, or Fly’s GPUs (which I believe can scale to zero), or GCP’s Cloud Run, etc., will be easier to manage. So with multiple LLM deployments, you’d also want a router that will route input requests, or prompts, to the appropriate LLM target. You can do this traditionally, utilizing an LLM to facilitate with the routing, or you could also do MoE (Mixture of Experts) deployments, where the routing is done within the LLM itself. One issue, however, is that there’s not a lot of open MoE models yet; there’s IBM’s Granite, Mistral-MoE, and Qwen-MoE models (I’m monitoring this space closely as well; I think there will be more improvements here in the near future).

Embeddings generators and vector databases are specific to RAG. To “feed” your enterprise data to an LLM (as opposed to fine-tuning), you need to generate “embeddings” for them first. Embeddings are vector representations (with semantic context) of your data. These embeddings will then be stored in a vector database for later use. The more data you have, or at least the more data you want an LLM to have access to, have, the more embeddings you will generate. How many of these will depend on the size of the context windows of the LLMs you choose. An example would be 1 page of a document is 1 embedding, or, if an LLM’s context window is smaller (looking at you, Gemma2), you could “chunk” your data into smaller pieces of defined length; 1 chunk will be 1 embedding, and so on and so on. Options for generators here are plenty; you can use the mainstream providers’ vector embedding APIs, such as OpenAI’s Vector Embeddings API, GCP’s Embeddings API, AWS’ Titan Text Embeddings, etc., or use open models, such as Word2Vec, LexVec, bert, chroma, etc., although you still need to host them internally. Choices for vector databases are also the same; you can use vendor-provided ones, or self-host open source ones. Note that these are still databases, so when self-hosting, you still need to tackle the headaches of deploying databases, even vendor-provided ones. Once these are in place, you do RAG by converting the input query, or prompt, to its embedding equivalent, do a semantic/similarity/distance search from your vector database, map the resulting embeddings to the real data you have, and then use those data as context to (or part of) your final prompt to the target LLM. So, still an ops-heavy deployment, as you can see, notwithstanding the costs, both upfront and operational, that you will incur in deploying these components.

I will write something about cost estimations or simulations regarding these deployments, and some ideas on actual deployments as well, but that will be on a separate blog.

As I mentioned, even though Go produces static binaries by default, introducing CGO into the mix makes your binary dynamically-linked instead, as it will now require a C runtime, usually glibc, to be present on the target environment. That in itself is not really such a bad thing as glibc is almost always present in most Linux distributions used in production. The annoyance, however, starts when you introduce a dependency through CGO, in which case, you will need to make sure that that dependency’s lib***.so is also present in your target environment, alongside the C runtime.

For a binary that depends on hedge-cb, and therefore, ClockBound, deploying it to, say, EC2 (it’s AWS-native after all), requires the following to be present in the VM:

• the ClockBound daemon running - our source for “true time”;
• libclockbound.so copied to /usr/lib/ or /usr/lib64/ - output when compiling ClockBound FFI;
• a C runtime - Amazon Linux comes with glibc preinstalled.

There’s no getting around the ClockBound daemon as it’s functionally required. But I can do away with libclockbound.so and glibc altogether if the binary is statically built. I will share what I did in this blog, although this is not the only way how. Now, there’s a fair bit of discussions on the interwebs about the pros and cons of static binaries when it comes to the C runtime; I’m not going to rehash them here. For example purposes only, I went with musl, a lightweight alternative to glibc, and the Zig toolchain. Zig here is probably optional as musl comes with musl-gcc, which can serve as my CGO compiler, but I was struggling to make it work. Zig (zig cc to be specific), on the other hand, was a breeze.

First, let’s build ClockBound’s FFI. Rust environment is required here.

$ cd /tmp/ && git clone https://github.com/aws/clock-bound
$ cd clock-bound/clock-bound-ffi/
$ rustup target add x86_64-unknown-linux-musl
$ cargo build --release --target=x86_64-unknown-linux-musl

The build outputs of note here are libclockbound.so, which we don’t need, and libclockbound.a, which we do. Now, let’s build musl.

$ cd /tmp/ && wget https://musl.libc.org/releases/musl-1.2.5.tar.gz
$ tar xvzf musl-1.2.5.tar.gz && cd musl-1.2.5/ && ./configure && make && sudo make install

Next, copy the FFI archive library to musl’s install location.

$ sudo cp -v /tmp/clock-bound/target/x86_64-unknown-linux-musl/release/libclockbound.a /usr/local/musl/lib/

Finally, build the binary using zig cc as our CGO compiler. Here, I’m building hedge-cb’s sample code.

$ cd /tmp/ && git clone https://github.com/flowerinthenight/hedge-cb && cd hedge-cb/example/
$ cp /tmp/clock-bound/clock-bound-ffi/include/clockbound.h .
$ CC="zig cc -target x86_64-linux-musl -I. -L/usr/local/musl/lib -lunwind" \
  GOOS=linux GOARCH=amd64 go build -v --ldflags '-linkmode=external -extldflags=-static'

# Output should be a static binary.
$ ldd ./example
      not a dynamic executable

So, yes, we can build static binaries even with CGO; but now, I’m stuck with musl. I’m sure musl is a fine piece of software, but I’m not really that familiar with it. I’m aware we use it at work since many of our containers in production use Alpine Linux as base. But would I exchange glibc for musl just for static binaries? I’m not sure yet. Probably.

As always, tradeoffs.

Related blogs:

1. AWS ClockBound client for Go
2. AWS ClockBound client for Go (update)
3. Distributed locking on AWS (ClockBound)
4. Cluster membership management on AWS
5. This blog

What remains now in the short term is figuring out how to make this thing work on EKS. Since the ClockBound daemon is a requirement, EKS pods need to be able to read its shared memory segment backing file which is /var/run/clockbound/shm. I have to mount that file (or maybe the whole directory) to the pod. I’m not sure how it will behave though; it’s not just some regular file.

Anyway, as far as testing goes, I’ve tried it with both single- and multi-zone AutoScaling Groups and it works as expected. Time will tell how stable it’s going to be once deployed to some production environment with larger cluster sizes.

Feature-wise, my initial intention was to make the cluster membership management part working first. That includes the ability to track member nodes dynamically, member-list, election of a leader node, the ability for any node to send both streaming and one-shot message(s) to the leader node, and broadcast-to-all mechanisms, both one-shot and streaming. And these are all working now. At the moment, I don’t have any plans to support the remaining features like distributed semaphores, and storage (both K/V and spill-over). I can use other libraries for those should the need arises.

Finally, I’m thinking of porting hedge to both Rust and Zig as well. Looking forward to that.

Related blogs:

1. AWS ClockBound client for Go
2. AWS ClockBound client for Go (update)
3. Distributed locking on AWS (ClockBound)
4. This blog
5. Static-linked CGO binaries using musl and Zig

As far as lock services are concerned, spindle-cb (and spindle for that matter) is what you call a coarse-grained lock (as opposed to fine-grained). Most of my use cases of distributed locks are really around application-level orchestrations across multiple nodes. That means using a single lock (or two) within the duration of the application, instead of multiple locks for multiple objects/data, although that’s not a limitation of the library by any means. With that said, I don’t usually use spindle[-cb] directly, but through a cluster-aware wrapper, like, say, hedge.

At this point, I couldn’t really vouch for spindle-cb’s reliability, at least not yet. But since it’s really just spindle with a different “true time” source, I have high hopes. And having used spindle in production for many years without any issues, I’m optimistic.

Related blogs:

1. AWS ClockBound client for Go
2. AWS ClockBound client for Go (update)
3. This blog post
4. Cluster membership management on AWS
5. Static-linked CGO binaries using musl and Zig

Either way, this is unusable for me at the moment. Instead of spending time debugging this, I wrote another client, called clockbound-ffi-go, utilizing the provided FFI (meaning, requiring CGO). So far, it works well, as expected. I would have preferred to not use CGO but I need a working version as soon as possible. I’ll come back on this when I have the time.

Related blogs:

1. AWS ClockBound client for Go
2. This blog post
3. Distributed locking on AWS (ClockBound)
4. Cluster membership management on AWS
5. Static-linked CGO binaries using musl and Zig

As great a tech Google’s TrueTime is, there is no available API for it. It is only through Spanner that spindle achieves its locking mechanisms with TrueTime. Currently, it’s the cheapest way, surprisingly, to do distributed locking that I’ve tried so far, compared to the likes of Redis, Zookeeper, etcd, Consul, etc. Yes, I could do VPC peering between GCP and AWS but it is quite costly at the moment.

I’ve heard of AWS’ TimeSync Service before but only in passing. Now that they’ve also released DSQL, and having seen the papers and blogs about it, turns out that TimeSync is their equivalent to TrueTime, and that there is an API for it!

Related blogs:

1. This blog
2. AWS ClockBound client for Go (update)
3. Distributed locking on AWS (ClockBound)
4. Cluster membership management on AWS
5. Static-linked CGO binaries using musl and Zig

I remember ages ago, a friend asked me what my tips would be to transition from a typical programmer to a “systems programmer”. And I distinctly remember answering him along the lines of “Y’know what, I don’t really know, maybe just get more experience?” And now, being an engineering leader to some extent, I get this question once in a while, but even now, I don’t really have a default answer, and a good one at that, to give. And writing this blog is, in part, me trying to remedy that.

Now, a bit of context, as this topic, I think, is a little vague, and you can tackle it from different perspectives. First, what is even a “typical programmer”? And what is a “systems programmer” anyway? I will try to narrow it down to my friend’s perspective. He’s a web developer, one you would consider as “full-stack”. He likes his programming languages, frameworks, design patterns, and we talk a lot about these things, which are all very interesting. We can pin the “typical programmer” definition to that. Does it make sense if we categorize junior, associate, mid-level, senior programmers to this group as well? Maybe. But I also have peers who are seniors for many, many years, and still enjoy what they do (e.g. maintaining legacy systems), and have no intentions/interest in going beyond that; it can be argued that they are also experts in their specific domains.

I also don’t think my friend meant those developers who specifically do close-to-hardware systems, like embedded, drivers, OS, robotics, automotive, etc.; I think he refers to the IC path, such as [Senior] Staff+, [Senior] Principal Engineers, etc.

We can also phrase it controversially, like, “How do I transition from a [insert language here|backend|frontend] programmer to an [expert generalist|staff+|principal] level programmer?”. There’s a lot of nuances to unpack in that line but let’s just try to leave it like that and interpret it as you would so we can move on. I will also limit my discussions to the technical side of it; the social/political/soft-skills side of it is another topic I intend to explore in a separate blog post.

Aside from checking the literature regarding the topic, I also tried asking colleagues/peers what they think of this question. In no particular order, some interesting replies I got were:

• Read a lot of papers - I suppose it makes sense to some extent?
• Get a masters or PhD - I’ll leave this one for you to interpret. There’s a spectrum here but I really admire PhD people with really deep levels of expertise in specific sciences as well as software in general.
• Job hop (specifically between startups and big tech) - You could also do that, I suppose?
• Contribute to OSS - not sure if this works; depends on the OSS in question I suppose?
• Join an early-stage startup - if you’re [un]lucky.

Now, to the topic: my one-line answer would be the title of this blog: know your target. Let me explain.

Know your target

When I say “target”, I don’t mean customers or users of your software, but your production target platform: mobile, browsers (if doing web development), architectures such as Intel’s x86 (x64), RISC architectures such as ARM, MIPS, and PowerPC, the GPU, cloud and its internals, the network your software is running on, storage systems, data centers, mainframes, HPC systems, IoT, etc. With that said though, I don’t really expect us to “know” all of these targets; we can go a step higher. Most of these targets are abstracted by the OS/software running on top of them: e.g. Linux, Unix, Windows, OSX, Android, IOS, browsers, the JVM, hypervisors, etc. Knowing these target OS/software is much more important than the language you use to program them. Actually, the language being used will be of lesser consequence to the system artifact itself compared to the target platform it will be running on. Therefore, I can further specify “know your target” to “know your target OS”.

You might say, “Of course! Isn’t that obvious/expected?” You might be surprised how many programmers have no idea, or don’t care, about the target OS their software will be running on. The facilities provided by the language (and the libraries/frameworks around the language ecosystem) are more than enough for them. So when you start talking about processes, threads, concurrency and synchronization at the OS level, or things like IPCs, memory models, I/O, the OS’ network stack, filesystems, virtualization, user/kernel mode, protection rings, context switching, etc., you’ll get the impression that for them, these things are what solution architects and tech leads are there for.

Take Linux for example. It’s only one of the “targets” in our list but Linux is a complex beast in and of itself. A programmer can probably reasonably pick a language up in weeks/months (maybe years for C++, Rust) but he/she might not really “know” Linux even until retirement. There are so many subsystems in Linux that learning all of them is probably an impossibility. One upside though is that your knowledge of Linux will more or less translate to other OSes without the need to start from scratch.

Understanding your target allows you to leverage what it can and can’t do. Its capabilities will have a big influence on how you design your system. Of course, the language matters to some extent but you will be better off approaching it in terms of its capabilities and limitations within the confines of your target.

Your knowledge of targets will also help you better understand how complex systems are being designed: even distributed systems, or systems you’re probably using now.

And lastly, debugging. Not language syntax debugging, but those pesky, difficult-to-reproduce, SLA-violating production bugs. I don’t think I need to expound about this more.

So, interested in becoming a systems programmer? First, know your target. Your language of choice is just one of your tools to program your target.

Closing

I acknowledge that there are many opinions out there that don’t necessarily align with this advice. For instance, a quick check on publicly available literature yields topics about architectural patterns, algorithmic thinking, FIRST and SOLID principles, systems thinking, formal verifications, thinking in terms of tradeoffs (which I agree by the way), thinking in terms of costs, etc. (which are all well and good), but I think that by being deliberate about target thinking takes you a long way towards becoming a systems programmer.

I had a think about this topic when I saw a recent Ars Technica news article about Winamp releasing their source code to GitHub and then deleting it after some backlash from the open source community. And we also hear about instances of leaked source codes as well, such as that of The New York Times, Okta, Intel’s UEFI source, Mercedes Benz, Microsoft, among others. Until now, I haven’t really put too much thought into this. I mean, from a business perspective, especially software companies who earn money through code, how valuable do we think our source code really is?

Before I dig deeper though, I think the value lies within a spectrum. And there are valid arguments from both sides. Forget the threat actors’ goal of obtaining sensitive information such as keys, secrets, credit card numbers, etc. from the source code, although one could argue that these information are also part of the source code itself (and you could also argue it’s technically not), making it extremely valuable; I’m generally referring to source code as a programming artifact, the text that contains the logic, and secondarily, the documentation that supports it. From this perspective, one expression of value would be does it matter if the source code is public or not. Think of Windows and Linux operating systems. One is closed source but the later is open. Is Linux’s source code more “valuable” because it’s publicly available? Or is it the other way around?

With that said, I want to dig a little deeper on an argument that leans toward source code, as presented above, not being really that valuable; Peter Naur’s “Programming as Theory Building”. Although the “Backus-Naur Form (BNF)” author didn’t explicitly say it as so, his arguments are quite fascinating, and maybe we could learn a thing or two from them.

Peter suggests that programming should be regarded as an activity by which the programmer form an insight, or theory, of how certain affairs of the world will be handled by, or supported by, code; a kind of mapping of these affairs, both characteristics and details, into program text, and any additional documentation. This is in contrast to the more common notion that programming is the production of programmed solutions, including design and implementation, and certain other texts, such as specifications, and documentation. It emphasizes that the knowledge possessed by the programmer by virtue of his or her having the theory, essentially transcends that which is recorded in the documented products. This transcendence can permeate in at least three different areas:

• The programmer having the theory of the program can explain how the solution relates to the affairs of the world that it helps to handle;
• The programmer having the theory of the program can explain why each part of the program is what it is, or is able to support the actual program text with a justification of some sort;
• The programmer having the theory of the program is able to respond constructively to any demand for a modification of the program so as to support the affairs of the world in a new manner.

For the first point, he posits that during theory building, a large part of the world won’t have a direct mapping to the code. That invisible contextual information can only be made relevant by someone who understands the program theory and its relation to the world. What is interesting is that he further argues that the code and its documentation is insufficient as a carrier of the most essential part of any program, its theory. It’s something that could not be conceivably expressed, but is inextricably bound to human beings.

The second part talks about the technical decisions and tradeoffs made as to why a specific implementation was chosen. Other implementations would have been correct but the programmer with the theory can justify the decisions being made at that time, which comes from intuition, and experience. Here, major decisions such as architecture can be documented, but not all. It is implied that, as the first point argues, it is impossible to document all these tradeoffs, and probably won’t have much value if done anyway.

The final point, I think, is the most interesting as it involves program modification, which I think we can all agree, is expected in software. And software modification is closely related to costs. This is the reason why onboarding of new programmers take time; new programmers need to understand the original theory. It is insufficient to only be familiar with the program text and other documentation. What is beneficial, or even required, is the opportunity to work in close contact with the programmer who had the original theory. This is similar to other educational problems of other activities such as writing, or learning a musical instrument. The student doing related activities under suitable supervision and guidance will be better off compared to the student who’s only looking at manuals and written instructions without teacher supervision.

One more point he emphasized is the idea of program life, death, and revival. The initial building of the program is the building of its theory. The program remains alive when the original programmer (or team) remains in charge, and retains control of its modifications. Program death means the programmer (or team) who understands the theory is dissolved, or is not in charge anymore. A dead software can continue to provide value during its operation; the actual state of death only becomes visible when demands for modification cannot be answered and/or addressed anymore. Finally, revival simply means the rebuilding of its theory (not necessarily the original) by new programmers. This is why it is often faster to write new code than modify an existing one since writing new code means rebuilding the whole theory, but now, it will be based on the intuition and perception of the world by the new programmer, without being beholden by the original theory.

Philosophy aside, what can we learn from these arguments? Here are some proposals:

• It is a good idea to retain the original programmers for as long as possible. (Perhaps easier said than done due to a lot of factors.)
• Onboarding is slow and expensive. Try to focus on the possibility of the new members having access to the original programmers during the transition.
• In general, keeping programmers longer is more cost effective than high turnover.
• Documentation, while unable to capture the whole program theory, is still helpful in providing context, however minimal, in theory knowledge transfer.

In this post, I want to talk a little bit about technical debt. In a recent conversation I had with one of our engineers, we sort of inadvertently enumerated some of the more pressing technical debts we have off top of our heads in our current infrastructure, which, now that I think about it, is quite a lot. But what is “technical debt”, really?

Techopedia defines it as “a programming concept that reflects the extra work required when developers choose an easy short-term solution over the best long-term approach. Like financial debt, it incurs interest in the form of increased maintenance costs and complexity over time.” I think I can get behind this definition. And while there are good debts and bad debts in finance, I think technical debt is mostly associated with a feeling of frustration, or resignation. I don’t think I’ve ever come across a situation where technical debt is being talked about in light of something akin to “good debt”. It’s always “bad code”, “bad design”, “bad choice of framework”, etc. But I think this association is a little precarious. For developers, they might assume that the previous developers were just rubbish at their job without considering the constraints and tradeoffs made at that time. Product people might think that developers were just using tech debt as an excuse to extend the delivery schedule to a month just so they can play around with the stuff that they care about more than the feature they’re being asked to do. Engineering leaders might assume that their developers are spending too much time debating on patterns, and differing opinions on the correct way of doing things instead of agreeing on tradeoffs and carrying on.

Another more interesting trap with that association is that if we just write good code, then there shouldn’t be any tech debt. But in reality, even a well-engineered solution will eventually become tech debt over time, especially when the original implementers leave the company, bringing that undocumented, contextual knowledge with them. And things like changing technologies, abandoned open source solutions, software license changes, leadership changes, budget changes, even bit rot, can render an almost-perfect implementation a tech debt in no time.

One common thing I observed teams do to mitigate tech debt is to conduct tech debt-only sprints, or tech debt hackathons. While good-intentioned and practical, it sometimes encourages a culture of postponing good design upfront just to honor the initial, usually very tight schedule, instead of negotiating and meeting in the middle. And while I’ve experienced some success with this approach, refactors and maintenance work are rarely rewarded. They’re invisible, thankless, boring, and soul-crushing, especially if it’s not your own. Now, I’m speaking in general terms here; I know a bunch of people who prefer maintenance work than endless pumping of new features. And more power to them. But I think this is not uncommon. Companies rarely reward tech debt maintenance, especially when there are a lot of initiatives and new features that drives the business forward.

So how to deal with it then? Unfortunately, I don’t really have a defined answer to this. At Alphaus, I advise our engineers to include tech debt (if there’s a need to deal with it) in their estimations, and think of it as just that: part of the solution. I don’t encourage separate, isolated efforts just to tackle tech debt, although there are times when we just admit defeat, deal with it, and then move on. I like to think of it as how we deal with financial debt, really; we pay in chunks monthly until it’s fully paid. It’s just part of the monthly budget. And occasionally, you get some extra cash and so you pay a little bit more.

On a more systematic approach though, tech debt is really part and parcel of software development, at least in the current state of things. There are schedules to be met, and customer requests to be fulfilled. Whatever method(s) one employ to formalize tech debt management, which I think really depends on the company size, some points need to be considered:

• Metrics for visibility and understanding of systems;
• Ownership at the higher level;
• Alignment (and agreement) on the value to the business if a specific tech debt is addressed;
• Understanding of cost and associated risks when not addressed; and
• Metrics to measure success (or completion).

Anyway, so what do I think of Zig? I think it has a lot of promise. It still feels incomplete, which it is, especially the standard library. Which is expected of course for a < v1.0 software. But coming from C, and Go, I think Zig is the sweet spot for me between C++ and Rust. But I don’t think I will be introducing it to Alphaus anytime soon though; maybe when it’s tagged v1.0+, which I think is still years from now.

~~

There’s no escaping GenAI nowadays, is there? I’m sure you’ve seen the full spectrum of its effects by now; from total naysayers to skeptics, to cautious optimists, to proponents and fanatics, to full-blown doom-bringers. In the cloud space, the big three cloud providers are “all in” on AI, as you can see in their headlines. Furthermore, there are thousands of AI-powered startups cropping up left, right, and center, with massive venture capital and valuations. If you’re not in the thick of these things, it gives the feeling that if you don’t invest, or apply AI in your business, or products, it’s just a matter of time when you’ll be left behind and eventually relegated into the pages of failures in history. It’s a scary thought. The FUD is real.

According to Gartner’s Hype Cycle for Artificial Intelligence, 2024, AI is now past the “Peak of Inflated Expectations”, although the hype about it still continues. So somewhere around here:

That means the “Trough of Disillusionment”, or in other words, the time of it not being “cool” anymore, is just around the corner (could still be years though, who knows). That’s the phase where you’re going to get the “You’re still doing AI? The world has moved on to Quantum Computing now! Catch up, or you’re gonna be left behind!” sort of feedback.

Remember Web3? According to Crunchbase, the uncontrolled hype of 2021 petered out dramatically around last year, with funding declining from $28 billion to $7.6 billion (73% decline) in 2023. $7.6 billion is not a slouch by any means, but you’d better have a credible business model to be able to survive that. With AI, you can kind of see signs of brakes being applied already with reports saying it might not perform well enough to justify the cost.

So what does that mean to us, system builders? I’m not an economist, nor can I predict the future, but I can form my opinion around what’s going on. Mind you, this is just my opinion; I’m not even representing the opinion of my employer, Alphaus. With that disclaimer out of the way, I think I’m somewhere in the middle; cautiously optimistic about AI and its uses. AI has been compared to the internet revolution in the 90’s, and rightly so, as a lot of the intricacies of AI still feels like “magic” to most people. And I think I can understand this sentiment as I’m in the generation that saw the rise of the internet and social media during their prime years and a lot of it felt like magic as well. Regardless of the dot-com crash in the 2000’s, now, the internet, just like the smartphone, is here to stay. We don’t really think about them that much anymore; they’re just, you know, part of our daily lives. And I think AI will be the same. It will find its place, whether niche or mainstream, in our daily lives in some form or another. Whether that’s in my lifetime, I don’t know.

We use AI at Alphaus. We use forecasting models to help our customers forecast budgets and their future cloud spendings based on their historical data. We use LLMs as part of our customer support. Our engineers use Copilot to aid them in their day-to-day coding tasks. I use both ChatGPT and Gemini in notetaking, summarization, and translation. And I know our CEO, Hajime Hirose, uses a lot of LLMs as well. On top of that, I read a lot of papers behind AI. It’s something I enjoy doing. As a CTO of a startup, it is part of my job to assess technology and leverage it to further our business. And that includes AI, among other things.

If you’re a software engineer who is currently employed, or looking for a job, or looking for something interesting to learn/build, or a technologist who is thinking of starting a company, I think it’s wise to consider AI. Whether we like it or not, we are part of this corporate, capitalistic world, and AI is the current game. And we have to play the game to survive.

Do I think AI will replace developers? To some extent, yes. And not just developers, but other professions as well. But I think it will also create new jobs and positions at the same time. Just like the internet and other transformative tech revolutions before that. I would imagine the industrial revolution has replaced a lot of professions but also created completely new ones. Think assembly lines, steel-making, steam-power-based automation and mass production, etc., which I think didn’t really exist before. Anyway, think of it this way: software engineering is about solving problems, not just coding. Coding is probably the easiest part of our job. LLMs might be able to replace that bit but, without sentience or consciousness, the social/human part of our job is still on us.

Do I think GenAI will cause human extinction? Or replace us humans? Well, it’s like asking a hunter-gatherer during the Stone Age if he thinks a jet engine would render his bow and arrow useless. He’ll probably just reply, “You mad? What are you on about?”.

This post will not be an X vs Y review, or at least, not intentionally. Programming languages, I think, are one of those things that developers tend to get attached to, alongside editors, OS’es (or distros in the Linux world). “Here be dragons.”, as they say. But I want to share my thoughts from a perspective of someone who can introduce a new programming language to teams, instead of someone who wants to make a case and convince somebody who can do so.

Before I joined, Alphaus was a PHP shop. Around early 2018, after about half a year, I introduced Go as our main programming language but only for new services; no rewrites. I didn’t know Go at that time, nobody in the company did. But why Go though? I could have chosen Java, or C#. Seven years on, now, I consider that a good decision. Did I know it would be? Of course not. Java probably would have been fine. But thinking about it now, I would attribute it mainly down to Go’s simplicity.

Before Alphaus, my background was embedded, low-level systems. Over the years of working with several programming languages, I’ve come to appreciate the simplicity of C. Even though experience-wise, I probably have written more C++ than C, there’s something in C’s simplicity that I yearn when working with more complex languages. And I think that translated to my choice of Go, really. It was the C equivalent in the high-level language world for me (apart from the fact that it’s created by C people). So you can say there’s definitely a bias there on my part. And I think it’s true. We always say (in software engineering circles) “use the right tool for the job”, but the “right” part there is the tricky bit. In my case, “right” was ultimately based on my own biases, my own preferences. Although in hindsight, it wasn’t really purely selfish. There was a big push for Go when going cloud native at that time as well. And Docker and Kubernetes were also written in Go, which I think also had an influence, as we based our stack on these tools too. And our product was in infrastructure (we pivoted later) so there’s definitely an alignment there.

Go’s simplicity, in my mind, also meant easy recruitment. I said nobody in the company knew Go at that time but it wasn’t really a big deal to me as I was quite sure the engineering team would pick it up and be productive with it in no time. The same with recruitment; I can hire non-Go engineers knowing that they can learn Go in about a week and then start contributing. With that said though, simplicity being translated to quick productivity is not really a deal-breaker to me. It only applies when you don’t have people who know the language. When you have a team who are already proficient with, say, C++, they will be productive with it.

You see, I could go on enumerating about the pros of Go by mentioning its merits as a language but what I’m really trying to say here is that the “objective” (or “right”) reasons why I should choose Go only had a slight influence on me. Even without all these, I’m sure I’d still end up with Go mainly because of my bias in C. I’m sure there are a lot of engineering leaders out there who are more objective or data-driven than me but again, programming languages are one of those things we get emotional about. And I’m sure I’m not alone in thinking this way.

Now back to the present. At the moment, I’m mulling about introducing a systems language to the company. Not a replacement of Go, but a compliment to Go. Of all the choices listed above, and based on the thought process I just laid out, it’s obvious that my choice would be Zig. It’s advertised as “the better C”, while Rust is “the better C++”. But I’m not quite sure yet.

You could argue that it’s impractical, maybe borderline irresponsible, of me to not choose Rust. It should be the “right” choice. It’s memory-safe, more mature, already included in the Linux kernel, proven in production by some of the big names in tech, etc. And more importantly, some of the people I follow (and look up to) are also advocating for it.

I know Rust a bit. I’ve dabbled with it outside work although I’m not really proficient with it since I haven’t written any production code with it. It is a complex language. Not C++-level kind of complex but it is still very complex. I believe it is “the better C++” and if I’m still working in C++ now, I’d choose it as well. But I’m not. And my bias is stronger now than ever before due to C, and now, Go.

I’m actually looking at Zig now. It resonates with me as it still feels “C”-ish but a lot better. I like the choice of control over memory allocations. It’s probably not as safe as Rust but it provides mechanisms to make it much safer than C. And it’s still a simple language overall. Having said that, no decisions yet. It’s only been 2-3 months. And I’m not in a hurry. I think the only real blocker at the moment is it not being tagged 1.x yet.

This post has been all over the place. In the end, you might not care whatever my choice will be and the justifications I have for it but another way of looking at it is that if you happen to be in a position of trying to convince your leader(s) to switch to your preferred programming language, basing your arguments only from an objective point of view might not hold much water. You’re probably going to fight an already losing battle from the start.

Or it could still work, you know. Who knows.

These pod resource limits are usually enough 80% of the time. However, since late last year, some of the accounts have datasets that are way, way beyond these limits causing persistent OOMKilled events.

Our first stop-gap solution was to increase the memory limit. The trouble was, even with 20GB+ of memory wasn’t really enough for some of the input datasets. And on top of this, GKE’s cluster autoscaler also started increasing the VM sizes to those which we don’t have CUDs for. Suffice it to say, it increased our monthly cloud spend to about +20% while delaying the overall processing time due to pods crashing (and restarting).

We tried other solutions. One was using local files which required increasing the size of the attached storage. While cost-effective, the performance drop was significant mainly because most of the datasets that were well within the memory limit were now moved to disk as well. We also tried using the database we are currently using which turned out to be worse in terms of perfomance and costs. We also tried to use our cache layer (named Jupiter) which was very performant but prohibitively expensive.

Enter Spill-over Store (SoS), our current solution. Inspired by Apache Ignite’s design, the idea is to stitch together the already available memory and storage across the running pods, providing an ad-hoc, on-demand storage for really big datasets.

From the image above, the pod that is assigned to load a huge dataset will exhaust its local memory first, then “spill over” to its local disk, then another pod’s memory (using gRPC streaming), then disk, and so on. Thus, our example 100GB dataset will utilize around 4 pods in total within the cluster.

This solution allowed us to actually revert back to our original pod resource limits. Both disk and network performance are acceptable (we don’t use GCP’s Tier 1 network) and still within our SLA as the solution only applies to about 20% of the ingestion pipeline. The majority still uses local in-memory processing.

As Alphaus grows (and therefore ingests more and more data) and serve more clients, maybe we will eventually end up using Apache Ignite or some other off-the-shelf distributed solutions, but as of now, SoS works. With that said, if you have a cost-effective (and better) product/solution in mind, please feel free to contact us. We’d love to talk.

Finally, you can find SoS’s implementation here (if you’re interested).

The idea of 50Gbps, 100Gbps, or even 200Gbps network speeds, and now with P4d’s advertised 400Gbps (yes, 400!) somehow didn’t translate to my internal insistence that SSDs, especially the NVMEs or even M.2s, should be faster. Now, there might be enterprise/military/space/etc…-grade SSDs that I’m not aware of, or don’t have access to, that have stupendous read speeds, but I’ve never heard of one so let’s leave them for now. Some of the faster M.2 SSDs can go up to ~15GB/s read speeds, which is quite impressive, but nowhere near as P4d’s. I still can’t wrap my head around it. It’s the sort of numbers you find listed on supercomputers with their custom Infiniband interconnects.

Anyway, after scouring the rabbit hole that is Reddit regarding discussions, debates, (and fights,) about the correctness of these numbers, I came across sirupsen’s napkin-math repo, which I think is a fascinating piece of information. If you’re a system designer, you really should check it out.

Again, it looks like network can indeed be faster than disk access. I really need to recalibrate my internal latency numbers table to accommodate for these more modern hardware capabilities. The good thing is, at least, these sort of information are now readily available anytime. With that said though, I still believe that system designers should still be able to roughly determine a system’s overall performance behavior pre-implementation using napkin-math latency calculations.

The migration was actually quite quick. I only copied my previous _posts/ folder into the new content/blog/ folder and added some of the needed front matter which was simply done using good ol’ grep + awk.

While I really liked the look and feel of my old site, this new one is also quite good. And Netlify’s support for Hugo is quite solid as well so less build headaches for me. Hopefully, for many years to come.

Thank you OSS and free services.

You’ve probably heard of the warning “Don’t ship the org chart” common among product development circles. I always thought of this as synonymous to Conway’s Law which states that organizations design systems that mirror their own communication structure. In my experience, I’ve come to believe that this is true. Whether you like it or not, it is an eventuality. At some point I thought that to be an effective solutions architect, you have to be an “org architect”. Or in order to achieve a certain system architecture, you’re better off rearranging your org structure than influence cross-functional architects or senior engineering leadership to agree with you. It may work for a time, especially during the start, but eventually, as engineers come and go, the org-mirrored design will out. But I think there’s more nuance to this than what’s obvious.

There are several types of org structures but I will cover only what’s relevant to my situation which is startups. This might be anecdotal but between peers, the most common types fall under these two broad classifications: less boundaries or bigger teams, and smaller, often siloed teams. Teams with less boundaries tend to move slower because more people need to communicate for alignment but will produce a more coherent product. Smaller teams on the other hand, move faster but their outputs are more difficult to integrate together, resulting in an inconsistent product experience. Personally, I don’t really have a strong opinion on which is better. I think there are phases in a startup where one works better than the other. But this requires you to understand what you want to build in the first place and then build your org chart around that.

There’s also the case of building your org structure based on product lines if you have multiple products as opposed to functional teams covering multiple products. Product-based org structures might promote autonomy and quicker response to industry changes but could easily lead to system duplication (wasted resources). Functional structure however, encourages specialization and focus but inhibits multi-boundary communications.

You may argue that, “Does it really matter? Customers don’t really care about how your org is structured as long as your product is usable and has great UX.” I would actually agree that this is a good baseline for structuring your org. You will have several vertical teams per product but an overlapping horizontal team that ensures UX coherence, usability, and branding. I think this is what the Mirroring Hypothesis study (confirms Conway’s Law) refers to as partial mirroring; in which technological knowledge are invested, shared, and acquired beyond operational boundaries. “API-first” companies come to mind as in partial mirroring, building contractual relationships (APIs) that support technical interdependency across boundaries seems to work. And more often than not, your org structure will undergo changes multiple times during your product’s lifetime, which will result into subsequent changes to your system’s design that will then mirror the new org, and so on and so forth, so you might as well embrace this dynamism instead of fighting against it.

Finally, the study also highlights collaboration patterns in the open source sphere that do not support the mirroring hypothesis (and thus, Conway’s Law). My experience with OSS collaboration is mostly outside work so I’m interested to know if there is a case for adopting OSS-style structure within the organization. I think that’s a good topic for another day.

To conclude, instead of going with the warning “don’t ship the org chart”, since you will ship your org chart anyway, be aware of it, understand it, and make sure it works on your favor.

Retries refer to the act of retrying a failed request. When a request fails, the client can retry the request, hoping that it will succeed the next time around. However, simply retrying the request immediately after a failure can be problematic. If the failure was caused by a temporary network issue, for example, retrying immediately will likely result in another failure. This is where backoff comes in.

Backoff refers to the practice of waiting a certain amount of time before retrying a failed request. The idea is to wait long enough for any temporary issues to resolve themselves before retrying. The amount of time to wait is typically increased with each retry, hence the term “backoff.” The idea is that if the request fails multiple times, the client will eventually back off enough to give the system a chance to recover.

There are several benefits to using retries with backoff in a distributed system. First, it can help reduce the impact of temporary failures. By waiting before retrying a request, the client can avoid bombarding the target with requests, which can exacerbate the problem. Second, it can help improve overall system availability. By retrying failed requests, the client can work around transient issues that might otherwise cause the entire system to fail.

There are several strategies for implementing retries with backoff. One common approach is exponential backoff, where the client waits an increasing amount of time between each retry. Another approach is jittered backoff, where the client adds a random amount of time to the wait period to avoid the so-called “thundering herd” problem, where multiple clients all retry at the same time.

Example 1:

import (
    ...
    backoffv4 "github.com/cenkalti/backoff/v4"
)

func main() {
    var n int
    operation := func() error {
        n++
        log.Printf("n=%v\n", n)
        if n >= 10 {
            return nil
        }

        return fmt.Errorf("backoff")
    }

    err := backoffv4.Retry(operation, backoffv4.NewExponentialBackOff())
    if err != nil {
        log.Println("final backoff failed")
    }
}

Example 2 (my preference):

import (
    ...
    gaxv2 "github.com/googleapis/gax-go/v2"
)

func main() {
    bo := gaxv2.Backoff{
        Initial: time.Second,
        Max:     time.Minute,
    }

    var n int
    operation := func() error {
        n++
        log.Printf("cnt=%v\n", n)
        if n >= 10 {
            return nil
        }

        return fmt.Errorf("backoff")
    }

    for {
        err := operation()
        if err != nil {
            time.Sleep(bo.Pause())
            continue
        }

        break
    }
}

In conclusion, retries with backoff are an important technique for improving the robustness and availability of distributed systems. By waiting before retrying failed requests, the client can help reduce the impact of temporary failures and improve overall system availability. There are several strategies for implementing retries with backoff, and choosing the right approach will depend on the specific requirements of the system.

Additional reading: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/

I just posted a blog about gRPC here. If gRPC and grpc-gateway is right up your alley, you might find it interesting.

After about a month of testing, I think it didn’t really turn out quite well in the end. It is stable enough for deployments that are not spike-y in terms of workloads (frequent scaling up/down). Or if I set min = max in the HorizontalPodAutoscaler. In these cases, memberlist can consistently track members just fine. What’s better is that it works even in multiple deployments in the same namespace which I thought was brilliant. For example, if I have a deployment app1 set to 10 pods in the default namespace using memberlist’s default port numbers, and then I deploy another set, say, app2, within the same namespace using the same ports, app1’s memberlist can track its 10 member pods just fine while app2’s memberlist is also separated. But when applied to my use case, which has a minimum pod of 2 and a max of 150, with frequent scale up/down frequency depending on load, it can’t seem to keep up. The potential for Byzantine faults is just too high: i.e. in a 50-pod scale, memberlist can end up having 2 groups of m-pods and n-pods where m+n=50. Very rarely, it can even go up to 3 groups.

I am a little frustrated. I really wanted it to work; I even attempted to update memberlist to incorporate hedge’s logic but it was too much for now, with my schedule. So now, back to the old one. By the way, the current logic is fairly rudimentary: all members in the cluster/group send a liveness heartbeat to the leader and the leader broadcasting the final list of members to all via hedge’s broadcast mechanism. CPU usage between the two is fairly similar depending on the sync timeout.

I’ve been trying to improve hedge’s member tracking system as I want to build a distributed in-memory cache within hedge itself. Most of the available ones are Raft-based, and I still haven’t figured out how to make Raft work in the same deployment configuration.