CTO Diaries #3: On choosing technology stacks

2023-04-01

When it comes to tech stack selection, in general, I always approach it in terms of tradeoffs. And when it comes to understanding tradeoffs; what you gain vs what you lose, I think experience plays a big part. In my 18+ years of experience building systems, I’ve been bitten enough times that I view most of them with a fair bit of cynicism. I wouldn’t really say that experience definitely makes me better at it; I still have my own biases and experience doesn’t really ‘fix’ the illogical side of me being a human being with emotions. It gets in the way at times. Then there’s the business side of it as well. An example would be to maintain business relations, or to honor partnership agreements, I would need to use (or integrate into) some tech stacks that I wouldn’t normally choose in the first place. And to add to all of these, I also work in a startup with limited resources so there are other criteria that have a bigger influence on my approach to balancing tradeoffs. And at our phase right now, probably the biggest one is cost. Not just monetary but engineering costs as well: cost of operating and managing these systems. And I must admit that cost considerations is at times so pervasive to me that I would choose a technology purely because I got a significant discount for it even though at the back of my head, I know it’s going to come back to me later on. So much for experience, huh.

The other criteria would be expertise. With our current budget, there is a certain ceiling to the level of engineering expertise I can afford to hire. And so far, ICs or principal level engineers who could handle such decisions for me are still out of my reach. So I still end up making the decisions most of the time. The one or two engineers I have who are fairly knowledgeable about systems and infrastructure are also doing development and technical support at the same time that there’s just not enough time to do validations. They will have opinions of course based on their readings and experience but at the end of the day, I know that they didn’t have enough time to validate so I roughly know the level of confidence they have on their opinions. And speaking of opinions, I am more interested in roughly knowing about the caveats, horror stories, real user feedback, etc., that means checking Reddit, Mastodon, Twitter to some extent, engineering blog posts, GitHub issues, HackerNews, etc. I would say that this is almost always a requirement of mine to myself as part of my due diligence.

One thing that I enjoy in all of these however is the part where I do a small prototype of the tech in question. It’s not all the time but when I do, it’s always a good feeling. It gives me a sense of what the tech is about, what it can do, how it works (sort of), etc. For the core parts of our systems, I think this is pretty much already an imposed personal requirement.

With all these requirements in mind, one might say that I will just end up choosing boring, mature, proven, old school tech all the way. For the most part, yes, but I’m also aware that tech is also part of the hiring strategy. New tech being exciting for most engineers is probably a safe assumption to make so there is also that consideration. With that said though, I’m very careful with this criteria. Over the years, I’ve come to accept that building systems is engineering/programming over time. Engineers come and go. And tech is evolving at a rapid pace. And so does expertise and opinions evolve with it. Systems are dynamic; they are reshaped and redesigned by the people working on them over time so you will eventually end up with an amalgamation of all sorts of things, even beliefs and convictions. And when all is said and done, the systems running in the company is ultimately my responsibility; I can’t just put the blame on somebody when things go wrong just because it was their choice and not mine. This leads me to another thing in my criteria: migration. For long-running systems, migrations are a matter of when, not if. I will try to imagine how it’s going to look like, how painful it will be, and roughly how expensive it will be when the time comes. Then balance it out with us being a startup where the possibility of throwing away code, tech, product(s) even, at any time, along with the engineering behind it, is pretty high.

Expanding on the migration bit, this is probably my first time dealing with systems with colossal amounts of data so the considerations for migration is significantly important. This covers our choices of data lakes, high traffic messaging systems, databases, and migrations between these systems. Based on experience, migrations of data-related systems or systems revolving around data are the most protracted and the most painful. Much more so than, say, compute.

What about hype, or in GitHub for example, the number of stars a project have? Or in Twitter/Reddit, the chatter? In a way, they are an important consideration as well. Hype results to more early users, which means more feedback, which means more bugs reported, which means more bug fixes. The hype bit though; I’ve been through several hype cycles of tech already that in a way, I can filter them out easily and see through the marketing PR. And I think I’m not really alone in this. Most of the colleagues I talk to with long-ish experiences under their belt, it’s true to them as well.

So, how do I keep track with all of these information around criteria? Journaling. I take notes on all of them. I’m still working on the organization part of it though. It was and is a need so I’m sort of forced into it. But overtime, I think I’ve improved a lot on my documentations (and writing them) and it has grown on me so I don’t mind it now.

Alright, that’s it for now. See you in the next one.