2024-08-25
TL;DR: To my fellow system builders, don’t dismiss it but understand how it works. As the saying goes, “a tool is only as good as the hands that wield it”. As a software craftsman, your tools are important. And equally so, are your skills in using them effectively. ~~ There’s no escaping GenAI nowadays, is there? I’m sure you’ve seen the full spectrum of its effects by now; from total naysayers to skeptics, to cautious optimists, to proponents and fanatics, to full-blown doom-bringers.
5 minutes
Thoughts on the newer systems programming languages
2024-08-20
I’m talking about the new crop of systems programming languages that advertise themselves as better replacements for C and/or C++: Rust, Zig, D, Odin, Nim , etc. I’m using the word “new” loosely here as Rust and Zig, for example, are almost a decade old now. The topic of systems programming has been on my radar (again) recently at work due to our attempts at improving the performance of some of the more critical parts of our stack.
Programming · Programming · Systems · Zig
6 minutes
How Alphaus saves on costs by ‘stitching storage’
2024-07-24
One of Alphaus’ data processing pipelines ingests around 10TB of client financial data per day. The processing engine is running on GKE with around 80-100 (depending on what week of the month) pods sharing the total workload. Each pod has around 10GB of memory and 30GB of attached storage. The consistency of this load allowed us to purchase enough Committed Use Discounts (CUDs) for the underlying VMs to save on compute costs.
Infra · Infra · System-Design
3 minutes
2024-06-28
“Back-of-the-envelop calculations”, “napkin-math”, “latency numbers every programmer should know” - yes, those numbers that usually come up during system design interview questions. This came into my periphery again while looking at RDMA latency checks and benchmarks with P4d instances in AWS (using SoftRoCE). As an old-timer with (most likely) outdated ideas about system design-related latency numbers, although I’m quite familiar with Jeff Dean’s “Numbers every one should know” approximations, I noticed that in a jiffy, I’m still (unconsciously) subscribed to the idea that disk access is most definitely faster than network.
Infra · Infra · Latency · System-Design
3 minutes
2024-06-27
It was simply because the Jekyll theme I was using before, a modified version of much-worse-jekyll-theme, doesn’t build anymore. Actually, no, that’s not quite right. I updated some of its npm dependencies which caused it to not build anymore. It’s using quite an old version of Ruby, including most of its dependencies, that I’ve been receiving a lot of warning emails about vulnerabilities from GitHub. Instead of potentially spending a lot of time just updating the build itself, I’d probably be better off moving to a ready-made, more modern templates/themes.
1 minute
memx - Get process’ memory usage (Linux)
2024-05-30
I shared a simple piece of code for getting a process’ memory usage in Linux. It’s called memx. It’s Linux-specific only as it reads the proportional set size (PSS) data from either /proc/{pid}/smaps_rollup (if present) or /proc/{pid}/smaps file. I’ve used this piece of code many times at work. We use memory-mapped files extensively in some of our services and this is how we get more accurate results. Very useful in debugging OOMKilled events in k8s.
Golang · Linux · Memory · Process · Tech · Usage
1 minute
2024-05-13
I posted a blog introducing the DIVS model, the process we use at Alphaus, the startup I work for. Check it out here.
Agile · Development · Process · Startup · Tech
1 minute
oomkill-watch - A tool to watch OOMKilled events in k8s
2024-05-03
I recently uploaded a tool to GitHub that wraps the kubectl get events -w command for watching OOMKilled events in Kubernetes. It’s called oomkill-watch. You can check out the code here. You might find this useful.
Events · Golang · K8s · Kubectl · Oomkilled · Tech · Watch
1 minute
CTO Diaries #4: On not shipping your org chart
2023-08-09
You’ve probably heard of the warning “Don’t ship the org chart” common among product development circles. I always thought of this as synonymous to Conway’s Law which states that organizations design systems that mirror their own communication structure. In my experience, I’ve come to believe that this is true. Whether you like it or not, it is an eventuality. At some point I thought that to be an effective solutions architect, you have to be an “org architect”.
Conways-Law · Cto · Cto-Diaries · Diaries · Tech
3 minutes
Announcing our new product, OCTO
2023-05-29
We just recently announced the public beta of our new product, OCTO. If you’re interested, you can join our waiting list at https://lp.alphaus.cloud/waitlist.
Alphaus · Octo · Product · Release · Tech
1 minute
Retries with backoff in distributed systems
2023-05-11
In a distributed system, where multiple processes communicate with each other over a network, failures are inevitable. Network partitions, hardware failures, and software bugs can all cause a request to fail. Retries with backoff are a critical technique to help mitigate these failures. Retries refer to the act of retrying a failed request. When a request fails, the client can retry the request, hoping that it will succeed the next time around.
Backoff · Distributed-Systems · Retries · Retry · Tech
3 minutes
2023-05-03
Hey there, I just posted a blog about gRPC here. If gRPC and grpc-gateway is right up your alley, you might find it interesting.
Alphaus · Blog · Engineering · Tech · Technical
1 minute
Attempt to replace hedge’s member tracking with hashicorp/memberlist
2023-04-28
I recently came across the hashicorp/memberlist library while browsing GitHub and I thought it would be a good replacement for hedge’s internal member tracking logic. It seems to be widely used (thus more battle-tested) as well. I was quite excited as I always thought that hedge’s equivalent logic is too barebones and untested outside of our use cases. It works just fine for its current intended purpose but I’ve been hesitating to build on top of it until I can really say that it’s stable enough.
Distributed-Computing · Hedge · K8s · Kubernetes · Memberlist · Tech
2 minutes
Alphaus engineering blog launched
2023-04-20
Hey there, I wanted to let you know that Alphaus, the startup I work for, has recently launched a new engineering blog. If you’re interested in checking it out, please go to https://labs.alphaus.cloud/blog/. We will be posting updates about our products and sharing some insights and experiences from a technical point of view. So if that sounds like something you’d be interested in, feel free to take a look. Thanks for your support!
Alphaus · Blog · Engineering · Tech · Technical
1 minute
Using Homebrew for distributing Go apps (part 2)
2023-04-13
For personal reference: This is a followup on a previous post that is a bit more manual. This is a little bit easier. a) Create your own Homebrew tap Create a new GitHub public repository with a prefix homebrew-, i.e. homebrew-tap. This will host all the apps that you want to distribute via your tap. Users will install your apps using the following commands: No need to include the 'homebrew-' prefix $ brew tap flowerinthenight/tap $ brew install <toolname> # Or one-liner $ brew install flowerinthenight/tap/<toolname> b) Let’s use Github Actions to setup goreleaser First, add a .
Go · Golang · Goreleaser · Homebrew · Ruby · Tech
1 minute
Restore default branch from tag while preserving history
2023-04-06
Fore self reference: To restore default branch from a tag while preserving history, do: $ git checkout tags/v1.2.3 -b v1.2.3 $ git diff main > /tmp/diff.patch $ git checkout main $ cat /tmp/diff.patch | git apply $ git commit -am "Rolled back to v1.2.3" $ git push origin main
1 minute
CTO Diaries #3: On choosing technology stacks
2023-04-01
When it comes to tech stack selection, in general, I always approach it in terms of tradeoffs. And when it comes to understanding tradeoffs; what you gain vs what you lose, I think experience plays a big part. In my 18+ years of experience building systems, I’ve been bitten enough times that I view most of them with a fair bit of cynicism. I wouldn’t really say that experience definitely makes me better at it; I still have my own biases and experience doesn’t really ‘fix’ the illogical side of me being a human being with emotions.
Cto · Cto-Diaries · Diaries · Startup · Tech
5 minutes
Authenticating Go AWS SDK v2 using external id
2023-03-01
For self reference: Sample code as to how to authenticate aws-sdk-go-v2 using external ids:
Assume · Aws · External-Id · Golang · Roles · Sdk · Tech · V2
1 minute
Authenticating Go AWS SDK using external id
2022-10-17
For self reference: Sample code as to how to authenticate aws-sdk-go using external ids:
Assume · Aws · External-Id · Golang · Roles · Sdk · Tech
1 minute
2022-09-27
Hi. It’s been a year since my last「CTO diaries」post. Never mind the excuses, there’s a lot. Anyway, so what does my typical, or normal day, look like? Well, I usually start the day with checking the overall health of our systems. We don’t really have very sophisticated monitoring and alerting systems at this point so this means checking on several areas such as our Kubernetes clusters, databases, and critical services. On top of my head, I think the most common service crashes I noticed so far are OOM-related.
Cto · Cto-Diaries · Diaries · Startup
3 minutes
hedge - A simple distributed computing library
2022-09-21
This library has been in our production for about a year already and is one of the critical components in our backend. We mainly use it for app-level orchestration between pods. It’s called hedge and you can find the code here. Maybe it will be useful to anybody out there.
Distributed-Computing · Gcp · Golang · Spanner · Spindle
1 minute
Download the latest Github release using command line
2022-01-17
For personal reference: Download the latest GitHub Releases asset using common command line tools: Update the url accordingly. The uname | awk subcmds will output 'linux'|'darwin'. $ curl -s https://api.github.com/repos/alphauslabs/bluectl/releases/latest | \ jq -r ".assets[] | select(.name | contains("$(uname -s | awk '{print tolower($0)}')")) | .browser_download_url" | \ wget -i -
Cmdline · Download · Github · Release
1 minute
Support the ‘Any’ protobuf type from an external grpc-gateway process
2021-12-31
This is just a quick one. I had some trouble making the Any protobuf type work when running grpc-gateway from a separate process with the gateway throwing an unknown message type error. A quick fix for this is to import the generated pb.go file to your proxy source. Something like: package main import ( … _ "github.com/username/pkgwithpbgo" )
Any · Grpc-Gateway · Protobuf
1 minute
Setting alarm from commandline
2021-11-08
This is just a quick one. Although I use Gnome Clocks occassionally, most of the time I set an alarm via the commandline. OS is POP_OS!, using VLC: $ sleep 10m && \ cvlc ~/alarm.mp3 –play-and-exit && \ notify-send 'alarm done'
Alarm · Cmdline · Commandline · Gnome · Linux
1 minute
2021-09-28
So far, I’ve been posting some techy stuff here and there but being a CTO of a relatively small, Japan-based startup for about three years now, I thought I’d share some of my experiences along the way. But first off, a little bit of context: the company I work for, Alphaus Cloud is fairly small; about 10 engineers scattered across three countries, although at one point, we hit at around 35.
Cto · Cto-Diaries · Diaries · Startup
3 minutes
Extract gRPC-generated functions to a list
2021-07-31
This might be hacky and there might be a proper way to do this but recently, I needed to generate all the functions’ gRPC-generated full names from our protobuf definitions. This is part of our RBAC module that needs to filter gRPC function calls. I got the list from our generated Go client using the following command(s): Main command: $ grep -o -R -i -E '"/blueapi..*"' . | awk -F':' '{gsub(/"/, "", $2); print "-", substr($2, 2);}' | sort | uniq # Actual commands; save as yaml: $ echo "functions:" > /tmp/funcs.
Cmdline · Golang · Grpc · Grpc-Gateway
1 minute
A hacky way to update problematic Go modules
2021-06-30
The only options for updating modules that I’m aware so far are a) go get -u all, and b) specific modules, i.e. go get -u domain.com/module[@v1.2.3]. For problematic ones, my only option is b), which is a bit time consuming. There must be some other way out there that I’m not aware of but at the moment, I use this simple command: $ cat go.mod | grep -i 'github' | grep -i -v 'indirect' | awk '{print $1}' > update; \ while read -r v; do go get -u $v; done < update; rm update
1 minute
Switching to different Go versions
2021-05-31
I use this handy little script to switch between different Go (golang) versions: #!/bin/bash $1 version &>/dev/null if [ $? -eq 0 ]; then ln -sf $GOPATH/bin/$1 $HOME/.local/bin/go go version exit 0 fi go get golang.org/dl/$1 && $1 download ln -sf $GOPATH/bin/$1 $HOME/.local/bin/go go version Saving this as an executable script goset, I could now switch to different versions like so: $ goset go1.16.4 go version go1.16.4 linux/amd64
1 minute
Authenticating Go AWS SDK v2 using assume roles
2021-04-30
For self reference: To authenticate the Golang AWS SDK v2 using assume roles, refer to the following code snippets:
Assume · Aws · Golang · Roles · Sdk · V2
1 minute
Updated config for Minimalis conky
2021-03-27
I am a fan of the Minimalis conky config for quite some time now. I’ve tweaked it a little bit due to the original config not displaying some of the information. Mind you, my system is Linux-based (Pop!_OS) using wired connection only, no WIFI, so I’ve removed the WIFI section and replaced it with the wired one. Here’s a screenshot of it from my desktop. The full configuration can be found here, in case you’re interested.
1 minute
Update Spacemacs from command line
2021-02-21
Reference: Ever since I’ve started using Spacemacs on Linux, I’ve been using this simple script as a command alias to update Spacemacs using the command line. alias upe='git -C ~/.emacs.d/ pull && \ emacs –batch -l ~/.emacs.d/init.el \ –eval="(configuration-layer/update-packages t)" 2>&1 | \ tee /tmp/emacs-update && \ grep -i -E "Found.*to.update." /tmp/emacs-update && \ emacs' I use the develop branch of Spacemacs, thus the update using git. Then it will check if there are package updates and if so, it will relaunch Spacemacs to actually install the updates.
1 minute
2021-01-30
It’s been a year since I’ve transitioned, sort of, from Vim to Emacs, so I thought I’d write something about it. Vim has been my primary editor of choice for many years. I think I started Vim around 2003 on a RedHat box writing x86 assembly. Ever since, throughout my working carrier, whether it’s on Windows, OSX, or Linux, it’s been always Vim for me. Yes, even on Windows where the experience is really not that good, I still use Vim.
3 minutes
A Spanner-based distributed locking library
2020-12-26
I uploaded a yet another distributed locking library, this time, based on Cloud Spanner and TrueTime technology. It’s called spindle and you can find the code here. Maybe it will be useful to anybody out there.
Distributed-Lock · Distributed-Locking · Gcp · Golang · Spanner
1 minute
An automation-friendly, highly-scalable, and scriptable testing tool
2020-10-31
I uploaded an automation-friendly, highly-scalable, and scriptable API/generic testing tool built to run on Kubernetes. It’s called oops and you can find the code here. Maybe it will be useful to anybody out there.
Api · K8s · Kubernetes · Testing
1 minute
Automate Spanner backup using Kubernetes CronJob
2020-09-11
As of this writing, GCP doesn’t have an option to create Spanner backups automatically. This could be available when you’re reading this in the future. At the moment, however, if you’re using Kubernetes, you can utilize CronJob to do a scheduled backup. Here’s a sample CronJob deployment that uses gcloud to create the backups. First, you need to create a service account that has permissions to create Spanner backups. Once you have downloaded the JSON file for the service account, store it as a Kubernetes Secret.
Backup · Cronjob · Gcp · K8s · Spanner
2 minutes
A tool for synching DynamoDB tables between accounts
2020-08-31
I uploaded a tool that can sync DynamoDB tables between accounts. It’s called dysync and you can find the code here. Maybe it will be useful to anybody out there.
Aws · Dynamodb · Golang · Sync
1 minute
dlock - package for distributed locks
2020-07-08
I’ve uploaded a package for distributed locks. It’s called dlock. At the moment, it supports using Kubernetes’ LeaseLock object and Redis. You might find it useful.
Distributed-Locks · K8s · K8slock · Leaselock · Redis · Redislock
1 minute
Simple string manipulations (splits/replaces) with awk
2020-06-17
For personal reference: Lately, I’ve been doing loads of CSV processing on the terminal with lots of string replacing/splitting. There are probably better (shorter) ways to do this but I’ve been leaning on awk a lot recently. Some examples. One-line separator replace: $ echo "1,2,3,4,5" | awk '{gsub(/,/," ");print}' 1 2 3 4 5 # Then I can use it with something like: for v in $(echo "1,2,3,4,5" | awk '{gsub(/,/," ");print}'); do echo "val=$v"; done val=1 val=2 val=3 val=4 val=5 # Another separator: $ echo "1|2|3|4|5" | awk '{gsub(/|/,",");print}' 1,2,3,4,5 # Join multiple lines with a separator.
Awk · Commandline · Split · String
1 minute
2020-05-13
I uploaded a tool that can query DynamoDB tables. It’s called lsdy and you can find the code here. Maybe it will be useful to anybody out there.
Aws · Dynamodb · Golang · Query
1 minute
2020-04-22
Most of the time when I open a man page for a certain command or tool, I use Vim: Open man page for the 'dd' command: $ man dd | col -b | vim -MR - It’s actually a handful to type but since I use zsh + fzf, it’s not really a big deal as recalling the command is pretty straightforward. But yesterday, I came across this nifty little plugin called vim-manpager.
1 minute
Connecting to Torguard VPN using Wireguard from Ubuntu
2020-03-25
As of this writing, Torguard’s desktop app still doesn’t support Wireguard out of the box. I’m using Ubuntu 19.10 for testing. If you haven’t installed Wireguard yet, you can refer to the installation instructions here. In my case, it was: $ sudo apt install wireguard Next, you need to login to your Torguard client area, go to Tools –> Enable Wireguard Access. Select a location (in my case, I chose Asia-Singapore2) then click ‘Enable Wireguard’.
Linux · Torguard · Ubuntu · Wireguard
1 minute
Follow-up on autocompletion with gocode and vim-go in Go1.11, transition to gopls
2020-03-25
Update vim-go has already transitioned to gopls as its default completion engine. I just noticed that my Vim is still using gocode even though I updated to the latest version of vim-go. Commenting/removing the line Plugin ‘stamblerre/gocode’, {‘rtp’: ‘vim/’} from my .vimrc did the trick for me. Hope this helps. Original post here.
Autocomplete · Gocode · Gopls · Vim · Vim-Go
1 minute
2020-03-18
If you’re using glog in your Go codes, you can output those when running go test … by using the –args parameter: $ go test -v ./… -count=1 -cover -race -mod=vendor –args –logtostderr –v=1
1 minute
Using tabwriter to improve on cobra’s help information
2020-02-27
If you’ve been following the blog, you know that I’m a fan of cobra as a CLI library. Let me share how I use tabwriter to compliment cobra’s autogenerated help information. For reference, you can check this post as well. The following code is a copy (not an exact copy) of one of the tools that I use at work. It will look something like this. Trigger a manual run, among other tools.
1 minute
Update to kubepfm, a kubectl port-forward wrapper for multiple pods
2020-01-27
Updated 2020/01/27: Support for forwarding to deployments and services Original post: I recently uploaded a tool to GitHub that wraps kubectl port-forward command and supports port-forwarding to multiple pods. It’s called kubepfm. I use this tool heavily in my development work. You can check out the code here. You might find this useful.
K8s · Kubectl · Kubepfm · Kubernetes · Port-Forward
1 minute
Extending the timeout of a Kubernetes service in GCP
2019-11-28
This is related to a previous post about Kubernetes services. This time, it’s about extending the timeout of an Ingress. We had a situation where we had to download a huge file from one of our exposed services. The download takes about two minutes to complete. This didn’t really worked out since by default, GCP load balancers that are associated with k8s Ingresses have a timeout value of 30s. For a time, we just did manual updates by going to the GCP k8s Services and Ingress console, opening the backend service under the Ingress, and editing the Timeout section to the desired seconds.
1 minute
Output golang cmdline tools to stdout and file using tee
2019-10-29
Some golang-based tools I’ve used (or even written) use the builtin log package that output logs to stderr by default. I also use tee for piping console outputs to file for later viewing. This is the command I generally use: Example tool: $ sometool –flag1 –flag2 2>&1 | tee out.txt # 2>&1 <– redirect stderr to stdout # tee <– pipe the console output to out.txt while retaining the actual console logs during command execution
Golang · Stderr · Stdout · Tee
1 minute
Using stern together with grep
2019-09-30
I’ve been using stern as my goto log viewer for Kubernetes. It supports multiple pods and other convenient functions on top of kubectl logs. And of course, grepping goes hand in hand with log viewing, doesn’t it? To combine grep with stern, I use the following commands: For OSX: # stern <some-prod-prefix> | grep -i –line-buffered -E '<extended-regex>' $ stern user | grep -i –line-buffered -E 'failed' # For Linux (Ubuntu specifically): # stern <some-prod-prefix> | grep -i -E '<extended-regex>' $ stern user -s 1h | grep -i -E 'error'
1 minute
Using SSH tunnelling to access home router
2019-08-14
For personal reference: My current setup: Router has a static IP provided by my ISP Router web server is accessible from 192.168.1.1 (home network) SSH server box in my home network with a static IP of 192.168.1.130 (home network) Router port forwarding is setup to forward router SSH port to my SSH server box (192.168.1.130:22) SSH server box is configured to only accept public/private key authentication; password disabled Reason for access:
1 minute
Using Homebrew for distributing Go (golang) apps
2019-07-30
For personal reference: a) Create your own homebrew tap Create a new GitHub public repository with a prefix homebrew-, i.e. homebrew-tap. This will house all the apps that you want to distribute via your tap. Users will install your apps using the following commands: No need to include the 'homebrew-' prefix $ brew tap flowerinthenight/tap $ brew install <toolname> The toolname part will correspond to the filename inside your repository tap.
2 minutes
Using kettle library for master election using distributed locking
2019-06-05
I uploaded a simple library that abstracts the use of distributed locking to elect a master among group of workers at a specified time interval. It’s called kettle. You can find the source code here. We’ve been using this library mostly on these two recurring use case patterns: Consuming a dynamodbstreams endpoint across multiple containers running in Kubernetes. In this case, the master tracks the dynamic addition/removal of shards of a specific endpoint and distributes those shards to the other worker containers.
Distributed-Locking · Master · Redis · Worker
1 minute
Adding CORS support to nginx proxy in Kubernetes
2019-05-26
At work, for a couple of months now, we’ve been using Ambassador as our main API gateway to our k8s services. We also have our own authorization service that uses Ambassador’s AuthService mechanism. Recently, we’ve had services that needed CORS support and although Ambassador has features that support the enabling of CORS, we had to update our authorization service to handle CORS-related requests. Instead of doing this, we tried adding the CORS support at the proxy level (nginx).
Cors · Kubernetes · Nginx · Proxy
2 minutes
UMDF driver code for DRV2605 haptic device
2019-04-28
I uploaded an ancient prototype driver code I used ages ago in testing the DRV2605 haptic device from Texas Instruments. I can’t figure out (and I also don’t have time either) how to make this thing build using the recent versions of Visual Studio. Nonetheless, I’m putting this out there as reference to somebody doing something similar to this project. Source code is here.
Driver · Drv2605 · Haptic · Umdf · Windows
1 minute
Updates to Camera Class Filter Driver for Windows
2019-03-13
Updated 2019/03/13: A pretty good writeup that explains the internals of writing a camera filter driver in Windows can be found here. I’m putting this information out as this repo is one of the references used in the writeup. Original post: This post is a bit of a departure from my usual golang/cloud-related ramblings. I posted an open-source camera class filter driver for Windows ages ago hoping that it would help someone working on a similar project.
Camera · Driver · Filter-Driver · Wdm · Windows
1 minute
Download AWS Athena query results as CSV
2019-03-09
I recently uploaded a tool to GitHub that downloads AWS Athena query results as CSV. It’s called athena2csv. You can check out the code here. You might find this useful.
Athena · Athena2csv · Aws · Csv · Golang
1 minute
Update to kubepfm, a kubectl port-forward wrapper for multiple pods
2019-02-20
Updated 2019/02/20: Support for namespaces https://github.com/flowerinthenight/kubepfm/issues/1 Improved handling of input patterns Original post: I recently uploaded a tool to GitHub that wraps kubectl port-forward command and supports port-forwarding to multiple pods. It’s called kubepfm. I use this tool heavily in my development work. You can check out the code here. You might find this useful.
K8s · Kubectl · Kubepfm · Kubernetes · Port-Forward
1 minute
Using k8s.io/klog together with cobra in golang
2019-02-05
This post is somehow related to a previous article about using glog together with cobra. This time, we will be using klog which is a Kubernetes fork of glog. run the -h command $ ./cobraklog -h Usage of ./cobraklog: -alsologtostderr log to standard error as well as files -log_backtrace_at value when logging hits line file:N, emit a stack trace -log_dir string If non-empty, write log files in this directory -log_file string If non-empty, use this log file -logtostderr log to standard error instead of files -skip_headers If true, avoid header prefixes in the log messages -stderrthreshold value logs at or above this threshold go to stderr (default 2) -v value log level for V logs -vmodule value comma-separated list of pattern=N settings for file-filtered logging # run cobra's help command $ .
2 minutes
Using nginx basic authentication in Kubernetes
2019-01-31
This post is somewhat related to a previous post about accessing k8s services using nginx reverse proxy. Let’s try to add a simple basic authentication to these services at the proxy level. Now, this may come in handy in non-production environments but at the very least, make sure that you are doing this over HTTPS as basic authentication credentials are not encrypted. We will be using the htpasswd tool to generate our passwords.
Basic-Auth · Kubernetes · Nginx
2 minutes
Mobingi’s Kubernetes clusters repo
2018-12-27
I posted an article for @mobingi about its Kubernetes clusters repo. Please check it out. Thanks.
Go · K8s · Kubernetes · Mobingi · Repo
1 minute
Monitoring dynamodb table using dynamodbstream, lambda, SNS and SQS
2018-11-19
I uploaded an example code on how to monitor a Dynamodb table (INSERT, MODIFY, and REMOVE) using dynamodbstreams, lambda, SNS, and SQS. Please have a look here.
Aws · Dynamodb · Dynamodbstreams · Lambda · Sns · Sqs
1 minute
AWS SQS lengthy message processing
2018-10-05
Before, I uploaded an example on how to handle lengthy processing of PubSub subscription messages in GitHub. This time, it’s for AWS SQS. Have a look at the sample code here.
Aws · Client · Lengthy · Sqs · Subscription
1 minute
2018-09-25
I posted an article for @mobingi about its Golang-based monorepo. Please check it out. Thanks.
Go · Golang · Mobingi · Monorepo
1 minute
Autocompletion with gocode and vim-go in Go1.11
2018-09-20
I use vim with vim-go as my main editor for Go-based projects (and for anything else, really). Recently, I’ve upgraded to Go1.11 and my autocompletion stopped working. I know vim-go uses gocode as it’s autocompletion engine but since vim-go’s :GoInstallBinaries command handled the installation for me, I didn’t really have to think about it. Fortunately, a quick update to my vimrc file did the trick for me. I just had to add Plugin ‘mdempsky/gocode’, {‘rtp’: ‘vim/’} to my vimrc (I use Vundle) and run :PluginInstall again.
Autocomplete · Gocode · Vim · Vim-Go
1 minute
GCP PubSub lengthy message processing
2018-08-13
I have uploaded an example of how to handle lengthy processing of PubSub subscription messages in GitHub. If you are using PubSub and are, at times, struggling with PubSub default deadlines, you might find this example helpful. The code can be found here.
Client · Gcp · Lengthy · Pubsub · Subscription
1 minute
kubepfm, a kubectl port-forward wrapper for multiple pods
2018-07-24
I recently uploaded a tool to GitHub that wraps kubectl port-forward command and supports port-forwarding to multiple pods. It’s called kubepfm. I use this tool heavily in my development work. You can check out the code here. You might find this useful.
K8s · Kubectl · Kubepfm · Kubernetes · Port-Forward
1 minute
Authenticating Google API Client Library for Go using Service Accounts
2018-06-09
This post is specifically for the autogenerated Google APIs Client for Go. I haven’t tried the other Google Cloud Library for Go since it didn’t have the compute library I needed. You can use the golang.org/x/oauth2/google library for authentication with this library. It can work with service accounts as well, which is what I am using at the moment. Using the GOOGLE_APPLICATION_CREDENTIALS environment variable After you have created your service account, downloaded the JSON file, and saved it in some location, you can set the GOOGLE_APPLICATION_CREDENTIALS environment variable with the path of your service account JSON file.
Api · Authentication · Gcp · Google · Service-Account
1 minute
Ubuntu 16.04 LTS to Ubuntu 18.04 LTS
2018-05-06
I just upgraded my home server from Ubuntu 16.04 LTS to 18.04 LTS without any problems so far. For context, this server runs a Samba file server using ZFS. Although after update, I had to run the following command to my pool: pool name is 'm0' $ sudo zpool upgrade m0 By the way, at the time of this writing, Ubuntu Server update is not yet available (I think you have to wait for 18.
Bionic · Ubuntu · Upgrade · Xenial
1 minute
Camera Class Filter Driver for Windows
2018-04-30
This post is a bit of a departure from my usual golang/cloud-related ramblings. I posted an open-source camera class filter driver for Windows ages ago hoping that it would help someone working on a similar project. If you are familiar with this type of driver, you probably know that it’s not that straightforward to write mainly due to it being generally undocumented. A lot of reverse engineering has been done to write this driver.
Camera · Driver · Filter-Driver · Wdm · Windows
1 minute
Accessing services in Kubernetes
2018-03-31
At Mobingi, when we are developing services that run on Kubernetes, we generally use Minikube or Kubernetes in Docker for Mac. We also have a cluster that runs on GKE that we use for development. In this post, I will share how we access some of the services that are running on our development cluster. Using kubectl port-forward Using kubectl port-forward is probably the cheapest and the most straightforward. For example, if I want to access a cluster service svc1 through my localhost, I use kubectl port-forward like this:
Ingress · Kubectl · Kubernetes · Nginx · Port-Forward
4 minutes
Creating a Kubernetes TLS secret using certificates from DigiCert
2018-02-20
It took me a while to make this work. I hope this will help someone out there who also is struggling with the same problem. We use DigiCert as our SSL certificate provider. The package I received contained three files: a keyfile, filename.key a certificate file, filename.crt an intermediate certificate file, DigiCertCA.crt I had to combine the two certificate files into a single file. I didn’t really check the order but I appended the intermediate certificate to my certificate file.
DigiCert · Kubernetes · TLS
1 minute
A golang-based monorepo example
2018-02-06
I uploaded an example of a golang-based monorepo in GitHub. This is a stripped down version of what we have @mobingi. If you are planning to have a golang-based monorepo, please have a look.
1 minute
Running Kubernetes on AWS using kops
2018-01-24
Overview This post will walk through the steps on how we provisioned our production Kubernetes cluster @mobingi. Some of the bits here are already automated in our case but I will try to include as much details as I can. Our goals would be the following: Provision a Kubernetes cluster on AWS using kops. The cluster will have two autoscaling groups: one for on-demand, one for spot instances. It’s going to be a gossip-based cluster.
Aws · Devops · Kops · Kubernetes
5 minutes
Using glog together with cobra in golang
2017-12-01
If you have been programming with golang, you’ve probably heard of cobra. I use it extensively at work and also in my personal projects. Recently though, I’ve been using glog more and more. And I quite like it. The thing is, it has a couple of flag definitions in its init() function using golang’s builtin flag library. And I wanted to include those flags into cobra’s flag definitions. This is how I did it.
2 minutes
Embed code from GitHub to GitHub Pages blog like Gist
2017-11-28
I’ve found this nifty little tool that can embed codes directly from your GitHub repositories without using Gist. It’s very easy to use. For example, if you want to embed, say, this whole file, you only need to add this snippet somewhere in your post.md file. <script charset="UTF-8" src="https://gist-it.appspot.com/github.com/flowerinthenight/rusttrace/blob/master/src/main.rs?footer=minimal"></script> The ?footer=minimal part is optional. It will look something like this: It can also embed just a section of code based on line numbers.
Gist-It · Github-Pages · Jekyll
1 minute
Using OS specific stores for storing CLI credentials for golang apps
2017-10-30
This post is to show a simple way of using Docker’s credential helper package to utilize the system’s native credential store as storage for your Golang-based CLI applications’ login credentials. This means Keychain for OSX, wincred for Windows, and pass for Linux. We use pass here since secretservice, although supported, doesn’t work out of the box in headless servers. Here’s a simple implementation of our Set, Get, and Del functions.
Go · Keychain · Nativestore · Pass · Wincred
1 minute
Using Docker’s ETW log driver in Windows
2017-09-20
In Docker’s ETW logging driver doc, it uses the tool logman to view the logs. In this article, I will show you how to use mftrace to view Docker ETW logs in real-time. First, here’s a simple application written in Go that logs to STDERR every second. Next, let’s create a Docker image (Windows) using the Dockerfile below. assuming the code above is saved in a directory called 'demoapp' $ docker build -t demoapp .
1 minute
2017-07-31
As of this writing, GitHub Pages still doesn’t support HTTPS for custom domains. Last week, I came across Netlify being able to serve GitHub Pages as is, with HTTPS support even with custom domains. I gave it a go. And it worked. If you’re in the same boat as me (using GH-Pages with custom domains), give Netlify a go. Note: I am not affiliated with Netlify in any way, whatsoever.
1 minute
Wrapper for amqp library for RabbitMQ
2017-06-30
I uploaded a simple wrapper to streadway/amqp library for RabbitMQ with support for auto reconnections. This is a simplified version of what I’m using in production.
1 minute
Start process as system using CreateProcessAsUser
2017-05-31
Note that this function assumes the caller to be running as SYSTEM as well (i.e. Windows service). For self reference:
1 minute
Using APC as FIFO queue in Windows
2017-04-20
In this way, we can implement a FIFO queue without using explicit locking/synchronization for enqueueing/dequeueing. For self reference:
1 minute
Simple touch/swipe input function in Windows
2017-03-28
For self reference: Simulate a touch input down, drag, up programmatically.
1 minute
Simple Innosetup installation script
2017-02-28
InstallShield seems ubiquitous when it comes to installer scripting, at least based on my experience with companies I’ve worked for so far. But for my side projects, I’ve always been a fan of InnoSetup. It is simple to use and Pascal is not really that bad. It is free and can do probably all the things you require for an installer. Here’s an example of an InnoSetup script that I have been using as baseline for creating installers for Windows.
Innosetup · Installer · Pascal
1 minute
2017-01-18
Yet another service code for Windows. This time, it’s an ATL service. ATL services are basically the same as traditional Windows services but with some advantages. Can be started on-demand automatically by the first client call (via COM). Clients can call service functions with parameters and return values using COM. In traditional services, clients normally communicate using service control codes and you need some kind of IPC (named pipes, shared memory, etc.
1 minute
2017-01-04
Happy new year to all! To the bookworms out there, I’ve added a new page in the site called Bookshelf, a list of all the books I’ve read so far. If you have any suggestions/recommendations, please don’t hesitate to leave a comment. Thank you.
1 minute
2016-12-26
I’ve been dabbling a bit with Rust and so far, I’m liking it. I’ll probably use it more as replacement for my C/C++ projects. And I just uploaded a very simple ETW wrapper I’m using in one of my side projects. You can check out the source code here.
1 minute
Encoding .NET bitmaps to H264 using FFMPEG
2016-11-28
Recently, I was working on a project that involves the encoding of .NET bitmaps using ffmpeg’s h264 encoder with mp4 as container. This video output will be used in a <video> tag in html5. Sample codes have been all over the place so it took me quite a while to come up with a working solution. The official sample from ffmpeg only encodes to raw h264 stream. This is just a basic sample code for reference.
1 minute
A simple cron-like Windows service
2016-10-07
A couple of days ago, I uploaded a generic service in GitHub that has a feature of updating itself. Check out the post here. Today, I uploaded the actual service I’m using in our CI environment, excluding the configuration file. Check out the source code here.
1 minute
A Windows service with an http endpoint for uploading a new version of itself
2016-09-27
Recently, I’ve been working on a service that runs on a lot of VM’s across different locations. If I have a new service build, updating all of the running instances quickly became a bit of a pain. I have to log in to every VM (in some cases through a VPN) and then do a manual upgrade. Now, there are probably tools that already exist for this type of use case but since I’m still learning Go at the moment, I thought this would be a good exercise.
1 minute
A simple REST client/server console app in C#
2016-08-24
For personal reference: Full source code here.
1 minute
2016-08-22
For personal reference: To view syslog in realtime tail -f /var/log/syslog
1 minute
How to serve expvar when using gorilla/mux
2016-08-20
For personal reference: Access root http://localhost:8000 Access expvar information http://localhost:8000/debug/vars
1 minute
My commonly used commands in GIT
2016-08-18
Update 2017/03/24: Transferred to a separate repository here. For personal reference: Reset a file git checkout HEAD – my-file.txt Delete last commit git reset –hard HEAD~1 Delete local branch git branch -d <branch-name> or to force delete git branch -D <branch-name> Delete branch from remote repository git push origin –delete <remote-branch-name> Search for the merge commit from a specific commit git log <SHA>..master –ancestry-path –merges Search for a commit message git log | grep <pattern> List commits on range line of codes for one file git blame -L<line#>,+<offset> – <filename> For example, three lines starting from line 257 of main.
2 minutes
2016-08-11
I use tmux heavily and in tandem with vim. Much more so now when it’s supported on Bash on Windows as well. I don’t have to spin up a Linux VM just for the purpose of being my tmux “server”. Set a Ctrl-b shortcut for reloading tmux config unbind r bind r source-file ~/.tmux.conf # Prefix is Ctrl-a unbind C-b set -g prefix C-a bind C-a send-prefix # Rename terminals set -g set-titles on set -g set-titles-string '#(whoami)@#h@#(curl ipecho.
1 minute
2016-07-27
Vim has always been my go to editor/IDE when I’m outside of Visual Studio. Here’s my base _vimrc for Windows. let mapleader = " " filetype off syntax on colorscheme darkblue " let pc=$PC " if pc == 'HOME' " set guifont=Letter\ Gothic\ Std:h11 " else " set guifont=Lucida\ Sans\ Typewriter:h9 " endif " Save marks to up to 100 files, save global marks as well (f1). To disable, f0 set viminfo='100,f1 " Folding options set foldmethod=indent set foldnestmax=20 set nofoldenable set foldlevel=0 set guifont=Lucida\ Sans\ Typewriter set lines=70 columns=160 set ai set nu set tabstop=4 set shiftwidth=4 set softtabstop=4 set expandtab set wrap set backspace=2 set encoding=utf-8 set fileencodings=utf-8 set nocompatible set noswapfile set shortmess+=I set ignorecase set guioptions-=T set guioptions-=r set guioptions-=m set splitright set splitbelow set ruler set rtp+=$HOME/vimfiles/bundle/Vundle.
4 minutes
A simple Powershell function to import an environment
2016-05-06
A couple of days ago, I was working on a powershell-based script for mstest automation and I needed to call vsdevcmd.bat from Visual Studio’s tools folder. function Invoke-Environment([Parameter(Mandatory=1)][string]$Command, [switch]$Output, [switch]$Force) { $stream = if ($Output) { ($temp = [IO.Path]::GetTempFileName()) } else { 'nul' } $operator = if ($Force) {'&'} else {'&&'} foreach($_ in cmd /c " $Command > "$stream" 2>&1 $operator SET") { if ($_ -match '^([^=]+)=(.*)') { [System.Environment]::SetEnvironmentVariable($matches[1], $matches[2]) } } if ($Output) { Get-Content -LiteralPath $temp Remove-Item -LiteralPath $temp } } To use the function:
1 minute
2016-04-10
Downloading… {% include image.html url="../../../../images/win10preview1.png" %} And here we go… {% include image.html url="../../../../images/win10preview2.png" %} Looking good so far.
1 minute
A simple unzip function in Powershell
2016-04-06
function UnzipFiles($ZipFileName, $DestDir) { Add-Type -Assembly System.IO.Compression.FileSystem [System.IO.Compression.ZipFile]::ExtractToDirectory($ZipFileName, $DestDir) } To use the function: UnzipFiles -ZipFileName .\folder\file.zip -DestDir .\destination
1 minute
A simple zip function in Powershell
2016-04-05
function ZipFiles($ZipFileName, $SourceDir) { Add-Type -Assembly System.IO.Compression.FileSystem $compressionLevel = [System.IO.Compression.CompressionLevel]::Optimal [System.IO.Compression.ZipFile]::CreateFromDirectory($SourceDir, $ZipFileName, $compressionLevel, $false) } To use the function: ZipFiles -ZipFileName test.zip -SourceDir .\folder\to\zip
1 minute
How I set up my Samba file server
2016-03-28
I installed Samba to my Ubuntu Server (which is an old ThinkPad laptop that has been gathering dust in my closet for ages): sudo apt-get install samba I have three 3-TB HDDs that I planned to use as my main server storage; one main server, and the other two for backup. All of these HDDs are in ext4 format. Then I created a new directory under /media where I will mount my main server storage:
1 minute
[Part 6] Logging with Universal Apps
2016-03-22
Check out the codes in GitHub. Basically, 2 steps: Create a C++ Windows Runtime Component (dll) that will use the ETW header file. Reference the WinRT DLL to C# project. WinRT DLL My logging class looks like this: public ref class TraceCore sealed { private: TraceCore(); ~TraceCore(); static TraceCore^ m_Instance; public: static property TraceCore^ Instance { TraceCore^ get() { if (m_Instance == nullptr) { m_Instance = ref new TraceCore(); } return m_Instance; } } void Verbose(Platform::String^ mod, Platform::String^ file, Platform::String^ func, Platform::String^ m); }; And heres the implementation:
1 minute
[Part 5] Logging with C# applications
2016-03-19
Check out the codes in GitHub. I generated a C# logging class using this command: mc -css <Namespace_here> jytrace.man So far, most of my event templates use an ANSI string data type which (I believe) is not supported by C#. As you can see in part 1, I use ANSI data type for my File and Function fields so that I can use FILE and FUNCTION as inputs in C++. Thats why I added a new event template with all fields using UNICODE data types.
1 minute
2016-03-07
Check out the codes in GitHub. Getting the logs Now that I have my modules spitting out logs for me, it’s time to actually consume (or view) them. Most of the time, I use MFTrace and logman tools. MFTrace Although MFTrace is primarily a tool for generating logs for Media Foundation apps, it is a great tool for viewing ETW logs in general as well. It is included in the MS SDK.
2 minutes
[Part 3] Logging with C/C++ applications
2016-03-03
Check out the codes in GitHub. If you remember in part 2, we compiled our manifest file with mc -um <manifest_file>.man and we got a header file as one of the outputs. We just have to include that header file to our sources and we are good to go. #include … #include "<manifest_file>.h" int main(…) { EventRegister<provider_name_in_manifest>(); … CreateFile(…); // Example of using the LastError event in our manifest file EventWriteLastError(L"THIS_EXE", FILE, FUNC, L"CreateFile", GetLastError()); .
1 minute
[Part 2] Manifest file compilation and setup
2016-03-02
Check out the codes in GitHub. Compiling the manifest file This is how I compiled my manifest file. Open Visual Studio command prompt. Compile manifest file. mc -um <manifest_file>.man When successful, output files are: <filename>.h <filename>.rc <filename>TEMP.BIN MSG00001.bin Creating the message resource DLL I like to create a separate dll just for the manifest file resource although you can also add the .rc file to any of your existing dll or exe.
1 minute
[Part 1] How I set up ETW for logging (native, .NET, Metro and UApp)
2016-03-01
Check out the codes in GitHub. Ive been using ETW as my go-to logging mechanism for Windows development. I will be sharing how I setup my environment here. Creating the manifest file (.man) I use ecmangen.exe to create my manifest file. This tool is included in the SDK. I have Visual Studio 2015 installed on a Win10 64-bit machine, and its location is C:\Program Files (x86)\Windows Kits\10\bin\x64. It looks something like this (my completed manifest):
3 minutes
How to apply for a Nanaco point card
2016-02-25
After six or so years living in Japan, I just got mine today. Yeah, I know… GaijinPot has an article about point cards from three of the most common convenience stores in the country. I already have Lawsons Ponta card and Family Marts T-Point ages ago but for some reason, I actually havent heard of 7-11s Nanaco. Or maybe I did but I probably didnt bother. But now that I have a Seven Bank account, I noticed from their online bank access site an option to register to their point service.
2 minutes
How I setup git for my WordPress installation in BlueHost
2016-02-24
I’m not sure if this is the “proper” way to do it. Well, it sort of works for me at the moment so I thought I’ll share it here. I did use FTP at first (using FileZilla) but I didnt really like the workflow. Host: BlueHost shared account Client: Windows 10 Prerequisites SSH/Shell Access should be enabled. I enabled this from my cPanel -> SSH/Shell Access menu. Server/Host side SSH to BlueHost host.
Bluehost · Git · Tech · Wordpress
2 minutes
SSH from a Windows client to a Linux server
2016-02-22
At the moment, I’m still in the process of setting up my home network server. I’ll be taking notes about my progress here so this might be a series of related posts. I’m using an Ubuntu 14.04 Server (headless). Client is a Windows 10 machine. The simple way Server side Install ssh. How to do this depends on what distro you are using. For Ubuntu, you can do an apt-get install ssh.
2 minutes
2016-02-21
Hey, there! Welcome to my blog. I hope you enjoy reading the stuff in here. Nothing fancy, really. Just bits and bobs about tech and random topics. Enjoy!
1 minute