Cluster membership management on AWS
2025-02-07
In continuation with my previous post, I have now finished porting hedge to AWS. It’s a trimmed-down version for now; only the features directly related to cluster membership are ported. I decided to make a separate repo, called hedge-cb (in keeping with the -cb theme), instead of updating hedge directly. And it’s mainly due to CGO. I didn’t really fancy the idea of introducing CGO to hedge as it could break a lot of the CI builds at work.
Aws · Cgo · Clockbound · Cluster · Distributed-Systems · Ffi · Go · Golang · Hedge · Leader-Election · Memberlist · Programming · Software · Systems · Tech · Timesync · True-Time
2 minutes
Distributed locking on AWS (ClockBound)
2025-02-02
After some testing time, I now have a working port of spindle in AWS. There were some slight changes from the original library to account for some of the differences between Cloud Spanner and PostgreSQL, but not really by much. It’s called spindle-cb, if you’re interested. It’s still half the battle though; I still have to port hedge as well before I could really use it for some of the planned projects in my pipeline.
Aws · Cgo · Clockbound · Distributed-Locking · Ffi · Go · Golang · Locking · Programming · Software · Spindle · Systems · Tech · Timesync · True-Time
2 minutes
AWS ClockBound client for Go (update)
2025-01-27
A week ago I published a short blog about clockbound-client-go. After some testing, turns out there’s an issue in reading the actual time from ClockBound’s shared memory segment; it seems to provide only the elapsed time since boot. However, using the Rust client and the FFI bindings produce the correct results. Either there is a problem in the code that reads the shared memory segment, or the SHM contents are wrong.
Aws · Cgo · Clockbound · Ffi · Go · Golang · Programming · Software · Systems · Tech · Timesync · True-Time
1 minute
2025-01-22
I’ve written a Go client for AWS ClockBound called clockbound-client-go. It uses the newer, shared memory segment protocol instead of the older, socket-based protocol. This is a prerequisite library needed to port spindle (and maybe even hedge) to AWS (for an upcoming project). As great a tech Google’s TrueTime is, there is no available API for it. It is only through Spanner that spindle achieves its locking mechanisms with TrueTime. Currently, it’s the cheapest way, surprisingly, to do distributed locking that I’ve tried so far, compared to the likes of Redis, Zookeeper, etcd, Consul, etc.
Aws · Clockbound · Go · Golang · Programming · Software · Systems · Tech · Timesync · True-Time
1 minute
Authenticating Go AWS SDK v2 using external id
2023-03-01
For self reference: Sample code as to how to authenticate aws-sdk-go-v2 using external ids:
Assume · Aws · External-Id · Golang · Roles · Sdk · Tech · V2
1 minute
Authenticating Go AWS SDK using external id
2022-10-17
For self reference: Sample code as to how to authenticate aws-sdk-go using external ids:
Assume · Aws · External-Id · Golang · Roles · Sdk · Tech
1 minute
Authenticating Go AWS SDK v2 using assume roles
2021-04-30
For self reference: To authenticate the Golang AWS SDK v2 using assume roles, refer to the following code snippets:
Assume · Aws · Golang · Roles · Sdk · V2
1 minute
A tool for synching DynamoDB tables between accounts
2020-08-31
I uploaded a tool that can sync DynamoDB tables between accounts. It’s called dysync and you can find the code here. Maybe it will be useful to anybody out there.
Aws · Dynamodb · Golang · Sync
1 minute
2020-05-13
I uploaded a tool that can query DynamoDB tables. It’s called lsdy and you can find the code here. Maybe it will be useful to anybody out there.
Aws · Dynamodb · Golang · Query
1 minute
Download AWS Athena query results as CSV
2019-03-09
I recently uploaded a tool to GitHub that downloads AWS Athena query results as CSV. It’s called athena2csv. You can check out the code here. You might find this useful.
Athena · Athena2csv · Aws · Csv · Golang
1 minute
Monitoring dynamodb table using dynamodbstream, lambda, SNS and SQS
2018-11-19
I uploaded an example code on how to monitor a Dynamodb table (INSERT, MODIFY, and REMOVE) using dynamodbstreams, lambda, SNS, and SQS. Please have a look here.
Aws · Dynamodb · Dynamodbstreams · Lambda · Sns · Sqs
1 minute
AWS SQS lengthy message processing
2018-10-05
Before, I uploaded an example on how to handle lengthy processing of PubSub subscription messages in GitHub. This time, it’s for AWS SQS. Have a look at the sample code here.
Aws · Client · Lengthy · Sqs · Subscription
1 minute
Running Kubernetes on AWS using kops
2018-01-24
Overview This post will walk through the steps on how we provisioned our production Kubernetes cluster @mobingi. Some of the bits here are already automated in our case but I will try to include as much details as I can. Our goals would be the following: Provision a Kubernetes cluster on AWS using kops. The cluster will have two autoscaling groups: one for on-demand, one for spot instances. It’s going to be a gossip-based cluster.
Aws · Devops · Kops · Kubernetes
5 minutes