Ceph is a distributed storage system. That severely understates its complexity and capabilities. Here is its documentation, and if you’re intending on trying Ceph out, I strongly recommend you spend some time thoroughly reading it. All of it. I spent about two weeks on and off studying it, along with Red Hat’s docs (which are […]
K8s Misadventures, Pt. 1 – Distributed Systems are Easy
It’s finally happening – I’m splitting compute and storage up, and using k8s to do so. I bought two more Dell R620s, bringing my total to three – behold, quorum. This will be a multi-part series, focusing on different things I did wrong. I want HA, or as close as I can. I know I […]
Testing VM immutability, secrets storage again, and a brief ode to ZFS
I’ve had my dev VM upgraded to Debian 11 for a few months, and have fiddled with from time to time. It’s not that I distrust Debian stable in the slightest to not be stable, but I also am not going to just throw prod a sudo apt full-upgrade (yes, with updating sources first…) and […]
Reluctant Servers
I may have mentioned this before, but I have two nearly identical Supermicro 2Us. One is the main server that hosts VMs under Proxmox to do various tasks – a NAS, a Docker host, jumpbox, etc. The other came about because I discovered that its motherboard hardware revision level was too old to support Xeon […]
A Modest Proposal for minimizing downtime and adding more blinkenlights to ye olde 25U rack
In case you haven’t been following my work (I absolutely can’t blame you, as this blog is self-congratulatory at best), I have a 25U rack with some stuff in it. In short, it consists of a Supermicro 2U X9, and a ZFS pool, with an almost-identical second Supermicro 2U X9 that exists as a cold […]
Baking VMs to Perfection
I’ve now accomplished one of my previously-mentioned desires, namely, to use Packer to make VMs for Proxmox. After much battling with YAML and esoteric bash commands, I have succeeded in being able to spawn endless VMs, ready to go just how I like them. As with many projects I’ve done, this was thanks to someone […]
Might As Well Jump
A friend wanted to learn Linux, so I offered to spin up a VM under Proxmox. Done. Just kidding. I mean, that would work (assuming you handled port forwarding) if you were hitting an IP, but FQDNs are much easier for people to remember. Except ssh isn’t based on HTTP, so how do you forward […]
Failed NVMe drive? Better change your upstream DNS resolver.
This is not entirely an “it’s always DNS joke,” I promise. But for the record, it is always DNS. I have an (three, actually – two are boot drives/VM storage) NVMe drive in my server that serves multiple functions – a Plex cache, a ZFS Intent Log (I should get rid of that, as it’s […]
Hardware Sucks
It’s coarse and rough and irritating and… it sucks. There’s a reason why cloud providers are wildly popular. Scaling in AWS is incredibly easy; not so much when you’re rolling your own. God help you if you’re doing this by cobbling together disparate groups of enterprise and consumer hardware. Oh wait, that’s exactly what I’m […]
Ascertaining Gender of Github Users to Determine PR Comment Sentiment, or How to Spend Money in the Cloud and Pretend you’re a Data Scientist
This is not so much of a SRE post, be ye warned, but it does discuss solving problems creatively. WARNING: Blindly following this will result in incurring costs from cloud providers, potentially quite a bit. It’s not my fault if you don’t calculate how much your task will cost. As an example, had I used […]