Kubernetes as a way to better support and manage the web applications our department developed. I learned a lot about Docker and Kubernetes very quickly. In case you aren't familiar, Docker is basically a way to package up server software so it can more easily be installed and run anywhere. It has its origins in lighter weight ways to separate and run multiple services on the same Linux host, like chroot, vservers, and containers. You write a smallish file that contains instructions to build an image of your application, basically its environment and resources, that can be run on any compatible system. Because Docker and similar technologies don't have to emulate an entire machine, only isolating and sharing the same underlying resources directly, they scale very well.
Most hosting providers use both kinds of virtualization. Instead of a regular operating system, they often install one designed to use full emulation to split a physical box with dozens of CPUs and obscene amounts of memory into many virtual servers. Each of these is as capable as dedicated hardware but way more cost efficient. Within a virtual server, Docker is a great way to split the resources up even further. Kubernetes is an open stack developed at Google that abstracts over whatever the underlying servers are, virtual or physical, to create one large, flexible resource pool into which Docker containers and other resources can be deployed and managed. I like it for separating out the lower level provisioning from the fiddly bits of any specific service.
After my crash courses in Docker and Kubernetes for work I really wanted to overhaul my Rackspace account to use both. This was before any providers had good support for Kubernetes. Rackspace had some support and some articles but the approach was very specific to their service. Part of the appeal of Kubernetes is it can be used to avoid provider lock-in. I did manage to introduce Docker with some scripts I created myself to make my setup less error prone and easier to update. This worked well enough for several years but the quality of support I received from Rackspace continually declined. One of my biggest frustrations was when I had to call someone to delete a server I was done with.
I used Docker for the same reason many dev ops teams adopted it. In Rackspace, I could create a new server and move all of my running services over to it easily. Update the DNS records and this was how I handled server upgrades. I had a few problems with Rackspace's lower level virtual servers when I tried to upgrade them like an actual machine in my office. Having to speak to a human to remove a server post-upgrade was a hassle. For a long time I thought I was stuck, that my other options were larger, worse "cloud" providers like AWS.
The end of my email hosting helped encourage me to reconsider my dissatisfaction with Rackspace and the current alternatives. An old, good friend was going to be in town soon. We realized that emails he had been sending were not reaching me. I had had enough and made the decision to find and pay a dedicated provider for what had become more of a pain than it was worth, even for the important principles at stake. When I migrated my mail to a dedicated, secure, privacy prioritizing provider, I had to call them again to delete the old server so I wouldn't continue to be billed for it. I started talking with friends who also self hosted, curious to learn what was new and worth considering. Maybe I could call them to cancel my account rather than delete yet another server.
One friend in particular kept raving about Digital Ocean. They have tons of creative projects and uses Digital Ocean to spin up servers as needed to help craft their desired experiences. They are also a programmer but may have the least patience for necessary infrastructure I have ever seen. That made for a hell of an endorsement. Inexpensive enough to casually spin things up to try them out. Easy enough that the focus can remain on the project rather than fighting the plumbing.
I decided to spend the long holiday weekend in November migrating another one of my servers over. In Rackspace I originally had one server for email, another for all of my websites, and a third for a grab bag of services. I thought my websites would be a good to start exploring how I could host them at Digital Ocean. I use a static site generator and was already deploying an nginx based image with the generated site copied and configured. Perfect for deployment in Kubernetes. After adding a couple of extra services to the stock configuration of Digital Ocean's managed Kubernetes service, I had my first site deployed. By the end of the first day, I had successfully migrated all of my websites and added another server to the list for deletion at Rackspace.
Kubernetes can be a steep learning curve at first. Once you understand a few concepts and ways of doing things, you can get a long way with only a little effort. When I last used Kubernetes it was along with jsonnet, a templating language that makes writing JSON and yaml far simpler. It especially shines for repetitive information like Kubernetes manifests. Digital Ocean has fantastic command line support for a cluster. I can shell into the underlying nodes if I need to but can do most of what I'd use that for anyway with their command line tool. I like that Kubernetes is also command line friendly. You can examine, deploy, configure, and delete anything and everything in a cluster from a terminal. There is a nice web console too, if you prefer, and Digital Ocean links to it from their dashboard. I added kubecfg which also uses jsonnet and each resulting configuration file for my my static websites requires less than ten lines. Pushing an update, like for a blog post, is a single command.
I was now down to one server in Rackspace. With a couple of days of the weekend left, I started work bringing over the remaining services. These were trickier. My RSS aggregator went into the cluster without any issues. Digital Ocean supports my favorite database, PostgreSQL, which I had been using in Rackspace. The export and re-import went smoothly, I didn't lose any of my reading history. My Taskwarrior server and Gitea (which I will replace with Forgejo this week) couldn't go into the cluster. The load balancer is still limited, mostly only supporting web traffic but not other protocols like SSH. I spun up a separate virtual server and moved those containers and their databases over. I couldn't yet simplify them with Kubernetes but my old configurations and containers worked just as well with the new virtual server.
Digital Ocean reminds me of AWS but if it had been created today rather than years ago and incrementally loaded down with more and more cruft. Digital Ocean doesn't have the same breadth of services as AWS. They offer the most common services and resources, several of which like Kubernetes can be augmented with 3rd party add ons. I added a pinned tab for my account, their dashboard is great. I especially enjoy watching the Kubernetes one as the cluster hums along, doing cluster stuff. There is even a running cost estimate, useful as you add more to your account to understand how it likely will affect your bill at the end of the month.
Speaking of the bill, that was a big and pleasant surprise. Hosting only what I used to run on Rackspace ended up costing me about half as much, even with the additional resources needed to spin up a modest Kubernetes cluster.
I was so pleased with my first experiences with Digital Ocean, I have since expanded my fleet. I mentioned one of the services I added in my last post. I'll save talking about the technical details of how I am running Mastodon in Digital Ocean for my next post, most likely tomorrow.