I have been self hosting my own services for years. I originally started doing so to practice new skills early in my career, then increasingly for privacy by definitively owning my own data. Since I started I have added more services to enrich and control my online experience. I recently overhauled and upgraded how I self host after a few years of inertia. I am pretty happy with where my efforts led and thought I'd share a little more about what I did.
I think the first server I stood up on the Internet may have been a web server on my home DSL connection, around the turn of the century. We had just bought our home, the apartment we moved out of had DSL paid for by work. The DSL market at that time in the US was this odd fabrication made up by the FCC, a compromise meant to preserve some open competition. The up side was being a Speakeasy customer for years. If you aren't familiar, they were a tech geek friendly ISP. The supported, and encouraged, using their always on connections for home hosting. I had a free static IP address before that devolved into the for pay only option it is with today's providers.
Next I started hosting my own email server. The website hosting was about practicing my web development skills, email was to receive messages at my own domain. Gmail didn't exist in any form back then, let alone the version, whatever Google calls it now, where you could use your own domain name. Email quickly became about privacy and for years was the main reason I persisted at self hosting. Oddly enough, I finally gave up on hosting my own email this past year. The Internet has changed enough that I made the next best compromise for privacy. I pay a credible provider with a track record of protecting their users' privacy. Gmail and spam have made email self hosting today in my view a non-starter.
My self hosting at home ended when we had a couple of bad winters. Verizon definitely overextended their DSL footprint in our area. All the copper here is above ground. When it rains, the added weight causes the lines to droop and stretch. The second or third time our DSL went out in response to bad weather, not even a power outage, we realized we likely were at the very limit of how far DSL could work over copper. Verizon never installed a repeater or invested in any way to shorten our loop which would have drastically improved our service. They still owned the copper even though we paid for service from Speakeasy. The techs at Speakeasy were saints for dealing with this situation for as long as we had their service. We finally upgraded to FiOS which has only been better because I think the tech itself is less prone to reliability issues like DSL. Verizon did just as bad a job at that install as the DSL. Some year I will need to pay someone to undo their terrible wiring and redo it properly.
I took the opportunity to create my first virtual private server around this time. Unlike DSL, there wasn't even a fig leaf of a requirement for open competition for fiber or cable. Verizon's ToS still are not compatible with self hosting. I cannot now remember the name of my original provider. They were like the Speakeasy of the early virtual hosting services. Until Rackspace bought them. I stayed at Rackspace for years out of inertia and because they were slightly less worse than AWS and the rest. I chose the provider they bought because I got full access to my virtual server. I could literally install anything I wanted, just like when I self hosted at home. I kept self hosting more like an old school systems administrator for years. I manually installed, configured, and backed everything up. This was before dev ops was a thing.
A handful of years ago at my day job I had the chance to move to a team focused on dev ops. That team led the adoption of Kubernetes as a way to better support and manage the web applications our department developed. I learned a lot about Docker and Kubernetes very quickly. In case you aren't familiar, Docker is basically a way to package up server software so it can more easily be installed and run anywhere. It has its origins in lighter weight ways to separate and run multiple services on the same Linux host, like chroot, vservers, and containers. You write a smallish file that contains instructions to build an image of your application, basically its environment and resources, that can be run on any compatible system. Because Docker and similar technologies don't have to emulate an entire machine, only isolating and sharing the same underlying resources directly, they scale very well.
Most hosting providers use both kinds of virtualization. Instead of a regular operating system, they often install one designed to use full emulation to split a physical box with dozens of CPUs and obscene amounts of memory into many virtual servers. Each of these is as capable as dedicated hardware but way more cost efficient. Within a virtual server, Docker is a great way to split the resources up even further. Kubernetes is an open stack developed at Google that abstracts over whatever the underlying servers are, virtual or physical, to create one large, flexible resource pool into which Docker containers and other resources can be deployed and managed. I like it for separating out the lower level provisioning from the fiddly bits of any specific service.
After my crash courses in Docker and Kubernetes for work I really wanted to overhaul my Rackspace account to use both. This was before any providers had good support for Kubernetes. Rackspace had some support and some articles but the approach was very specific to their service. Part of the appeal of Kubernetes is it can be used to avoid provider lock-in. I did manage to introduce Docker with some scripts I created myself to make my setup less error prone and easier to update. This worked well enough for several years but the quality of support I received from Rackspace continually declined. One of my biggest frustrations was when I had to call someone to delete a server I was done with.
I used Docker for the same reason many dev ops teams adopted it. In Rackspace, I could create a new server and move all of my running services over to it easily. Update the DNS records and this was how I handled server upgrades. I had a few problems with Rackspace's lower level virtual servers when I tried to upgrade them like an actual machine in my office. Having to speak to a human to remove a server post-upgrade was a hassle. For a long time I thought I was stuck, that my other options were larger, worse "cloud" providers like AWS.
The end of my email hosting helped encourage me to reconsider my dissatisfaction with Rackspace and the current alternatives. An old, good friend was going to be in town soon. We realized that emails he had been sending were not reaching me. I had had enough and made the decision to find and pay a dedicated provider for what had become more of a pain than it was worth, even for the important principles at stake. When I migrated my mail to a dedicated, secure, privacy prioritizing provider, I had to call them again to delete the old server so I wouldn't continue to be billed for it. I started talking with friends who also self hosted, curious to learn what was new and worth considering. Maybe I could call them to cancel my account rather than delete yet another server.
One friend in particular kept raving about Digital Ocean. They have tons of creative projects and uses Digital Ocean to spin up servers as needed to help craft their desired experiences. They are also a programmer but may have the least patience for necessary infrastructure I have ever seen. That made for a hell of an endorsement. Inexpensive enough to casually spin things up to try them out. Easy enough that the focus can remain on the project rather than fighting the plumbing.
I decided to spend the long holiday weekend in November migrating another one of my servers over. In Rackspace I originally had one server for email, another for all of my websites, and a third for a grab bag of services. I thought my websites would be a good to start exploring how I could host them at Digital Ocean. I use a static site generator and was already deploying an nginx based image with the generated site copied and configured. Perfect for deployment in Kubernetes. After adding a couple of extra services to the stock configuration of Digital Ocean's managed Kubernetes service, I had my first site deployed. By the end of the first day, I had successfully migrated all of my websites and added another server to the list for deletion at Rackspace.
Kubernetes can be a steep learning curve at first. Once you understand a few concepts and ways of doing things, you can get a long way with only a little effort. When I last used Kubernetes it was along with jsonnet, a templating language that makes writing JSON and yaml far simpler. It especially shines for repetitive information like Kubernetes manifests. Digital Ocean has fantastic command line support for a cluster. I can shell into the underlying nodes if I need to but can do most of what I'd use that for anyway with their command line tool. I like that Kubernetes is also command line friendly. You can examine, deploy, configure, and delete anything and everything in a cluster from a terminal. There is a nice web console too, if you prefer, and Digital Ocean links to it from their dashboard. I added kubecfg which also uses jsonnet and each resulting configuration file for my my static websites requires less than ten lines. Pushing an update, like for a blog post, is a single command.
I was now down to one server in Rackspace. With a couple of days of the weekend left, I started work bringing over the remaining services. These were trickier. My RSS aggregator went into the cluster without any issues. Digital Ocean supports my favorite database, PostgreSQL, which I had been using in Rackspace. The export and re-import went smoothly, I didn't lose any of my reading history. My Taskwarrior server and Gitea (which I will replace with Forgejo this week) couldn't go into the cluster. The load balancer is still limited, mostly only supporting web traffic but not other protocols like SSH. I spun up a separate virtual server and moved those containers and their databases over. I couldn't yet simplify them with Kubernetes but my old configurations and containers worked just as well with the new virtual server.
Digital Ocean reminds me of AWS but if it had been created today rather than years ago and incrementally loaded down with more and more cruft. Digital Ocean doesn't have the same breadth of services as AWS. They offer the most common services and resources, several of which like Kubernetes can be augmented with 3rd party add ons. I added a pinned tab for my account, their dashboard is great. I especially enjoy watching the Kubernetes one as the cluster hums along, doing cluster stuff. There is even a running cost estimate, useful as you add more to your account to understand how it likely will affect your bill at the end of the month.
Speaking of the bill, that was a big and pleasant surprise. Hosting only what I used to run on Rackspace ended up costing me about half as much, even with the additional resources needed to spin up a modest Kubernetes cluster.
I was so pleased with my first experiences with Digital Ocean, I have since expanded my fleet. I mentioned one of the services I added in my last post. I'll save talking about the technical details of how I am running Mastodon in Digital Ocean for my next post, most likely tomorrow.