Running your side-project databases on the cheap

I thought I’d start a thread for readers who’re working on a side-project that will, one day, take over the world. You might need a relational database, and despite the economies-of-scale promises of the cloud, most things don’t seem to fit for broke hobbyists.

So this is for you if your project permanently has precisely one user (two if you include your lazy dog, :poodle: and I’m pretty sure she didn’t create an account).

Hosted

To start with, have a peek on Free For Dev, an awesome stack of curated engineering resources. The only small caution I might apply is that offering services for free costs someone some money, so we probably should approach the list intending to cough up some cash eventually.

I have a side project currently running on the free tier of Supabase Postgres; it’s reliable, I am using both of my free database slots, one of them goes to sleep every two weeks, and I click a thing to wake it up. That’s in dev, so a bit of downtime is OK.

I have a separate solo project that is probably more likely to see the light of day, and for that I presently need MySQL. I am minded to try filess.io first. Hailing from Spain, the UI is fine, and I created a small 10MB database without fuss. I liked that they create resources in Europe, but it was less good that I could not decide where the resources are created. Their online reviews are a mixed bag, and their lack of social media activity makes me wonder if this is in itself a side-project. (I like side-projects, but infrastructure probably should not itself be a side-project :see_no_evil_monkey:).

PlanetScale looks interesting, and they have a 5USD tier, but it’s Postgres only. To be fair, I’ve enough integration tests that I could probably switch to that… For completeness I’ll mention these folks do have a MySQL tier, but it looks posh and expensive.

I’ll add a note for Layerbase; I might try them too. They’ve a free 5GB tier that sleeps if necessary and insta-wakes, and weirdly you can have MariaDB but not MySQL. However they have a 5USD tier where MySQL is a permitted choice. The data storage is US-only, but perhaps distributed systems fans can work around that with on-container caching.

Finally, Neon looks like it has a free tier, with databases limited to 0.5G in size. Postgres-only, like Supabase. If you upgrade though the costs are usage-based; that can be good or bad depending on how you look at it.

Self-hosted

Whilst writing up this list, I’ve pondered two more options. I offer these with the observation that precarious one-node unmanaged clusters gets a weekly mention :laughing: in this very sub, and a few of us are minded to bop offenders on the head with an extremely bristly broom. :broom: However, I do think it can be OK if the operator is mindful of the risks.

My first approach is to find a bare-metal traditional VPS company. I can give a UK recommendation if anyone fancies it; they’re not really competing with Fly, as they have a very different product. I’d wager MySQL would run on, say, a 1GB RAM instance for months before needing a hypervisor reboot. Of course, configure hourly and weekly backups, and make sure they are rsynced to another place frequently. Often you can get away with a IPv6-only machine, which should be a bit cheaper.

The other approach is to use Fly, and to run a cluster in a simple configuration. Rather than quorum cluster systems, I’d opt for a fixed leader of 1024MB and a fixed follower of 512MB; of course this is still prone to single-node failure, so monitoring would be worthwhile. It’s also more expensive than all the other options, so only do this if you want to control where your data gets to.

Plus, if you’re just in testing mode, you can always set up Cron to switch off your db server compute in the small hours, squeezing out another saving.


per the valuable thread discussion, it’s worth emphasising that no types of hosting are completely resilient to disk failure.

Any more for any more?

Feel free to chip in with any of your own things y’all using alongside Fly. FWIW I’ve no financial interest in any companies listed above, just creating a resource. Maybe every time we see a sword-of-Damocles database running in Fly, we can point folks here?

beyond “bare-metal VPS” not being correct, I will note that VPS companies also suffer from host failures the same way Fly does! just this week I received an email that a VPS I use for some personal things had a hypervisor failure and lost the disk.
of course, this is why you use backups, and VPSes have other advantages to Fly Machines, but still worth pointing out.

I probably didn’t use the right terminology here. By “bare-metal VPS”, I had in mind smaller companies running VMWare etc on a machine they own and that they know the location of.

Perhaps my mental model of cloud is inaccurate (feedback welcome) but I tend to think of it as a farm of bare-metal boxes, virtualised firstly using traditional VMs, the locations of which are unknown (and regarded as unimportant except to the orchestrator). On top of that, each traditional VM is subdivided into containers or micro-VMs. I suppose I do think of that model as more brittle, fitting Fly’s approach of throwaway VMs, with clustering being recommended to make up for a resilience shortfall.

But yes, I don’t mean to suggest that old-school machines can’t fail. :+1:

the virtualisation method between Fly (I can’t speak for other cloud) and a traditional VPS is actually pretty much the same. we have a fleet of bare metal servers, which we run Fly Machines directly on top of; VPS providers do the exact same thing. now, of course, our lifecycle and network model is very different, and the advantage of a VPS is that the intention is for it to never reboot, and some providers even do live migrations between hypervisors.
notably no respectable cloud provider will do VMs inside of VMs as that is very insecure (see recent Januscape/Zapscape etc vulnerabilities)

Ooh, good information Lillian, for which thanks. Fly is running on flatter infra than I had guessed.

The traditional VPS provider I have in mind uses software RAID, and disks are hot-swapped in the case of a single drive failure. For reasons I don’t recall, I have it in mind that Fly has relied on single NVMe drives, which in my head explained why we get occasional volume corruption (i.e. there is no disk redundancy). Do I have an inaccurate picture?

I think at one point we might’ve used no-raid on workers but today we do use raid everywhere; disk failures are relatively common and in 99% of cases users will see no impact when a disk fails. (the 1% is when both disks fail at the same time, which is actually rare.)

Ah, good to hear RAID’s in use. I wonder if the single-node-Postgres fires we’ve seen here produce an unfair perception of unreliability; I should think that where something works fine, we don’t ever hear of it!

I seem to recall a data backup company once mentioned a peculiarity of single-disk redundancy; they said that they try to use mismatched brands, so that the failure times are likely to be far apart. The trick is in hoping that resilvering a new drive does not cause the one remaining good drive to keel over.