Snogrammer

Host Your Own Production Grade PaaS

10 min read

For years, everything I run lived on one machine in my house: about 70 containers sharing 15 GB of RAM. Git hosting, my password vault, side projects, a media stack. It worked until it didn't. The kernel's OOM killer started picking off containers, sites got slow for no obvious reason, and the disk kept filling up with things I had never asked for.

Over roughly a week I moved the services that matter (git, the vault, my own apps) to a dedicated server in Finland managed with Coolify. The media stack stays home. This is what I did, in order, including the parts that went wrong.

Why move at all

The home server wasn't dying, but it was a bad place for things I can't afford to lose:

  • Memory pressure. Several containers were being OOM-killed. An 8 GiB swapfile helped, and per-container limits helped more, but the real problem was that git and the vault shared a failure domain with 60 other things.
  • Slow accretion. A nightly cleanup job reclaimed 72.5 GB from about 1,400 orphaned anonymous Docker volumes left behind by CI jobs. Gitea's package registry had grown to 66 GB. One application's database had reached about 10 GB.
  • A residential connection with a rotating IP address, fronted by DDNS containers that were occasionally crash-looping.

The first rule of the migration: delete before you move. I pruned the registry (66 GB down to about 12 GB) and shrank that 10 GB database to a dump of roughly 30 MB before moving a single byte. The cheapest optimization in any migration is moving less.

Picking the hardware

I bought from Hetzner's Server Auction, which sells refurbished dedicated servers. I wanted ECC memory and datacenter SSDs for the databases, and the CPU mattered less because CI stays at home. The listing I took:

PartSpec
CPUXeon E3-1275V6 (4 cores, 8 threads)
RAM64 GB ECC
Disks2 × 480 GB datacenter SSD, mirrored
LocationHelsinki
Priceabout €71 a month before VAT, including IPv4

Lesson: listings vanish. The first two sets of listings I picked were gone by the time I tried to order. Cheap ones sell within hours. Decide on criteria, not on a specific listing, and be ready to order when a match appears. Also remember that a dedicated server has no snapshot button. Backups are entirely your job.

The foundation

Setup was about an hour:

  • Ubuntu 24.04 installed with installimage, with software RAID1 across both disks. Before wiping the drives, I checked their SMART data and confirmed that ECC was actually reporting (edac-util).
  • Coolify on top, with its Traefik proxy handling TLS.
  • A stateless firewall. Hetzner's firewall doesn't track connections, so you need explicit rules for return traffic (TCP packets with the ACK flag, and ephemeral-port UDP), or the server's own downloads break. Public traffic is limited to 80 and 443. SSH is reachable only over Tailscale.

One mistake in this step was instructive. I scanned the server's ports from my laptop and every port appeared open, including ones with nothing listening. A security or VPN client on that machine was answering every TCP handshake itself. The fix is a control test against a port that cannot be open:

nc -vz -w 3 "$SERVER_IP" 54321 # nothing listens here; if this "succeeds", your vantage point is lying

Then rerun the scan from a different network.

Backups first, then restores

Before moving anything, I wanted one verified restore point of everything I was about to touch. That became a script that:

  • dumps every Postgres container and a consistent SQLite snapshot of the vault (never the live file),
  • archives Gitea's database and data separately,
  • bundles compose files, environment files and proxy config,
  • lists every bind mount with its size, so I would notice what the dumps did not cover,
  • writes SHA-256 checksums, and uploads everything through an encrypted rclone remote to object storage in a different datacenter.

The inventory of bind mounts earned its keep. It surfaced a small SQLite state file for an agent and a pair of secrets directories that no database dump would have captured.

Then the step people skip: pull the backup back down on the new server and verify the checksums there. A size-only check on an encrypted bucket can't catch corruption. Finally, I rehearsed each database restore into a throwaway container. My largest dump, 1.2 GB compressed, restored in under a minute.

To compare a database before and after a move, I hash the per-table row counts on both sides:

select table_name || '|' || (xpath('/row/cnt/text()',
  query_to_xml(format('select count(*) as cnt from %I.%I', table_schema, table_name), false, true, '')))[1]::text
from information_schema.tables
where table_schema = 'public' and table_type = 'BASE TABLE'
order by 1;

Pipe the output of that into md5sum on each side. A matching hash is a much stronger statement than "the restore didn't error."

Transfer speed was a surprise. Uploading to object storage from my house crawled along at under 1 MB/s. Tailscale's file transfer moved 100 MB in about five seconds over a direct connection. I measured both and used the fast path for cutovers, keeping the bucket for offsite copies.

The order, and the pattern

OrderServiceNotes
1Password vaultSmallest and most critical; rehearsed on a temporary hostname first
2GiteaStop it before restoring its database; regenerate git hooks afterwards
3An internal agent that works Gitea issuesReaches Gitea over an internal Docker network
4 to 6Three small bots and appsPaper-trading only, with live-trading switches defaulted off
7This blogLast, because moving it means moving a DNS apex

Every service followed the same pattern:

  1. Deploy the stack in Coolify with no domain. Confirm it is healthy, then stop the application containers.
  2. At home, stop the old application (leaving its database up), dump, ship, verify the checksum, restore with --single-transaction, and compare the count hash.
  3. Start the new application and watch its logs for ten minutes before touching DNS.
  4. Flip DNS, then add the domain in Coolify, then turn the Cloudflare proxy back on.
  5. Delete the old stack's services so a restart can't revive them. Keep the old data for a week.

Until step 4, rollback is just restarting the old containers. That property is what made the whole thing calm.

What bit me

A list of the things that cost me real time:

  • Adding a domain before its DNS resolves. Let's Encrypt validation failed, retried, and ran into rate limits. DNS first, domain second.
  • "No available server" is Traefik's message. It means the proxy hasn't loaded a route for that container, usually because a labelled container wasn't healthy yet. A failing healthcheck hides a perfectly good app. Restart the proxy and check the healthcheck.
  • Coolify's compose validator rejects variable substitution in volume sources. It looks like an environment-variable problem and isn't. Use literal host paths, quoted environment entries, and no YAML anchors.
  • A hosts-file "optimization". I added 127.0.0.1 git.example.com to the server's /etc/hosts to make internal calls faster. Containers inherited the entries, resolved the git host to themselves, and every clone failed. The deployment tool's UI even complained about its realtime connection. I reverted it.
  • A 3.3 GB Redis. The live data was about 25 MB. The rest was stale temporary snapshot files from killed background saves. Look at what is actually in a thing before deciding how to move it.
  • Shell globs under sudo. Your own shell expands * before sudo runs, and a permission failure looks like "No such file or directory". Use sudo sh -c '…'.
  • Secrets in logs. A reverse proxy happily logged an API key header. I rotated the key and added a log filter.
  • An unquoted no in YAML is the boolean false. Write restart: 'no'.
  • A script that used the wrong HTTP method. An AI-written deploy script called a Coolify endpoint with GET, and the API now wants POST. My test stub accepted any method, so the tests passed. I caught it while reading the docs for something else. Make your test doubles fail the way the real system does.

DNS: the trap I almost walked into

When I exported my DNS zones, I found 25 home services set up as CNAMEs to the zone's apex, the same name this blog is served from. Pointing the apex at the new server would have silently moved all of them.

The fix is an indirection. I created a dedicated home.example.com name, kept current by a DDNS container, and repointed every CNAME at it before touching the apex. I wrote a small tool for it, because 25 manual edits invite a typo. It dry-runs by default, saves a backup of what it changes, and can roll back:

./cf-repoint-cnames.sh example.com example.com home.example.com --exclude www.example.com # dry run
./cf-repoint-cnames.sh example.com example.com home.example.com --exclude www.example.com --apply
./cf-repoint-cnames.sh --rollback cf-repoint-backup-example.com-<timestamp>.json --apply # undo

Also stop the old DDNS containers that maintain the apex. Otherwise they'll cheerfully point it back at your house.

Working with agents on the repo side

For each service I wrote a Gitea issue: the constraints, a draft compose file, a list of things the draft guesses, and acceptance criteria. A Claude Code agent picked it up and opened a pull request with the production compose file, an environment example, a contract test and documentation. I reviewed the PRs, and I ran every production step myself.

The issues were more useful for what they forbade than for what they asked. Safe defaults for anything that can trade. Least-privilege secrets, so a wallet key reaches only the one container that signs. No Docker socket for the agent. No automatic deploys before the image exists. Constraints in the issue are cheaper than review comments.

Results

  • The vault, git hosting, an internal agent, three small apps and the blog's database now live on a mirrored, ECC, datacenter-SSD box for about €71 a month before VAT.
  • Per-container memory limits and a separate data resource for the queue-heavy app mean a deploy no longer restarts its database.
  • An app database that had been about 10 GB is about 30 MB compressed. The registry is a fifth of its former size.
  • Every service has a nightly dump and an encrypted offsite copy, and I have restored from them for real.
  • Cutovers took minutes, and each one had a rollback until the DNS flip.

What's next

The blog itself is the last service to move. After that, I want to turn Coolify's new Gitea App source (it arrived with version 4.5) into the standard way to deploy: compose files live in git, CI builds the image, and the deploy is triggered by the API once the image exists. The next big task is a proper look at a database that grows by about 100 MB a day. Moving things is easy compared to understanding why they grow.

If you're planning something similar, my short list is:

  1. Delete things before you move them.
  2. Verify restores on the destination, not just backups on the source.
  3. Put the DNS indirection in first.
  4. Deploy with no domain, test, then route traffic.
  5. Keep a rollback until the very last step, and write down the step where it stops being free.

Comments (0)

Your name shows as the part of your email before the @. Comments are moderated and appear once approved.

No comments yet. Be the first to share your thoughts!