The pipeline that filled my disk: a short story about defaults

A straight red card is rare. What happens is a yellow in the 20th minute for a minor foul, then another in the 70th, equally minor. Neither deserved discussion. The accumulation sends you off.

My VPS played that match for three months. On 31 July at 15:44, it took the second yellow:

PANIC: could not write to file "pg_logical/replorigin_checkpoint.tmp":
No space left on device

Postgres stops. Four applications go down. And the cause fits in two missing characters in a configuration file I wrote myself.

The autopsy

df -h: 58 GB, 100% used, zero bytes free.

First reflex, look at logs. Good instinct, wrong target: 996 MB. Real, but nowhere near enough.

docker system df gave the answer in one second, while a du across the whole disk had been running for ten minutes and was killing my SSH session:

TYPE            TOTAL     ACTIVE    SIZE       RECLAIMABLE
Images          41        19        38.43 GB   20.49 GB (53%)
Containers      19        19        6.07 MB    0 B
Local Volumes   5         5         425.8 MB   0 B
Build Cache     28        0         8.93 GB    2.48 GB

47 GB of Docker images and cache out of 57 used. The application data itself came to 426 MB.

The detail that stings: the full dump of my Supabase database, taken right after recovery, was 1.2 MB. My entire production fitted in a file lighter than a phone photo, and it was down because of obsolete image layers.

The two characters

Here is the cleanup step in my deployment pipeline, written in good faith:

docker image prune -f

And here's what it actually does: remove dangling images — the ones with no tag left.

My pipeline tags every build:

docker build -t "$IMAGE_NAME:$CI_COMMIT_SHA" -t "$IMAGE_NAME:latest" .

Every image carries a commit SHA. None is ever dangling. The cleanup cleaned nothing.

It was missing -a:

docker image prune -af --filter "until=168h"

With -a, the command removes every image no container is using, tagged or not. The until=168h filter keeps a week, so you can roll back quickly without accumulating forever.

When I ran the corrected version: 974 MB free → 23 GB. Twenty-two images removed, including eight versions of the same app at 1.45 GB apiece.

What makes this outage interesting

The more I worked, the closer I got to the wall

That's the most perverse property. A pipeline that rarely runs would never have filled the disk. It was my delivery pace that armed the bomb. Four projects deployed regularly on the same server, each leaving an image behind.

Incidents that punish activity are the hardest to anticipate, because nothing in daily experience signals the drift.

No degradation before the stop

There was no "it's getting slow". The disk went from 97% to 98% to 99% with no perceptible effect, then hit 100% and everything stopped at once.

Threshold resources behave like this: linear until the cliff. Which is why alerting at 95% is nearly useless — by the time it fires, both your feet are over the edge. 80% is the right threshold: it leaves you time to treat the problem as a task, not an emergency.

Docker lied to itself

A troubling detail: after the outage, docker ps -a showed supabase-db | Up 2 months (unhealthy). The container was in fact stopped — docker exec answered "container is not running".

The explanation is logical once stated: the Docker daemon could no longer write its state files, for lack of space. So it displayed the last state it managed to record.

When the disk is full, your diagnostic tools lie too. A docker compose restart was enough once space was freed, but you had to first disbelieve what Docker was showing you.

Defaults decide on your behalf

That's the real subject of this incident, and it goes well beyond my server.

Three defaults combined:

1. Docker doesn't cap logs. Without max-size, a container's file grows forever. During the outage, Postgres was logging its own failure in a loop — which filled the disk, which worsened the failure.
2. prune without -a removes almost nothing. The command's name suggests a cleanup; the default behaviour is to keep everything.
3. No disk monitoring is enabled by default on a standard install.

In isolation, each default is defensible. Docker doesn't want to lose your logs. prune doesn't want to delete an image you might need. Combined on a 58 GB VPS hosting four projects, they produce a certain outage — the only unknown being the date.

We constantly inherit decisions made by other people, for contexts that are not ours. A developer who deploys on their own server becomes an operator, generally without having decided to.

The configuration I now apply everywhere

Cap the logs, in /etc/docker/daemon.json:

{ "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" } }

Thirty megabytes per container instead of infinity. Followed by systemctl restart docker.

Actually clean up, in every pipeline:

docker image prune -af --filter "until=168h"
docker builder prune -af --filter "until=168h"

Monitor content, not just status codes. My site returned 200 with an empty page for nine days. A classic uptime check saw nothing. A verification like "does the home page contain at least one project card?" would have caught it in five minutes.

Alert at 80%, and on the trend rather than the threshold alone: "the disk grew 5 GB this week" is more useful than "the disk is at 82%".

What I'd do differently from the start

With hindsight, the root mistake isn't the missing -a. It's having treated the server as an implementation detail.

I looked after my application architecture, my tests, my types. The server was configured once, in a hurry, then forgotten. It held for fifteen months — long enough for me to stop thinking about it, not long enough to hold indefinitely.

Three things I should have done on day one:

1. An INFRA.md file in the repository: what runs, where, with what limits, and what to watch. Written for the version of me who will have forgotten everything.
2. An automated backup, not a manual one. I got lucky: my data was intact. A PANIC on a checkpoint write is exactly what the WAL exists for. Next time it might be a bad sector.
3. A resource quota per project. Four applications sharing 58 GB with no quota is a race to see who fills the disk first.

None of this is hard. It's simply invisible while it works.

This outage also revealed an application-level fragility, which I describe in "My site showed zero projects for nine days". If you want an audit of what would fail silently in your setup, write to me.