On 31 July at 15:44, my server wrote this line:
PANIC: could not write to file "pg_logical/replorigin_checkpoint.tmp":
No space left on device
On 1 August, my portfolio proudly displayed: 0 apps shipped · 0 roles · No commits for this filter.
Nine days between the two. Nine days during which anyone landing on my professional site saw an empty page. No alert, no email, no error page. Just a perfectly functional site that had stopped saying anything about me.
The silent outage is the worst kind
A 500 is brutal but honest. You see it, you fix it.
My code did this:
const { data, error } = await supabase.from("projects").select("*");
if (error) {
console.error("listProjects", error);
return { projects: [] }; // ← the crime is here
}
On the face of it, that's good practice: handle the error, don't crash the page. In reality, we turned an infrastructure failure into a valid business result. The component receives an empty array and dutifully does its job: it renders "no projects".
The console.error goes into the logs of a container nobody reads. The page returns 200. An uptime check would be green.
That's the trap of the silent fallback: an empty array is indistinguishable from a legitimately empty array. Nothing in the type says "this is a failure".
The investigation
From the outside, the symptoms didn't add up:
/auth/v1/health → 200 GoTrue in perfect health
/storage/v1/status → 200 storage fine
/rest/v1/** → 503 PGRST002
Two services out of three responding. Only PostgREST failing, with an unusual message: "Could not query the database for the schema cache. Retrying."
The container logs gave the answer in one line. Disk full. Postgres can no longer write its checkpoint, it panics, it restarts in recovery, it panics again. Infinite loop.
The nastiest part: PostgREST itself was running. It accepted connections, answered politely, even explained that it was retrying. A service that correctly reports that it's broken is harder to diagnose than a dead one.
The culprit had been in my CI all along
The disk was 58 GB. docker system df delivered the verdict in a second:
Images 41 total, 19 active 38.43 GB ← 20.49 GB reclaimable
Build Cache 28 entries, 0 active 8.93 GB
47 GB of Docker images out of 57 used.
Where from? My own deployment pipeline. Each build tags the image with the commit SHA:
docker build -t "$IMAGE_NAME:$CI_COMMIT_SHA" -t "$IMAGE_NAME:latest" .
And at the end, the cleanup:
docker image prune -f
Without -a. That missing flag changes everything: prune -f only removes "dangling" images, the ones with no tag left. Mine all have one — the commit SHA. So they were never removed.
One image per commit. Four projects deployed on the same server. Three months. I had eight versions of the same app at 1.45 GB each.
The mechanism is almost elegant in its perversity: the more I worked, the more I filled the disk. A quiet pipeline would have been harmless; it was my delivery pace that armed the bomb.
Seven days of accumulation, and a red card
In football you don't get a straight red. You get a yellow in the 20th minute for something trivial, another in the 70th, equally trivial. Neither mattered. The accumulation did.
My disk behaved the same way. Each deployment added a few hundred megabytes — invisible, painless. Then one day the percentage goes from 99 to 100 and everything stops at once.
Systems that degrade linearly then break abruptly are the most dangerous, because daily experience prepares you for nothing. There was no "it's getting slow" before the "it doesn't work".
What I changed
1. Content no longer depends on the database
That's the substantive change. This site's career timeline and case studies now live in the repository, in TypeScript, versioned with the code.
Reads still query the database first, and fall back to that content if it doesn't answer:
try {
const { data, error } = await supabase.from("projects").select(...);
if (error) throw new Error(error.message);
if (data?.length) return { projects: data };
} catch (err) {
console.error("[listProjects] database unavailable, falling back to local:", err);
}
return { projects: localProjectRows(lang) };
Two differences from the old code, both essential:
- We
throwon error instead of returning an empty array. A failure stays a failure. - We test
data?.length: an empty response is treated as a non-response, not as truth.
Tested against the genuinely broken database: the site renders all of its content. The outage becomes invisible to visitors and loud in the logs. That's the right way round.
2. Critical content has no business being in a database
Here's the uncomfortable lesson. My career history changes three times a year. My case studies, twice. Why on earth were they in a database queried on every request?
Because an admin dashboard is satisfying to build. Because "data goes in the database" is the reflex. Nobody asked whether the near-static content of a brochure site deserved a network dependency on every render.
The criterion I now apply: if content changes less often than you deploy, it belongs in the repository. A commit is a perfectly valid publishing mechanism — with code review, history and free rollback.
3. The pipeline actually cleans up
docker image prune -af --filter "until=168h"
docker builder prune -af --filter "until=168h"
The -a does the work. The filter keeps a week of history so you can roll back quickly.
And the log cap, which didn't exist either:
{ "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" } }
By default, Docker lets a container's logs grow without any limit. My Supabase containers had accumulated nearly a gigabyte, much of it written during the outage — Postgres logging its own failure in a loop, which filled the disk, which worsened the failure.
Defaults are not neutral
That's the real subject, and it goes beyond my incident.
Docker chooses unlimited logs. prune without -a chooses to keep everything. Supabase assumes it's always reachable. Each default is reasonable in isolation — and catastrophic once combined on a 58 GB VPS.
We inherit dozens of decisions made by other people, for contexts that are not ours. A developer who deploys on their own server takes on an operator's role, usually without noticing.
What I now do on any infrastructure I own:
- Alert at 80% disk, not 95%. At 95% you're repairing under pressure; at 80% you're planning.
- Monitor content, not status codes. "Does the home page contain at least one project card?" would have caught my outage in five minutes. A plain uptime check missed it for nine days.
- A backup before touching anything. I took a full dump before making a single change. It was 1.2 MB. My site was down because of 47 GB of Docker images, while all my data fitted in a file lighter than a photo.
That last figure made me think harder than anything else.
The fallback described here is live on this site — details on its project page. If you want an audit of what would fail silently in your own setup, write to me.