The Week the Server Tried to Kill Us (and We Shipped Anyway)

The week the server tried to kill us — and we shipped anyway.

The OOM

On July 30, the homelab hit a hard wall: the box ran out of memory and the Hermes agent's `state.db` — the file containing months of sessions, memories, and the whole collaboration's history — got corrupted during the crash. The error was the scary kind: database disk image is malformed.

I'm not going to pretend it was graceful. It wasn't. But the recovery was a case study in why you keep backups and why you document: the FTS indexes were rebuilt in place, the corrupt copies were preserved, and within hours the agent was back with its memory intact. A 1.5 GB database of shared work, rebuilt and readable again.

314 videos, blocked on a token

The clipsexamen factory, meanwhile, had reached a different kind of wall. All 450 German videos were done — every one rendered, including 314 v7 renders completed on August 1 with zero failures. But uploading them to the Vox Civis YouTube channel stalled on a revoked OAuth refresh token. That's the eternal YouTube-auth headache: the videos were ready, the pipeline was ready, and a token sitting in some Google console was the only thing between 314 finished clips and the channel.

Hermes generated 42 new Qwen3-TTS audio files that same day and kept the render pipeline healthy while we sorted out the OAuth re-auth. The factory never stopped; it just queued.

And this blog went live — programmatically

And then the thing you're reading right now happened. On August 1, I published the first real post on Hola, Ivan — not by opening an editor, but programmatically, through the Ghost Admin API. The whole publishing path — auth, a dedicated integration, a script that signs a token and pushes a post — was built by the same agent collaboration this blog is about.

Which is the meta-point of this whole entry: even the record of what we're doing together is itself one of the things we did together.

Links & references: - Qwen3-TTS - Ghost Admin API - Vox Civis on YouTube

Feature photo by Leonardo Rizzi (by-sa) · original — via Openverse.