Three separate fires this week 🔥 Each one looking contained at first glance and spreading the moment I looked closer. None of them were dramatic in isolation, but together they exposed real gaps in my monitoring, alerting, and defensive coding habits. I'm writing this less as a postmortem and more as a record of what to do differently next time, because all three of these were preventable.
The Worker That Kept Dying
A worker in one of my side projects started restarting every few minutes. By the time I noticed, it had restarted nearly 800 times. The obvious suspects looked fine: memory usage was well under the container limit, the OOM flag was false, and the exit code reported clean. That last part was the trap. Those fields describe the current running container, not the one that just died.
The actual problem was an archive export job that loaded entire JSONB payloads into memory all at once. The October data partition had just closed, and it happened to include some very large backfill pages. The process was quietly eating gigabytes per run. The kernel killed it, the container restarted, and the job fired again 35 seconds later. Repeat 800 times.
The fix was streaming rows in small batches and passing JSONB columns as text so Python never decoded them into dicts. Memory per batch went from "whatever fits in RAM" to something predictable and small. I also moved the export into a child process, so a future overrun kills the child instead of the whole container and surfaces as a job failure rather than a silent restart. When you have a job that processes variable-size data, isolate it so the blast radius is scoped to that job, not the whole process.
flowchart TD
A[Worker Process] --> B[Spawn Child Process]
B --> C[Export Job: Stream Rows in Batches]
C --> D{Memory OK?}
D -- Yes --> E[Continue Batch]
E --> C
D -- No --> F[Child Process Killed]
F --> G[Job Failure — Surfaced in Monitoring]
G --> H[Worker Process Survives]
The restarts exposed a second bug: every time the worker came back up, it lost its in-memory state for quality monitoring, so every open alert refired. One issue had accumulated nearly 750 duplicate events before I found it. That pointed to a deeper problem with how the worker handled shutdown signals. Uvicorn, since version 0.29, replaces the SIGTERM handler with its own and re-raises on exit, so the worker was never getting a clean stop. Once that was fixed, shutdowns actually cleaned up state before exiting. Test shutdown behavior explicitly, not just startup. A process that starts cleanly but dies messily will cause subtle state corruption that only shows up under restart pressure.
There was also a scheduling quirk worth noting: APScheduler fires interval jobs one full interval after startup, not immediately. With a 24-hour interval and several deploys per day, the archive export almost never ran at all. Setting a short initial delay fixed that, but the more important fix is having visibility into whether scheduled jobs are actually running. A job that silently skips is indistinguishable from a job that runs successfully if you're not tracking execution events.
The prevention playbook here is straightforward: track restart counts on your containers and alert when they cross a threshold, not just when a container is currently down. Log scheduled job execution and alert on missed runs. Treat memory-intensive batch work as inherently risky unless it's either streamed or isolated.
The Cascading Data Problem
The worker restarts had been hiding a data integrity problem that had been building for weeks.
Stock tickers get reused. When a company renames, it often gives up a ticker that another company picks up later. My securities master has logic to track these renames, but it ran into a case where the rename record and the SEC's current ticker file pointed to different companies for the same ticker. The rename logic matched the wrong one.
The result was a chain reaction. A metadata feed updated the ticker. A FIGI lookup updated the security type. Price feeds started pulling data for the wrong company. Historical bars before the rename were still correct, but everything after was wrong, quietly, with no alert. The same pattern showed up across 187 securities.
flowchart LR
A[Rename Record Arrives] --> B{Continuity Check}
B -- Pass --> C[Apply Rename]
B -- Fail --> D[Reject + Alert]
C --> E[Metadata Feed Updates]
E --> F[Identifier Lookup Updates]
F --> G[Price Feed Targets Wrong Company]
G --> H[Silent Bad Data in DB]
style D fill:#f96,stroke:#c33
style H fill:#f96,stroke:#c33
The guard I added checks whether a security has held a ticker continuously since before the rename date. If a source is reporting the rename years after the event while the security never actually changed tickers in the data, that's a signal to reject it. Running cleanup against all 187 suspects restored old tickers, deleted the false ones, reverted any downstream identifier changes, and removed bars ingested during the bad window.
That mostly worked, but the existing healing mechanism had created a different problem. When the system saw a ticker it thought belonged to a new issuer, it created a new security record instead of correcting the old one. So some securities ended up with twins: one with the correct history and one with bars from the contaminated window. A merge step found pairs sharing both an open ticker and an open company identifier, kept the older record, repointed every foreign key, and collapsed duplicates.
The lesson is that automated healing logic needs its own guard rails. Any process that creates new records as a recovery action should check whether a valid record already exists before creating another one. Silent duplication is harder to detect than a loud failure. Cross-referencing identifiers against their expected continuity dates before applying any rename is the kind of validation that should have been in the original logic.
API Retirement and Data Research
Partway through the week I noticed three features in another side project had silently degraded. The cause was straightforward: Polymarket retired their Data API v1, and every old route started returning a 410. The strategies were written to tolerate failures gracefully, which meant they kept running on neutral fallback values instead of real data. Nothing crashed, nothing alerted, and the degradation went unnoticed for days.
This is the defensive coding trap. Graceful degradation is good until it makes failures invisible. A 410 Gone is not a transient error worth retrying through, it's a permanent signal that something has changed and a human needs to know. The right behavior on a 410 is to alert immediately, not fall back silently. I've updated the error handling to treat 4xx responses from external APIs as paging events rather than soft failures, so they surface in monitoring instead of disappearing into fallback logic.
flowchart TD
A[External API Call] --> B{Response Code}
B -- 2xx --> C[Process Data Normally]
B -- 5xx / Network Error --> D[Retry with Backoff]
B -- 429 --> E[Respect Rate Limit, Retry]
B -- 410 Gone --> F[Alert Immediately — Do Not Retry]
B -- Other 4xx --> F
F --> G[Human Reviews + Updates Integration]
The v2 migration itself wasn't bad, but there was one non-obvious behavior worth documenting: v2 silently ignores unknown query parameters. If you pass a cursor without also resending the original filters, you get the global feed instead of your filtered one. I pinned the exact query strings in tests to make sure any regression there is immediately visible. When an API can silently change the scope of your query based on parameter combinations, the only real defense is testing the actual response shape, not just the status code.
With v2 working, I shifted to building a copy-trading strategy. The idea is to follow wallets with demonstrated edge on prediction markets. Building the trader census required reconstructing win rates from scratch because the platform's leaderboards don't expose them directly. Losing positions are never redeemed, so a simple redemption count only ever sees winners, and historical boards don't exist. Win rates have to be computed from settled positions, keyed on settlement date rather than last activity. That matters because a losing position's last event might be a buy from months ago, which would make a losing trader look quiet rather than wrong.
The admission gate also requires positive realized PnL in the category being copied, not just good-looking board numbers. A spot check found one wallet whose leaderboard stats looked reasonable but had lost a substantial amount across its actual settled markets over 90 days.
The backtest ran the pre-registered decision rule against 90 days of cached data. The copies themselves are profitable, and the copied side wins more often than its price implies across every price band. The data initially looks like a positive step forward for building a profitable system, but I will be back-testing and optimizing strategies as I continue to ingest and analyze the data.

Every day I am continuing to scale and harden my production systems and it's feeling very motivating to see my projects mature along the way. Even if I am spending cycles debugging and fixing production issues that takes away from my feature building, I am continuously learning and building my AI knowledge base and building resilient systems along the way.
Fail Fast. Keep Vibin' 🤘