Agents On Call: DNS Races, Feature Files, and the AI-Assisted Postmortem

Agents On Call (Jul 2025 – Aug 2026) This window opened with a dormant config error and got worse from there. AWS, Azure, and Cloudflare each had a headline global outage within a single month of autumn 2025, and “the internet is three companies in a trench coat” became a mainstream joke. A year later, July 2026 delivered four provider incidents in nine days. Underneath all of it, the biggest shift since the SRE book has been quietly happening: AI agents moving from summarizing incidents to actually responding to them. ...

August 6, 2026 · July 2025 – August 2026 · Retrospective · living document — updated August 6, 2026

Patch Notes #331 — Four Providers, Nine Days

I said last time that the world’s ledger was resuming its ordinary cadence. The world heard me and put four cloud providers on the board in nine days. The run, in order. July 16: AWS CloudFront couldn’t load updated network config into the fleet handling private VPC origins, and three and a half hours of 5xx errors landed on everyone downstream — Hugging Face, Canvas, Blackboard, a long tail of SaaS whose customers had no idea they were CloudFront customers. July 20: a power disturbance in a Google Cloud zone in the Netherlands defeated the backup power and the cooling, and the machines shut themselves down rather than cook. Fifteen hours. July 23: an Azure maintenance process in West US removed more network routes than it meant to — five hours where the workloads were fine and reaching them wasn’t. July 24: Oregon lost internet connectivity for eighty minutes and took Apple Pay, Reddit, Hulu, DoorDash and PlayStation Network with it. That last one was AWS’s third distinct incident in eleven weeks, which is the part the trade press led with, correctly. ...

August 5, 2026

Patch Notes #330 — The Morning After the Continent

The World Cup final was Sunday night. It’s Tuesday, the wall chart is still up, and I’m going to do something I’ve been saving for four World Cups: I’m not naming the champion. Not to be cute. You know who won. The whole planet knows who won. That’s exactly why this archive doesn’t need it. What the record books won’t keep is what I want to keep: the viewing room at 4am IST for the final. Eleven flags. Three adopted underdog nations’ worth of heartbreak folded into one match. The CFO’s “pick whoever the German engineer fears most” pool entry finishing where fear-based strategies always finish — second. ...

July 21, 2026

Patch Notes #329 — The Sharp End (Entry 329 of ∞)

Filing from the tournament’s sharp end, exactly as promised. The round of 16 has the continent by the throat. The knockout concurrency peaks arrived on schedule, and my last infrastructure prediction graded correct in both halves: the stack held, and the apology-discourse found its platform during a penalty-shootout traffic spike — somewhere, someone else’s incident channel is drafting the postmortem genre’s soccer chapter as I type. A debutant nation reached the round of 16 and declared a spontaneous national holiday. The German engineer’s posture — cautious, wounded, eternal — has survived to the sharp end intact, which the group chat’s constitution now recognizes as its own kind of championship. Quarterfinals load Thursday; the algorithm has pre-cleared every slot. ...

July 6, 2026

Patch Notes #328 — Four Matches a Day

The prophecy fulfilled itself precisely: this entry is written by a person watching four matches a day. Format verdict, mid-tournament: the pre-registered split is holding. The dead rubbers exist and nobody cares, because the debutant joy engine is running at full capacity. The office’s adopted second-teams span three continents. The viewing room has learned four new anthems well enough to hum. And the daily texture — someone’s cousin’s homeland in its first World Cup, weeping captains at anthems, upset-watch notifications at 10am — is the format’s entire case. Closed. ...

June 21, 2026

Patch Notes #327 — Eve of the Continental Cup

Five days. The World Cup opens Thursday at Estadio Azteca in Mexico City — 1970, 1986, 2026, the only stadium ever to host three, a legacy system with the sport’s best uptime story. Eve entries are for predictions, so here they are, on the record for grading: One: the streaming stack holds through the group stage and strains at the knockout rounds’ single-match concurrency peaks, with an apology-thread by the round of 16. Two: the 48-team format’s fears — diluted groups, dead rubbers — prove half-true and get forgiven anyway, because a dozen nations making their first-ever appearance outweighs the purists’ math. (I am a purist. I’m pre-registering my own forgiveness.) Three: the provenance layer holds at the institutional tier and fails picturesquely at the playful tier — some AI-generated goal celebration fools a news desk by the quarterfinals. Four: the pool is won by whoever weighted chaos correctly. It’s always whoever weighted chaos correctly. ...

June 6, 2026

Patch Notes #326 — Twenty Days: The Continent Prepares

Twenty days to the World Cup, and this fortnight’s entry is about the tournament as the infrastructure story it genuinely is. Forty-eight teams — the biggest expansion in the format’s history. A hundred and four matches, up from sixty-four. Sixteen cities across three nations and four timezones, which makes the Winter Games’ Alpine arc look like a campus. The operational ledgers, by trade: the streaming plan is the largest scheduled-traffic test the civilian internet has ever run — simultaneous group-stage kickoffs plus a potential home-nation run would stack concurrent load that no Champions League final approaches. On the record, per tradition: the rehearsed platforms hold, the apology-discourse finds someone anyway, and the postmortem genre gains a soccer chapter. The provenance layer faces its biggest stress test too — a tournament’s worth of viral moments, every one a deepfake target. And the visa-and-borders discourse is warming up: three nations, one tournament, and the friction between “global event” and “border regime” producing a steady drumbeat of team-logistics stories. Every global system’s seams show at its celebrations. ...

May 22, 2026

Patch Notes #325 — The Anniversary Retrospectives and the Unresolved Question

The six-month anniversary retrospectives of the outage autumn landed this fortnight — the long-form reconstructions of how businesses on three continents survived the AWS-Azure-Cloudflare trilogy, or didn’t. The genre has grown up. The best pieces document Nairobi payment processors and Jakarta logistics firms rerouting around us-east-1’s bad Monday: cloud concentration as a development-economics story. The regulatory count went from three jurisdictions to four, with the postmortems still serving as committee exhibits. And the datapoint I find most satisfying: multi-provider degraded-mode programs like ours are now common enough to have survey categories and consultant practices. The incident wrote the industry’s homework. ...

May 7, 2026

Patch Notes #324 — Earth Day for the Loom

Earth Day fortnight, and the datacenter-energy question is now the AI buildout’s binding public negotiation. Grid-interconnection queues are still the limiting reagent, three years running. This year’s addition is utility-rate politics: “are residential bills subsidizing hyperscale load” is the recurring local-news investigation, and rate shocks write regulation the same way incidents do. Nuclear’s renaissance proceeds from announcement era to construction era at nuclear’s eternal cadence — the physics is sound, the timelines are the product, and the bridge decade belongs to efficiency and storage. The counter-ledger, because both hands: inference efficiency keeps improving faster than the discourse tracks, so the net is still growing but the composition is shifting the way every infrastructure boom shifts. The boom builds overcapacity, the overcapacity gets priced, the efficiency inherits it. ...

April 22, 2026

Patch Notes #323 — Thirteen Years Since the Restore Drill

Thirteen years ago this month, entry #007 recorded a four-hour backup-restore drill and a tech lead’s proverb that started my whole file: “a backup you haven’t restored is a rumor.” The anniversary fortnight delivered a perfect echo. Our quarterly restore drill ran against the newest layer of the stack for the first time — not just databases but the model portfolio’s routing state, eval baselines, and agent intent docs, all from cold storage. The finding was pure #007 lineage: the golden test sets restored perfectly, and the routing config’s restore procedure referenced a secrets path that had been migrated in November. The rumor was in the plumbing’s plumbing, where it always is. Fixed by Tuesday, drill re-run clean by Friday. Thirteen years, four title changes, one unbroken discipline: restore-test everything, especially the things that restore the other things. ...

April 7, 2026