Agents On Call: DNS Races, Feature Files, and the AI-Assisted Postmortem

Agents On Call (Jul 2025 – Jul 2026) This window opened with a brutal autumn. AWS, Azure, and Cloudflare each had a headline global outage within a single month, and “the internet is three companies in a trench coat” became a mainstream joke. Underneath that, the biggest shift since the SRE book has been quietly happening: AI agents moving from summarizing incidents to actually responding to them. The incidents defining the period (so far) AWS us-east-1, October 20, 2025. A latent race condition in DynamoDB’s automated DNS management produced an empty DNS record for the regional endpoint. The automation couldn’t fix itself, and the failure cascaded through the many AWS services (and thousands of customer apps) that depend on DynamoDB in us-east-1. Roughly 14–15 hours of disruption, and Snapchat alone drew about 3 million outage reports. The biggest us-east-1 event since December 2021, and a case study in automation deadlock: the fix meant humans disabling the automation that was supposed to prevent exactly this. Azure Front Door, October 29, 2025. An accidental configuration change broke Microsoft’s global edge and CDN layer for about eight hours, taking down the Azure portal, M365 entry points, and customer sites, days before earnings and a week after AWS’s turn. A separate East US2 networking config outage lasting about 50 hours showed that regional incidents can now outlast the news cycle. Cloudflare, November 18, 2025. A database permissions change caused the Bot Management feature file to double in size and blow past a hard-coded limit in the core proxy, and processes crash-looped globally. X, ChatGPT, and Canva threw 5xx errors for hours. Cloudflare’s same-week postmortem (blog.cloudflare.com) echoed their 2019 regex writeup: an internally generated file, pushed globally, hitting an untested limit. Cloudflare, December 5, 2025 and February 20, 2026. A roughly 25-minute traffic outage, then a BGP withdrawal affecting bring-your-own-IP customers. Smaller events, but notable for how fast and detailed the disclosures now are. (This is a living post, updated through July 2026.) ...

July 21, 2026 · July 2025 – July 2026 · Retrospective · living document — updated through July 2026

Patch Notes #330 — The Morning After the Continent

The World Cup final was Sunday night. It’s Tuesday, the wall chart is still up, and I’m going to do something I’ve been saving for four World Cups: I’m not naming the champion. Not to be cute. You know who won. The whole planet knows who won. That’s exactly why this archive doesn’t need it. What the record books won’t keep is what I want to keep: the viewing room at 4am IST for the final. Eleven flags. Three adopted underdog nations’ worth of heartbreak folded into one match. The CFO’s “pick whoever the German engineer fears most” pool entry finishing where fear-based strategies always finish — second. ...

July 21, 2026

Patch Notes #329 — The Sharp End (Entry 329 of ∞)

Filing from the tournament’s sharp end, exactly as promised. The round of 16 has the continent by the throat. The knockout concurrency peaks arrived on schedule, and my last infrastructure prediction graded correct in both halves: the stack held, and the apology-discourse found its platform during a penalty-shootout traffic spike — somewhere, someone else’s incident channel is drafting the postmortem genre’s soccer chapter as I type. A debutant nation reached the round of 16 and declared a spontaneous national holiday. The German engineer’s posture — cautious, wounded, eternal — has survived to the sharp end intact, which the group chat’s constitution now recognizes as its own kind of championship. Quarterfinals load Thursday; the algorithm has pre-cleared every slot. ...

July 6, 2026

Patch Notes #328 — Four Matches a Day

The prophecy fulfilled itself precisely: this entry is written by a person watching four matches a day. Format verdict, mid-tournament: the pre-registered split is holding. The dead rubbers exist and nobody cares, because the debutant joy engine is running at full capacity. The office’s adopted second-teams span three continents. The viewing room has learned four new anthems well enough to hum. And the daily texture — someone’s cousin’s homeland in its first World Cup, weeping captains at anthems, upset-watch notifications at 10am — is the format’s entire case. Closed. ...

June 21, 2026

Patch Notes #327 — Eve of the Continental Cup

Five days. The World Cup opens Thursday at Estadio Azteca in Mexico City — 1970, 1986, 2026, the only stadium ever to host three, a legacy system with the sport’s best uptime story. Eve entries are for predictions, so here they are, on the record for grading: One: the streaming stack holds through the group stage and strains at the knockout rounds’ single-match concurrency peaks, with an apology-thread by the round of 16. Two: the 48-team format’s fears — diluted groups, dead rubbers — prove half-true and get forgiven anyway, because a dozen nations making their first-ever appearance outweighs the purists’ math. (I am a purist. I’m pre-registering my own forgiveness.) Three: the provenance layer holds at the institutional tier and fails picturesquely at the playful tier — some AI-generated goal celebration fools a news desk by the quarterfinals. Four: the pool is won by whoever weighted chaos correctly. It’s always whoever weighted chaos correctly. ...

June 6, 2026

Patch Notes #326 — Twenty Days: The Continent Prepares

Twenty days to the World Cup, and this fortnight’s entry is about the tournament as the infrastructure story it genuinely is. Forty-eight teams — the biggest expansion in the format’s history. A hundred and four matches, up from sixty-four. Sixteen cities across three nations and four timezones, which makes the Winter Games’ Alpine arc look like a campus. The operational ledgers, by trade: the streaming plan is the largest scheduled-traffic test the civilian internet has ever run — simultaneous group-stage kickoffs plus a potential home-nation run would stack concurrent load that no Champions League final approaches. On the record, per tradition: the rehearsed platforms hold, the apology-discourse finds someone anyway, and the postmortem genre gains a soccer chapter. The provenance layer faces its biggest stress test too — a tournament’s worth of viral moments, every one a deepfake target. And the visa-and-borders discourse is warming up: three nations, one tournament, and the friction between “global event” and “border regime” producing a steady drumbeat of team-logistics stories. Every global system’s seams show at its celebrations. ...

May 22, 2026

Patch Notes #325 — The Anniversary Retrospectives and the Unresolved Question

The six-month anniversary retrospectives of the outage autumn landed this fortnight — the long-form reconstructions of how businesses on three continents survived the AWS-Azure-Cloudflare trilogy, or didn’t. The genre has grown up. The best pieces document Nairobi payment processors and Jakarta logistics firms rerouting around us-east-1’s bad Monday: cloud concentration as a development-economics story. The regulatory count went from three jurisdictions to four, with the postmortems still serving as committee exhibits. And the datapoint I find most satisfying: multi-provider degraded-mode programs like ours are now common enough to have survey categories and consultant practices. The incident wrote the industry’s homework. ...

May 7, 2026

Patch Notes #324 — Earth Day for the Loom

Earth Day fortnight, and the datacenter-energy question is now the AI buildout’s binding public negotiation. Grid-interconnection queues are still the limiting reagent, three years running. This year’s addition is utility-rate politics: “are residential bills subsidizing hyperscale load” is the recurring local-news investigation, and rate shocks write regulation the same way incidents do. Nuclear’s renaissance proceeds from announcement era to construction era at nuclear’s eternal cadence — the physics is sound, the timelines are the product, and the bridge decade belongs to efficiency and storage. The counter-ledger, because both hands: inference efficiency keeps improving faster than the discourse tracks, so the net is still growing but the composition is shifting the way every infrastructure boom shifts. The boom builds overcapacity, the overcapacity gets priced, the efficiency inherits it. ...

April 22, 2026

Patch Notes #323 — Thirteen Years Since the Restore Drill

Thirteen years ago this month, entry #007 recorded a four-hour backup-restore drill and a tech lead’s proverb that started my whole file: “a backup you haven’t restored is a rumor.” The anniversary fortnight delivered a perfect echo. Our quarterly restore drill ran against the newest layer of the stack for the first time — not just databases but the model portfolio’s routing state, eval baselines, and agent intent docs, all from cold storage. The finding was pure #007 lineage: the golden test sets restored perfectly, and the routing config’s restore procedure referenced a secrets path that had been migrated in November. The rumor was in the plumbing’s plumbing, where it always is. Fixed by Tuesday, drill re-run clean by Friday. Thirteen years, four title changes, one unbroken discipline: restore-test everything, especially the things that restore the other things. ...

April 7, 2026

Patch Notes #322 — European Predictions and the Provenance Dividend

The Champions League knockouts performed their annual rite: the datacenter-megawattage methodology died in the round of 16 (the science records its sacrifice), and the pool is currently led by the office’s one methodology-free entrant, who picks by “which club has the cooler crest.” Fourteen years of prediction pools; the crest method remains undefeated where it counts. Also for the permanent record: the women’s Champions League carried the cultural mainline for the third straight year. That’s not a surge anymore. It’s the sport. ...

March 23, 2026