CHYP.ai · ORBIT // Cloudy_Day Security Suite

OUTAGE DISSECTOR · CASE FILE · Meltdown Index · vendor post-incident report

Internal-network surge, retries, and control-plane API failures

Amazon Web Services resolved M7.0 Severe storm 2021-12-07

Amazon Web Services: Internal-network surge, retries, and control-plane API failures. Scope: Regional (control planes); existing data-plane workloads largely continued.

M7.0

Severe storm

How CHYP Magnitude works

  • 6h 52mDurationstart → resolved
  • 8Systems namedfrom the source
  • 1Places named1 on the map
  • 136Dependentscatalogued, inferred
  • 2/4Evidence kindsindependent families
  • 2Updatesofficial revisions

1Scrub the outage

full window
Severityofficial state Dec 7, 7 AMDec 7, 8 AMDec 7, 10 AMDec 7, 11 AMDec 7, 12 PMDec 7, 2 PM Official statusDec 7, 7:30 AM PT · Impact begins (vendor-stated)Dec 7, 2:22 PM PT · Mitigated (vendor-stated)Network & routingnothing in this windowEarly signalsnothing in this windowNews & chatternothing in this windowCoincidence (KEV)nothing in this window ▼ startresolved ▼

Drag or press Play to replay what was public at each moment. Each dot links to its evidence. Lanes are kept separate on purpose.

What happened

Amazon Web Services: Internal-network surge, retries, and control-plane API failures. Scope: Regional (control planes); existing data-plane workloads largely continued.

What broke

EC2 control-plane APIs, Elastic Load Balancing APIs, Route 53 APIs, AWS Management Console and 4 more.

Who was hit

136 catalogued services run on Amazon Web Services (inferred). Hit: Regional (control planes); existing data-plane workloads largely continued.

How it recovered

Resolved after 6h 52m.

4Failure-mode fingerprint

RetriesControl planeConfig pushCascadeCapacity
  • Retry amplificationvendor post-incident report (Meltdown Index)
  • Control-plane couplingvendor post-incident report (Meltdown Index)
  • Capacity exhaustionvendor post-incident report (Meltdown Index)
  • Config propagationno sign
  • Dependency cascadeno sign

Failure modes as stated in the vendor's own post-incident report (Meltdown Index).

5Evidence matrix

Official statusOfficial status · Dec 7, 1:30 AM PT: 0 itemsOfficial status · Dec 7, 2:02 AM PT: 0 itemsOfficial status · Dec 7, 2:34 AM PT: 0 itemsOfficial status · Dec 7, 3:06 AM PT: 0 itemsOfficial status · Dec 7, 3:38 AM PT: 0 itemsOfficial status · Dec 7, 4:10 AM PT: 0 itemsOfficial status · Dec 7, 4:43 AM PT: 0 itemsOfficial status · Dec 7, 5:15 AM PT: 0 itemsOfficial status · Dec 7, 5:47 AM PT: 0 itemsOfficial status · Dec 7, 6:19 AM PT: 0 itemsOfficial status · Dec 7, 6:51 AM PT: 0 itemsOfficial status · Dec 7, 7:23 AM PT: 1 itemOfficial status · Dec 7, 7:56 AM PT: 0 itemsOfficial status · Dec 7, 8:28 AM PT: 0 itemsOfficial status · Dec 7, 9:00 AM PT: 0 itemsOfficial status · Dec 7, 9:32 AM PT: 0 itemsOfficial status · Dec 7, 10:04 AM PT: 0 itemsOfficial status · Dec 7, 10:36 AM PT: 0 itemsOfficial status · Dec 7, 11:09 AM PT: 0 itemsOfficial status · Dec 7, 11:41 AM PT: 0 itemsOfficial status · Dec 7, 12:13 PM PT: 0 itemsOfficial status · Dec 7, 12:45 PM PT: 0 itemsOfficial status · Dec 7, 1:17 PM PT: 0 itemsOfficial status · Dec 7, 1:49 PM PT: 0 itemsNetwork & routingNetwork & routing · Dec 7, 1:30 AM PT: 0 itemsNetwork & routing · Dec 7, 2:02 AM PT: 0 itemsNetwork & routing · Dec 7, 2:34 AM PT: 0 itemsNetwork & routing · Dec 7, 3:06 AM PT: 0 itemsNetwork & routing · Dec 7, 3:38 AM PT: 0 itemsNetwork & routing · Dec 7, 4:10 AM PT: 0 itemsNetwork & routing · Dec 7, 4:43 AM PT: 0 itemsNetwork & routing · Dec 7, 5:15 AM PT: 0 itemsNetwork & routing · Dec 7, 5:47 AM PT: 0 itemsNetwork & routing · Dec 7, 6:19 AM PT: 0 itemsNetwork & routing · Dec 7, 6:51 AM PT: 0 itemsNetwork & routing · Dec 7, 7:23 AM PT: 0 itemsNetwork & routing · Dec 7, 7:56 AM PT: 0 itemsNetwork & routing · Dec 7, 8:28 AM PT: 0 itemsNetwork & routing · Dec 7, 9:00 AM PT: 0 itemsNetwork & routing · Dec 7, 9:32 AM PT: 0 itemsNetwork & routing · Dec 7, 10:04 AM PT: 0 itemsNetwork & routing · Dec 7, 10:36 AM PT: 0 itemsNetwork & routing · Dec 7, 11:09 AM PT: 0 itemsNetwork & routing · Dec 7, 11:41 AM PT: 0 itemsNetwork & routing · Dec 7, 12:13 PM PT: 0 itemsNetwork & routing · Dec 7, 12:45 PM PT: 0 itemsNetwork & routing · Dec 7, 1:17 PM PT: 0 itemsNetwork & routing · Dec 7, 1:49 PM PT: 0 itemsEarly signalsEarly signals · Dec 7, 1:30 AM PT: 0 itemsEarly signals · Dec 7, 2:02 AM PT: 0 itemsEarly signals · Dec 7, 2:34 AM PT: 0 itemsEarly signals · Dec 7, 3:06 AM PT: 0 itemsEarly signals · Dec 7, 3:38 AM PT: 0 itemsEarly signals · Dec 7, 4:10 AM PT: 0 itemsEarly signals · Dec 7, 4:43 AM PT: 0 itemsEarly signals · Dec 7, 5:15 AM PT: 0 itemsEarly signals · Dec 7, 5:47 AM PT: 0 itemsEarly signals · Dec 7, 6:19 AM PT: 0 itemsEarly signals · Dec 7, 6:51 AM PT: 0 itemsEarly signals · Dec 7, 7:23 AM PT: 0 itemsEarly signals · Dec 7, 7:56 AM PT: 0 itemsEarly signals · Dec 7, 8:28 AM PT: 0 itemsEarly signals · Dec 7, 9:00 AM PT: 0 itemsEarly signals · Dec 7, 9:32 AM PT: 0 itemsEarly signals · Dec 7, 10:04 AM PT: 0 itemsEarly signals · Dec 7, 10:36 AM PT: 0 itemsEarly signals · Dec 7, 11:09 AM PT: 0 itemsEarly signals · Dec 7, 11:41 AM PT: 0 itemsEarly signals · Dec 7, 12:13 PM PT: 0 itemsEarly signals · Dec 7, 12:45 PM PT: 0 itemsEarly signals · Dec 7, 1:17 PM PT: 0 itemsEarly signals · Dec 7, 1:49 PM PT: 0 itemsNews & chatterNews & chatter · Dec 7, 1:30 AM PT: 0 itemsNews & chatter · Dec 7, 2:02 AM PT: 0 itemsNews & chatter · Dec 7, 2:34 AM PT: 0 itemsNews & chatter · Dec 7, 3:06 AM PT: 0 itemsNews & chatter · Dec 7, 3:38 AM PT: 0 itemsNews & chatter · Dec 7, 4:10 AM PT: 0 itemsNews & chatter · Dec 7, 4:43 AM PT: 0 itemsNews & chatter · Dec 7, 5:15 AM PT: 0 itemsNews & chatter · Dec 7, 5:47 AM PT: 0 itemsNews & chatter · Dec 7, 6:19 AM PT: 0 itemsNews & chatter · Dec 7, 6:51 AM PT: 0 itemsNews & chatter · Dec 7, 7:23 AM PT: 0 itemsNews & chatter · Dec 7, 7:56 AM PT: 0 itemsNews & chatter · Dec 7, 8:28 AM PT: 0 itemsNews & chatter · Dec 7, 9:00 AM PT: 0 itemsNews & chatter · Dec 7, 9:32 AM PT: 0 itemsNews & chatter · Dec 7, 10:04 AM PT: 0 itemsNews & chatter · Dec 7, 10:36 AM PT: 0 itemsNews & chatter · Dec 7, 11:09 AM PT: 0 itemsNews & chatter · Dec 7, 11:41 AM PT: 0 itemsNews & chatter · Dec 7, 12:13 PM PT: 0 itemsNews & chatter · Dec 7, 12:45 PM PT: 0 itemsNews & chatter · Dec 7, 1:17 PM PT: 0 itemsNews & chatter · Dec 7, 1:49 PM PT: 0 itemsCoincidence (KEV)Coincidence (KEV) · Dec 7, 1:30 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 2:02 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 2:34 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 3:06 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 3:38 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 4:10 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 4:43 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 5:15 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 5:47 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 6:19 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 6:51 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 7:23 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 7:56 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 8:28 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 9:00 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 9:32 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 10:04 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 10:36 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 11:09 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 11:41 AM PT: 0 itemsCoincidence (KEV) · Dec 7, 12:13 PM PT: 0 itemsCoincidence (KEV) · Dec 7, 12:45 PM PT: 0 itemsCoincidence (KEV) · Dec 7, 1:17 PM PT: 0 itemsCoincidence (KEV) · Dec 7, 1:49 PM PT: 0 itemsDec 7, 1:30 AM PTDec 7, 2:22 PM PT

Correlation is not causation

Five independent lanes across the window. Brighter = stronger item. Items landing in the same column are coincidence unless the provider says otherwise.

6In the vendor's words

This resulted in a large surge of connection activity that overwhelmed the networking devices between the internal network and the main AWS network, resulting in delays for communication between these networks. These delays increased latency and errors for services communicating between these networks, resulting in even more connection attempts and retries.

Network devices fully recovered at 2:22 PM PST. Some services recovered later (STS full recovery 4:28 PM PST; new EC2 launches 2:40 PM PST). Duration uses the official 2:22 PM PST network-device recovery.

Verbatim excerpt from Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region.

7Historical twins

Nearest matches in the Control-Plane Meltdown Index (1.0.0, vendor-sourced, tracked since 2024 with earlier landmark cases). Similarity = provider + failure modes + scope.

Cloudy_Day take

  • 136 catalogued services lean on one provider. That's concentration risk, and it's the real headline.
  • Longer than 10 of 19 indexed control-plane meltdowns with a stated duration.

Automated take: rules written by Cloudy_Day, applied to public data. Not a human verdict on this incident, not a root-cause finding, not affiliated with Amazon Web Services.

8Dissection capsule

Share card for this dissection

Everything above as one shareable, citable capsule with a content hash.

JSON capsuleMarkdownShare card (SVG)

Open Amazon Web Services in the War Room →Status page →

Embed this card
<a href="https://www.chyp.ai/dissect/m/aws-use1-2021-12-07/"><img src="https://www.chyp.ai/dissect/m/aws-use1-2021-12-07/card.svg" alt="Internal-network surge, retries, and control-plane API fail… · ORBIT Outage Dissector" width="600"></a>

Sources & limits

  • Taken from the vendor's own post-incident report; quotes are verbatim excerpts.
  • Dependents are today's inferred catalogue, not who was affected on the day.
  • Correlation is not causation. Lanes are kept apart so overlap never reads as cause.
  • ORBIT is passive: it never contacts the provider or probes its systems. Independent researcher, not affiliated.