CHYP.ai · ORBIT // Cloudy_Day Security Suite

Developer tools

Operational CircleCI status

circleci.com · Official status page (Statuspage-compatible JSON)

FROM ALL SOURCES

CircleCI looks fine itself, but Amazon Web Services, a provider it may run on (inferred), reports major outage. Watch for knock-on effects.

confidence: low Vendor: operational · users: normal · routing: no data · upstream providers: major.

Vendor status vs. reality: Vendor status and outside signals roughly agree

Regions mentioned: CircleCI API, CircleCI UI

The four clocks
ClockStartedRunning
Vendor clock when the vendor's open incident started42 days
Crowd clock when user reports began rising above normalnot seenn/a
Network clock when routing anomaly data first flagged itnot seenn/a
ORBIT clock when our checker first recorded the changenot seenn/a

Evidence

  • Official page: Operational (“All Systems Operational”). evidence confidence: high
  • Vendor stage: postmortem on “Delay in data appearing in UI and API”. evidence confidence: high
  • User reports are within normal range (0 in 15 min). evidence confidence: low
  • Amazon Web Services reports an issue. CircleCI may depend on it (inferred from public sources). Coincidence, not causation. evidence confidence: low

What we don't know: what is happening inside CircleCI's own systems, how many users are actually affected (reports are self-selected and unverified), and anything from routing / bgp. Routing data shows reachability, not whether the app works.

Coincidence, not causation: signals that overlap in time are shown side by side, not as proof of cause.

Automated analysis by Cloudy_Day NetWeather; not a personal statement. Generated .

Cloudy_Day's take

Stage vs. health

The vendor's stage is “postmortem”. Stage tells you where they are in their process; health is what users experience. Right now vendor: operational · users: normal · routing: no data · upstream providers: major. Vendor status and outside signals roughly agree.

Who depends on whom

CircleCI may run on Amazon Web Services, Google Cloud (inferred). We don't map other services that depend on CircleCI. Dependency chains are where single points of failure hide.

Identity & trust lens

Even without an identity system involved, outages push people toward workarounds: password resets, unofficial apps and “status” links in DMs. Each one is a trust decision. Use official channels.

For users

Check the official status page before changing settings or reinstalling. If you must act, use known-good bookmarks. Reporting what's broken on ORBIT helps everyone see the real footprint.

For security teams

Treat this as an availability event unless an official source says otherwise. Check which of your systems sit on CircleCI or its upstream providers, watch for outage-themed phishing aimed at your users, and log what you see with timestamps (all four clocks).

What to watch next

The vendor stage moving to “monitoring”, then “resolved”; the user-report curve falling back under baseline; and whether dependents start reporting issues.

Automated analysis by Cloudy_Day NetWeather; not a personal statement. Independent researcher; not affiliated with CircleCI. No blame or attack claims without an official source. Generated .

INCIDENT REPLAY

Delay in data appearing in UI and API

major postmortem · started · resolved

25 updates

  1. identified

    Between 11:15 UTC and about 13:00 UTC, pipelines were created but not processed, and outbound webhooks and transactional notifications (Slack and email) were not sent. What's impacted: Customers triggering pipelines, and customers relying on outbound webhooks and email and Slack notifications. What can you expect: - Since about 13:00 UTC, pipelines are being processed again and jobs are starting. - Most jobs are currently starting within about 4 minutes, and some are waiting up to about 10 minutes. Pipelines triggered during the incident are still being processed and may take longer to start. - Outbound webhooks and notifications have resumed. - Pipelines are still being processed from the backlog, so retriggering may result in duplicate pipelines. Thank you for your patience while our engineers work to resolve this. Next update: We will provide another update by 14:00 UTC.

  2. identified

    Since 11:15 UTC, customers have experienced disruption to pipelines and notifications. Between 11:15 UTC and about 13:00 UTC, pipelines were created but not processed, and outbound webhooks and transactional notifications (Slack and email) were not sent. What's impacted Customers running workflows and jobs, and customers viewing usage data on the Project Usage page. What can you expect - Pipelines are being processed again, including those triggered during the incident. - Since about 13:25 UTC, workflows and jobs have been starting with increasing delays as we work through the backlog. Most jobs are currently starting within about 10 minutes, and some are waiting up to about 25 minutes. - We expect to work through this backlog by about 14:35 UTC. - Workflows from pipelines triggered during the incident are still waiting to start, so retriggering may result in duplicate workflows. - Usage

  3. identified

    Since 11:15 UTC, customers have experienced disruption to pipelines and notifications. Between 11:15 UTC and about 13:00 UTC, pipelines were created but not processed, and outbound webhooks and transactional notifications (Slack and email) were not sent. What's impacted Customers running workflows and jobs, and customers viewing usage data on the Project Usage page. What can you expect - Delays starting workflows and jobs are improving. Most jobs are now starting within about 3 minutes, down from about 10 minutes at 13:50 UTC. - Some workflows from pipelines triggered during the incident are still waiting up to about 40 minutes to start. - Working through the backlog is taking slightly longer than expected. We now expect to clear it by about 14:45 UTC. - Workflows from pipelines triggered during the incident are still waiting to start, so retriggering may result in duplicate workflows. -

  4. identified

    Since 11:15 UTC, customers have experienced disruption to pipelines and notifications. Between 11:15 UTC and about 13:00 UTC, pipelines were created but not processed, and outbound webhooks and transactional notifications (Slack and email) were not sent. Between about 13:00 UTC and 14:40 UTC, workflows and jobs started with delays while we worked through the backlog. What’s impacted Customers running workflows and jobs, and customers viewing usage data on the Project Usage page. What can you expect - The backlog of workflows from pipelines triggered during the incident cleared at about 14:40 UTC. - Most jobs are now starting within about 30 seconds, and some are waiting up to about 3 minutes. - Usage charge data on the Project Usage page is currently stale. Thank you for your patience while our engineers continue to work on the underlying issue. Next update We will provide another update

  5. monitoring

    Since 11:15 UTC, customers have experienced disruption to pipelines and notifications. Between 11:15 UTC and about 13:00 UTC, pipelines were created but not processed, and outbound webhooks and transactional notifications (Slack and email) were not sent. Between about 13:00 UTC and 14:40 UTC, workflows and jobs started with delays while we worked through the backlog. What’s impacted Customers running workflows and jobs, and customers viewing usage data on the Project Usage page. What can you expect - The backlog of pipelines and workflows from the incident has cleared. - Since about 14:45 UTC, most workflows and jobs are starting within about 15 seconds, and start times are continuing to return to normal. - Usage charge data on the Project Usage page is currently stale. Thank you for your patience while our engineers monitor the recovery. Next update We will provide another update by 15:

  6. monitoring

    Since 11:15 UTC, customers have experienced disruption to pipelines and notifications. Between 11:15 UTC and about 13:00 UTC, pipelines were created but not processed, and outbound webhooks and transactional notifications (Slack and email) were not sent. Between about 13:00 UTC and 14:40 UTC, workflows and jobs started with delays while we worked through the backlog. What's impacted Customers viewing pipelines, workflows and jobs in the CircleCI UI, and customers viewing usage data on the Project Usage page. What can you expect - Workflows and jobs are running on time. - The CircleCI UI is about 30 minutes behind. New workflows and status changes may take that long to appear. - Our engineers are actively working to speed up processing so the UI can catch up. - Please avoid retriggering workflows that don't yet appear in the UI, as this may create duplicates. - Usage charge data on the Pr

  7. monitoring

    Since 11:15 UTC, customers have experienced disruption to pipelines and notifications. Between 11:15 UTC and about 13:00 UTC, pipelines were created but not processed, and outbound webhooks and transactional notifications (Slack and email) were not sent. Between about 13:00 UTC and 14:40 UTC, workflows and jobs started with delays while we worked through the backlog. What's impacted Customers running workflows and jobs, customers viewing pipelines, workflows and jobs in the CircleCI UI, and customers viewing usage data on the Project Usage page. What can you expect - Since about 15:15 UTC, workflows and jobs have been starting with increasing delays. Most jobs are currently starting within about 4 minutes, and some are waiting up to about 20 minutes. - The CircleCI UI remains about 30 minutes behind on average, and some updates are taking longer. New workflows and status changes may take

  8. monitoring

    Since 11:15 UTC, customers have experienced disruption to pipelines and notifications. Between 11:15 UTC and about 13:00 UTC, pipelines were created but not processed, and outbound webhooks and transactional notifications (Slack and email) were not sent. Between about 13:00 UTC and 14:40 UTC, workflows and jobs started with delays while we worked through the backlog. Since about 15:15 UTC, workflows and jobs have again been starting with delays. What's impacted Customers running workflows and jobs, customers viewing pipelines, workflows and jobs in the CircleCI UI, and customers viewing usage data on the Project Usage page. What can you expect - Delays starting workflows and jobs have stopped increasing and are beginning to ease. Most jobs are starting within about 4 minutes, and some are waiting up to about 25 minutes. - The CircleCI UI is now about 20 minutes behind, down from about 30

  9. monitoring

    Since 11:15 UTC, customers have experienced disruption to pipelines and notifications. Between 11:15 UTC and about 13:00 UTC, pipelines were created but not processed, and outbound webhooks and transactional notifications (Slack and email) were not sent. Between about 13:00 UTC and 14:40 UTC, workflows and jobs started with delays while we worked through the backlog. Since about 15:15 UTC, workflows and jobs have again been starting with delays. What’s impacted Customers running workflows and jobs, customers viewing pipelines, workflows and jobs in the CircleCI UI, and customers viewing usage data on the Project Usage page. What can you expect - Delays starting workflows and jobs are holding steady. Most jobs are starting within about 4 minutes, and some are waiting up to about 8 minutes, down from about 25 minutes. - The CircleCI UI remains about 20 minutes behind on average, and some u

  10. identified

    Since 11:15 UTC, customers have experienced disruption to pipelines and notifications. Between 11:15 UTC and about 13:00 UTC, pipelines were created but not processed, and outbound webhooks and transactional notifications (Slack and email) were not sent. Between about 13:00 UTC and 14:40 UTC, workflows and jobs started with delays while we worked through the backlog. Since about 15:15 UTC, workflows and jobs have again been starting with delays. What’s impacted Customers triggering pipelines and running workflows and jobs, customers viewing pipelines, workflows and jobs in the CircleCI UI, and customers viewing usage data on the Project Usage page. What can you expect - Since about 17:00 UTC, delays have increased significantly. Most jobs are now taking about 12 minutes to start, and some are waiting up to about 30 minutes. - Since about 17:15 UTC, some jobs are failing to start. - Pipel

  11. identified

    Since 11:15 UTC, customers have experienced disruption to pipelines and notifications. Between 11:15 UTC and about 13:00 UTC, pipelines were created but not processed, and outbound webhooks and transactional notifications (Slack and email) were not sent. Between about 13:00 UTC and 14:40 UTC, workflows and jobs started with delays while we worked through the backlog. Since about 15:15 UTC, workflows and jobs have again been starting with delays. What’s impacted Customers triggering pipelines and running workflows and jobs, customers viewing pipelines, workflows and jobs in the CircleCI UI, and customers viewing usage data on the Project Usage page. What to expect - Since about 17:00 UTC, we are seeing an increase in delays. Most jobs are now taking about 25 minutes to start, and some are waiting up to about an hour. - Since about 17:15 UTC, some in-progress workflows have failed and a sm

  12. identified

    What's changed since our last update * The CircleCI UI is about an hour behind, and up to about 90 minutes for some updates. Workflows and jobs may already have run, or may be running, even if the UI shows them as not started or does not show them at all. * Jobs that are running are starting faster. Most are now starting within about 15 minutes, down from about 28 minutes, though some are still waiting over an hour. * New pipelines are increasingly delayed. * Some workflows are still failing. Workflows that could not be processed in time are being marked as failed. * Please avoid retriggering workflows that don't yet appear in the UI, as this may create duplicates. Still ongoing: Usage charge data on the Project Usage page is stale. Next update We will provide another update by 19:00 UTC.

  13. identified

    What's changed since our last update * Jobs are starting faster. Most are now starting within about 8 minutes, down from about 15 minutes, and the longest waits have dropped from over an hour to about 45 minutes. * The CircleCI UI is beginning to catch up, but it is still behind. Workflows and jobs may already have run, or may be running, even if the UI shows them as not started or does not show them at all. * Pipelines are being processed faster, but new pipelines are still delayed. Still ongoing * Some workflows and jobs are still failing. * Please avoid retriggering workflows that don't yet appear in the UI, as this may create duplicates. * Usage charge data on the Project Usage page is stale. Next update We will provide another update by 19:30 UTC.

  14. identified

    Since our last update, jobs are starting faster: most are now starting within about 6 minutes, down from about 8 minutes. The CircleCI UI is catching up but still about an hour behind. New pipelines are being processed as they arrive. Jobs may already have run even if the UI does not show them yet, so please avoid retriggering workflows that don't appear in the UI. We will provide another update by 20:00 UTC.

  15. identified

    Jobs are starting faster and fewer are failing: most jobs are now starting within about 4 minutes, down from about 6 minutes, and the longest waits have dropped from about 30 minutes to about 20 minutes. Pipelines from earlier in the incident are now being worked through. The CircleCI UI is still behind, and some older updates may take longer to appear. Jobs may already have run even if the UI does not show them yet, so please avoid retriggering workflows that don't appear in the UI. We will provide another update by 20:30 UTC.

  16. identified

    Since about 20:48 UTC, new workflows and jobs are not starting. We have temporarily paused starting new work so that our systems can catch up on work already in progress, including delayed updates to the CircleCI UI and API. Older updates are now appearing in the UI and API as this backlog clears. New pipelines are still being accepted and queued Please avoid retriggering pipelines or workflows during this time, as this may create duplicates once work resumes. We will provide another update by 22:00 UTC.

  17. identified

    Since our last update, most of the pipelines that were delayed earlier in the incident have now been processed, and their workflows are queued to start. New workflows and jobs remain paused while we prepare to resume work safely. Delayed updates continue to appear in the CircleCI UI and API. Please continue to avoid retriggering pipelines or workflows, as this may create duplicates once work resumes. We will provide another update by 22:30 UTC.

  18. identified

    Since our last update, we have begun gradually resuming work, and some queued workflows and jobs are starting again. Most queued work is still waiting, and jobs that do start may have waited over an hour. We are increasing the rate carefully to avoid overloading our systems. Delayed updates continue to appear in the CircleCI UI and API. Please continue to avoid retriggering pipelines or workflows, as this may create duplicates as queued work resumes. We will provide another update by 23:00 UTC.

  19. identified

    A large number of queued workflows and jobs have started, and the CircleCI UI and API have caught up and are showing updates normally with few delays. We are continuing to work through the remaining queued work. Jobs from the queue may have waited about an hour or more to start, and some Linux jobs are waiting up to about 10 minutes for capacity. Please continue to avoid retriggering while queued work continues to process. We will provide another update by 23:30 UTC.

  20. identified

    We are still working through a significant backlog of queued workflows and jobs. Wait times are improving but they remain longer than normal and some jobs have waited over an hour. The CircleCI UI and API remain up to date. Please avoid retriggering pipelines or workflows while the backlog clears, as this may create duplicates. We will provide another update by 00:00 UTC.

  21. identified

    Since our last update, the backlog of queued workflows and jobs has cleared, as of about 23:47 UTC. Workflows and jobs are now starting within seconds, and the CircleCI UI and API are up to date. We are monitoring our systems closely to make sure they remain stable. We will provide another update by 00:30 UTC.

  22. identified

    Workflows and jobs have continued to start normally, within seconds, and the CircleCI UI and API remain up to date. Slack and email notifications delayed earlier in the incident are still being delivered. Usage charge data on the Project Usage page is stale. We are continuing to monitor our systems closely and will provide another update by 01:15 UTC.

  23. monitoring

    Workflows and jobs are starting and running normally, and the CircleCI UI and API are up to date. We are continuing to monitor our systems closely to make sure they remain stable. We will provide another update by 02:00 UTC.

  24. resolved

    Between 11:15 UTC and 23:47 UTC on October 9, customers experienced delays and failures across pipelines, workflows and jobs, along with delays in the CircleCI UI, API and notifications. From 11:15 to about 13:00 UTC, pipelines were created but not processed, and outbound webhooks and Slack and email notifications were not sent. From about 13:00 to 14:40 UTC, workflows and jobs started with delays while we worked through the backlog. From about 15:15 to 20:48 UTC, workflows and jobs started with increasing delays, of up to about an hour, and the CircleCI UI and API fell behind by up to about 2 hours. During this time, some jobs failed to start and some in-progress workflows failed. From 20:48 to about 22:10 UTC, we temporarily paused starting new workflows and jobs so our systems could recover. New pipelines were accepted and queued during this time. From about 22:10 to 23:47 UTC, queued

  25. postmortem

    ## Summary Between 11:15 UTC and 23:47 UTC on October 9, CircleCI customers experienced hours-long delays and failures across pipelines, workflows and jobs, along with delays in the CircleCI UI, API and notifications. * From 11:15 to 13:00 UTC, pipelines were created but not processed, and outbound webhooks and Slack and email notifications were not sent. * From 13:00 to 14:40 UTC, workflows and jobs started with delays while we worked through the backlog. * From 15:15 to 20:48 UTC, workflows and jobs started with increasing delays, of up to about an hour, and the CircleCI UI and API fell behind by up to about 2 hours. During this time, some jobs failed to start and some in-progress workflows failed. * From 20:48 to 22:10 UTC, we temporarily paused starting new workflows and jobs so our systems could recover. New pipelines were accepted and queued during this time. * From 22:10 to 23:47

Official incident page ↗

Community comments (0)

Comments are opinions of visitors, not statements by Cloudy_Day. Every comment is held for review before it appears. Links are removed, and personal data is redacted. Community guidelines.

No approved comments yet.

How to read this: official status comes from each provider's own public status page. User reports are unverified visitor reports. Things that happen at the same time are coincidence, not causation. Cloudy_Day is independent and not an official status source for any company listed.

Where to next