+− THE DAILY DIFFdev & AI news
Amavidiyo / postmortem
SHIP IT

Inkambu eyodwa eyengeziwe yenze amakhompyutha angu-8.5 million ashayise. Imizuzu engu-78.

Ishicilelwe Sep 14, 2026 · 2:52 · Niko from Axrisi

Julayi 19, 2024, 04:09 UTC: I-CrowdStrike icindezela i-Channel File 291 — isibuyekezo se-Rapid Response Content — kuwo wonke ama-Windows Falcon sensor aku-inthanethi.

Julayi 19, 2024, 04:09 UTC: I-CrowdStrike icindezela i-Channel File 291 — isibuyekezo se-Rapid Response Content — kuwo wonke ama-Windows Falcon sensor aku-inthanethi. Isifanekiso esisondla simenyezelwe ngamafildi angu-21 okokufaka ngoFebhuwari; ikhodi ehlinzeka ngokufaka yakha uhlu lwama-20. Izinyanga ezine akukho okufunda ifildi 21, ngoba yonke imithetho iyishiya njenge-wildcard. Lesi sibuyekezo sibeka iphethini yangempela lapho. I-Content Validator eseceleni kwefu ibala ama-21 ngokumelene nesimemezelo bese iyidlulisa; i-Content Interpreter ye-sensor, ngaphandle kokuhlola imingcele, ifunda indawo 21 kumodi ye-kernel nama-Windows page-faults: PAGE_FAULT_IN_NONPAGED_AREA, csagent.sys. I-driver ilayisha ekuqaleni kokubhutha ngenhloso, ifunda ifayela elifanayo, iphinde ishayise — ukubhutha okujikelezayo. Ibuyiselwe emuva ngo-05:27 UTC, imizuzu engu-78 kamuva, kodwa imishini engu-8.5 million (ukubala kwe-Microsoft, ngaphansi kuka-1% we-Windows) iselilenzile ifayela, futhi isilungiso ngumuntu kuyo yonke ikhibhodi: Imodi Ephephile, susa i-C-00000291*.sys, ukhiye wokubuyisela i-BitLocker. Izinkampani zezindiza, izibhedlela, amabhange nezikhungo ze-911 ziyama; i-Delta ikhansele izindiza ezilinganiselwa ku-7,000 futhi ifuna okungenani $500 million ku-CrowdStrike naku-Microsoft; i-Parametrix ibeka isikweletu se-Fortune 500 eduze $5.4 billion. I-postmortem evela ku-Root Cause Analysis ye-CrowdStrike, i-Preliminary Post Incident Review, ibhlog le-Microsoft, i-8-K ye-Delta kanye nobufakazi beNdlu. I-Git blame: 65 CrowdStrike / 25 idizayini yemodi ye-kernel / 10 uLwesihlanu. Isinqumo ngesilungiso: SHIP IT, ngokuqinile.

Funda uhlelo olubhaliwe (isiNgisi) ↗

Lokho okuhlanganiswa yile vidiyo

  • Feb 28 – Ephreli 24, 2024: i-sensor 7.11 ithumela i-IPC Template Type (amafildi angu-21 amenyezelwe, ama-20 ahlinzekiwe); ukuhlola ukucindezeleka kudlula ngoMashi 5; izimo ezine zisebenza ngoba ifildi 21 iyi-wildcard
  • Julayi 19, 04:09 UTC: izifanekiso ezimbili ezintsha; iphutha le-logic le-Content Validator livumela elingeyona i-wildcard ukuba lidlule; ukufunda okuphumile emingceleni kumodi ye-kernel; ngo-05:27 UTC kubuyiselwe emuva
  • Julayi 19–29: ukulungiswa ngesandla (Imodi Ephephile, susa ifayela, ukhiye we-BitLocker); cishe u-97% wama-sensor abuyile ngoJulayi 24, cishe u-99% ngoJulayi 29
  • Julayi 25 – Agasti 19: ukuhlola imingcele ngesikhathi sokusebenza, ukuqinisekiswa kwenani lamafildi ngesikhathi sokuhlanganisa, ukuhlola kwe-Validator, i-canary + izindandatho zokuthunyelwa, ukulawula kwamakhasimende nge-Rapid Response Content, ukubuyekezwa okubili okuzimele; i-RCA ephelele nge-crash dump ngo-Agasti 6

Umbhalo obhaliwe ohunyushwe

Kuhunyushwe kusuka ekulandiseni kokuqala kwesiNgisi. Umsindo namazwibela atholakalayo alawulwa yi-YouTube.

0:00 Isibuyekezo esisodwa sokuphepha sithwala amafildi angamashumi amabili nanye ku-driver eyakhelwe amashumi amabili, futhi izigidi eziyisishiyagalombili nengxenye zemishini ye-Windows ziba yi-blue screen ngaphambi kwasekuseni. Julayi 19, 2024, 04:09 UTC. I-CrowdStrike icindezela i-Channel File 291 kuwo wonke ama-Falcon sensor. Ngemizuzu engamashumi ayisikhombisa nesishiyagalombili kamuva bayayisusa; ngaleso sikhathi izinkampani zezindiza, izibhedlela namabhange abuka i-rectangle eluhlaza okwesibhakabhaka efanayo. Okokuqala okuthunyelwe yi-CEO kusho ukuthi akukona ukuhlasela nge-cyber, kuyiqiniso, futhi akuyona inkinga. Kwenzeka kanjani, kungani kungenzeka, futhi ubani osolwa.

0:27 Lena yi-The Daily Diff, i-postmortem. Febhuwari 28. I-Sensor 7.11 ithumela i-detector entsha yamapayipi anamagama. Imemezela amafildi angu-21 okokufaka; ikhodi eyihambisayo yakha uhlu lwama- amashumi amabili. Akekho obonayo: izinyanga ezine yonke imithetho iyishiya ifildi engamashumi amabili nanye njenge- i-wildcard, futhi akekho ofunda i-wildcard. Ukuhlola ukucindezeleka kuyadlula, imithetho emine iyathunyelwa, konke kuhamba kahle. Julayi 19, 04:09. Imithetho emibili emisha; enye ibeka iphethini yangempela efildini lamashumi amabili nanye.

0:51 I-cloud validator ibala amashumi amabili nanye ngokumelene nesimemezelo bese ithi kuhle. Wonke ama-Windows host aku-inthanethi ayalanda ngesikhathi esisodwa, ngoba akukho ring ye-canary. I-sensor ifunda indawo yamashumi amabili nanye, okungeyona inkumbulo yayo, futhi i-kernel yenza lokho i-kernel ekwenzayo. 05:27, kubuyiselwe emuva. Sekwephuze kakhulu kunoma ubani oye waqala kabusha: i-driver ilayisha ekuqaleni kokubhutha, ifunda ifayela elifanayo, iphinde ishayise. Ukubhutha okujikelezayo. Kungani lokhu kungenzeka? Imithetho iyisilungiselelo esiphushwa kusuka efwini, ngakho akukho khodi eyeqa i-code

1:16 pipeline. I-interpreter ayinakho ukuhlola imingcele; iyathemba i-validator, futhi i-validator ithemba isimemezelo. Futhi konke kusebenza kumodi ye-kernel, lapho i-pointer embi kungesona isiphazamiso kodwa i- page fault i-Windows ithi ayikwazi ukuvikelwa yi-try-except. U-Patrick Wardle uyayifunda ku-crash dump ngalelo langa ntambama: i-index zero x fourteen, indawo yamashumi amabili nanye. Isilungiso singesandla. Imodi Ephephile, susa i-C-00000291, qala kabusha. Nge-BitLocker, qala ngokuthayipha inombolo engamashumi amane nesishiyagalombili

1:43 ukhiye wokubuyisela ohlala ku- server nayo eluhlaza okwesibhakabhaka. git blame. I-CrowdStrike, amaphesenti angamashumi ayisithupha nanhlanu: i-RCA ibala izitholakele eziyisithupha, futhi isitholakele sesithupha singumusho ogcwele: Izifanekiso kufanele zibe nokuthunyelwa okugunyaziwe. Idizayini yemodi ye-kernel, amashumi amabili nanhlanu: I-CrowdStrike ithi i-Windows ayikwazi okwamanje ukusingatha ukuphepha ngaphandle kwe-kernel, i-Microsoft ithi akusona isigameko se-Microsoft, futhi i-page fault ayikhathali.

2:05 ULwesihlanu, amashumi. Umyalezo wokuzinikela uthi akukona ukubhala ikhodi. Ubukhulu bokungcola: amayunithi ayizigidi eziyisishiyagalombili nengxenye, ngaphansi kwephaphesenti elilodwa le-Windows. I-Delta ikhansele izindiza eziyizinkulungwane eziyisikhombisa futhi imangalela ngamakhulu amahlanu ezigidi, i-Microsoft ifakiwe. Abashushisi babeka isikweletu se-Fortune 500 eduze kwezigidigidi ezinhlanu. I-Hacker News: amaphuzu ayizinkulungwane ezine namakhulu amahlanu. I-irony: isibuyekezo esihlose ukuthola amasu amasha okuhlasela silethe eyodwa. Isinqumo, i-postmortem: SHIP IT, ngokuqinile.

2:27 Ukuhlola imingcele ezinsukwini eziyisithupha, izindandatho ezigunyaziwe, ukulawula kwamakhasimende ngezingxenyekazi zokwenziwa kwezibuyekezo, ukubuyekezwa okubili kwangaphandle. Ngokuqinile, ngoba konke lokho kwakuyinto ejwayelekile kakade. UMsombuluko: noma yini umenzeli wakho ayikhipha kusuka efwini iba ne-canary ring. Ikhodi yilungiselelo lapho okuthile kuyihlaziya lapho ingakwazi ukuphonsa khona. Ngithumele isigameko ongasavumelekile ukukhuluma ngaso, kumazwana, noma ku-daily diff dot dev. Futhi yilokho okuhlukile kwanamuhla.

2:48 NginguNiko ovela ku-Axrisi. Hlanganisa ngendlela efanele.

Imithombo

  1. CrowdStrike, Root Cause Analysis — Channel File 291 (Aug 6, 2024, 12 pp.)www.crowdstrike.com
  2. CrowdStrike, Executive Summary of the RCA (Aug 6, 2024)www.crowdstrike.com
  3. CrowdStrike, Preliminary Post Incident Review (Jul 24, 2024)www.crowdstrike.com
  4. CrowdStrike, Remediation and Guidance Hub — Channel File 291 (the workaround steps)www.crowdstrike.com
  5. CrowdStrike, tech alert of Jul 19, 2024 (archived: Safe Mode → delete C-00000291*.sys → BitLocker key)web.archive.org
  6. Microsoft, David Weston, "Helping our customers through the CrowdStrike outage" (Jul 20, 2024; the 8.5 million figure)blogs.microsoft.com
  7. Delta Air Lines, Form 8-K (Aug 8, 2024; ~7,000 cancellations, at least $500 million)www.sec.gov
  8. Adam Meyers (CrowdStrike SVP), written testimony, House Homeland Security Subcommittee (Sep 24, 2024)homeland.house.gov
  9. George Kurtz on X, Jul 19, 2024, 09:45 UTCx.com
  10. Patrick Wardle on X, Jul 19, 2024 (the crash-dump index 0x14)x.com
  11. Hacker News, Jul 19, 2024 — "CrowdStrike Update: Windows Bluescreen and Boot Loops" (4,489 points)news.ycombinator.com
  12. The Guardian, Jul 24, 2024 — "CrowdStrike global outage to cost US Fortune 500 companies $5.4bn" (Parametrix estimate)www.theguardian.com
  13. Hacker News, Jul 21, 2024 — "Initial details about why CrowdStrike's CSAgent.sys crashed" (519 points)news.ycombinator.com
  14. Wikipedia — 2024 CrowdStrike-related IT outagesen.wikipedia.org

Amavidiyo ahlobene