Unjiniyela usule i-database yokukhiqiza ye-GitLab. 300 gigabytes.
NgoJanuwari 31, 2017, 23:27 UTC: unjiniyela we-GitLab, elwa nekhophi eliphukile ekupheleni kobusuku obude, ususa uhla lwemibhalo yedatha ye-PostgreSQL ku-db1 esikhundleni se-db2.
NgoJanuwari 31, 2017, 23:27 UTC: unjiniyela we-GitLab, elwa nekhophi eliphukile ekupheleni kobusuku obude, ususa uhla lwemibhalo yedatha ye-PostgreSQL ku-db1 esikhundleni se-db2. I-db1 ingeyokuqala. Cishe ama-300 GB we-database ye-GitLab.com ashabalale ngesekhondi noma amabili, kanti kunezindlela eziyisihlanu zokwenza ikhophi yasenqolobaneni nezokuphendula, ayikho esebenzayo. Ukuhlolwa kokwenzekile: umugqa wesikhathi kusukela ekukhuphukeni kwe-spam kuya egameni elingelona, kungani i-pg_basebackup yabukeka ibambekile, kungani i-pg_dump ibihluleka buthule (9.2 binaries ku-9.6 database, ama-imeyili okuhluleka abuyiselwe yi-DMARC), ukubuyisela kwamahora ayi-18 kusuka kusifinyezo se-staging eseneminyaka engu-6 esakazwa bukhoma ku-YouTube, nokuthi ubani ngempela okumele asolwe. Isinqumo ngempendulo: SHIP IT.
Funda uhlelo olubhaliwe (isiNgisi) ↗
Lokho okuhlanganiswa yile vidiyo
- Jan 31, 2017: rm -Rvf kuhla lwemibhalo yedatha yokuqala; ~300 GB kususiwe, 4.5 GB kusale
- 5 kwabangu-5 abenzi bekhophi yasenqolobaneni bayahluleka: ibhakede le-S3 elingenalutho (ukungafani kwenguqulo ye-pg_dump), akukho zifinyezo ze-Azure ku-DB, ikhophi esusiwe, ikhophi ye-LVM yansuku zonke ngaphandle kwama-webhook
- Feb 1, 18:00 UTC: I-GitLab.com ibuyile kusuka kusifinyezo esenziwe ngesandla esineminyaka engu-6; idokhumende elibukhoma, ukusakazwa bukhoma, ukuhlolwa kokwenzekile okungenasici ngohlu lokulungisa
Umbhalo obhaliwe ohunyushwe
Kuhunyushwe kusuka ekulandiseni kokuqala kwesiNgisi. Umsindo namazwibela atholakalayo alawulwa yi-YouTube.
0:00 Unjiniyela kwa-GitLab usebenzisa i-rm -rf kuseva yedatha engafanele, futhi amakhulu amathathu ama-gigabytes we-GitLab dot com ayanyamalala ngesekhondi noma amabili, cishe isikhathi esithatha ukufunda igama leseva. Januwari 31st, 2017, 11:27 p.m. UTC. I-GitLab ihlenga ukuthi isule ngephutha idatha yokukhiqiza, ivula amanothi ayo esigameko ku-inthanethi, futhi isakaza ukubuyisela ku-YouTube, ukusakazwa okubili okubukhoma endaweni. Ngosuku olulandelayo, ngokubhala: kuzindlela eziyisihlanu zokwenza ikhophi yasenqolobaneni,
0:25 ayikho esebenza ngokuthembeka. Kwenzeka kanjani, kungani kungenzeka, nokuthi ubani ngempela okumele asolwe. Lena i-The Daily Diff, ukuhlolwa kokwenzekile. 5:20 p.m.: unjiniyela wenza isifinyezo sokukhiqiza ukuhlola i-load balancer ku-staging. 7 p.m.: i-spam ihlasela idatha, kanye nomsebenzi osusa umsebenzi we-GitLab owayebikiwe ngomonakalo. 11 p.m.: ikhophi isale ngemuva kakhulu kangangokuthi eyokuqala isivele isusile
0:48 ilog elidingayo; ukulungisa okuwukuphela kwakho ukuvala ikhophi bese ukopisha eyokuqala futhi. I-pg_basebackup iyahluleka ngaphandle kwemiphumela. Empeleni ilindile, buthule, eyokuqala; akekho owaziyo lokho, futhi incwadi yemiyalelo ayisho. Unjiniyela, owayehlose ukuphuma ngo-11, unquma ukuthi uhla lwemibhalo yedatha engenalutho luyinkinga futhi ulususa. Ku-db1. Eyokuqala. Uyaqaphela ngemuva kwesekhondi noma amabili; cishe amakhulu amathathu ama-gigabytes,
1:11 4.5 asale. Amakhophi asenqolobaneni. Okokuqala: pg_dump ku-S3, nsuku zonke. Ibhakede lingenalutho. Umsebenzi we-cron usebenza kuseva yohlelo lokusebenza ngaphandle kwedatha, ngakho iphakethe likhetha I-PostgreSQL 9.2 binaries ye-9.6 database, iyahluleka, futhi ithumela ama-imeyili ukuhluleka, okubuyiselwa ngenxa yokushoda kwe-DMARC. Okwesibili: I-Azure disk snapshots, enikwe amandla kumaseva amafayela, hhayi ama-database.
1:32 Okwesithathu: ikhophi, esuswe ngamabomu ngehora eledlule. Okwesine: isifinyezo sansuku zonke, esineminyaka engu-24, wonke ama-webhook asuswe yi- ukuvumelanisa kwe-staging. Okwesihlanu: isifinyezo esenziwe ngesandla kusukela ngo-5:20, sokuhlolwa okungahlobene. Leso siwinile. Ukubuyisela kusho ukukopisha idiski ye-staging emuva ekukhiqizeni nge-Azure eshibhile isitoreji ngamashumi ayisithupha megabits ngesekhondi: amahora ayishumi nesishiyagalombili. I-GitLab dot com ibuyile ngoFebhuwari 1st ngo-6 p.m. UTC, idatha endala ngamahora ayisithupha.
1:58 git blame: amagama amabili eseva ahlukene ngohlamvu olulodwa, nezinhlelo eziyisihlanu zokwenza ikhophi yasenqolobaneni akekho owake ake abuyisela kusuka kuzo. Hhayi unjiniyela. Ukuhlolwa kokwenzekile, okusayinwe yi-CEO, kumenza angaziwa, kufaka umbala obomvu kumkhombandlela wokukhiqiza, futhi kunikeza ukuqina kwedatha umnikazi, ngoba kuze kube manje bekungekho noyedwa. Ubukhulu bokulimala: amahora ayishumi nesishiyagalombili phansi, amahora ayisithupha edatha isuswe, cishe amaphrojekthi ayizinkulungwane ezinhlanu, amazwana ayizinkulungwane ezinhlanu,
2:18 abasebenzisi abasha abangamakhulu ayisikhombisa, nabantu abayizinkulungwane ezinhlanu ababuka ibha yokuqhubeka. I-Hacker News inika idokhumenti elibukhoma amaphuzu angu-1,162 futhi icaphuna olulodwa umugqa kubo: kwamakhophi asenqolobaneni ayisihlanu, akekho. Isinqumo, ukuhlolwa kokwenzekile: ship it, ngempendulo. Bayenza isigameko esidlangalaleni, basola inqubo, futhi bashicilela uhlu lokulungisa ngezinombolo zezinkinga. Isenzo sangoMsombuluko: buyisela ikhophi yasenqolobaneni. Uma ungakaze uyibuyisele, awunayo.
2:42 Ngithumele isigameko ongakavunyelwa ukukhuluma ngaso, kumazwana, noma ku-the daily diff dot dev. Futhi yilokho umehluko wanamuhla. NginguNiko wase-Axrisi. Hlanganisa ngokuzibophezela.
Imithombo
- GitLab, "Postmortem of database outage of January 31" (Feb 10, 2017)about.gitlab.com
- GitLab, "GitLab.com database incident" (Feb 1, 2017)about.gitlab.com
- @gitlabstatus, "We accidentally deleted production data…"twitter.com
- @gitlabstatus, emergency maintenance noticetwitter.com
- Hacker News, "GitLab Database Incident – Live Report" (1,162 points, 598 comments)news.ycombinator.com
- Hacker News, the postmortem thread (377 points)news.ycombinator.com
