Failure Lab · offline editionOne file · no network · nothing saved or sent

2017

Database recovery

GitLab, 31 January

Replication has stalled. Choose the cleanup host, check recovery readiness and select a fallback copy. Then compare your modelled result with the postmortem.

Facilitator mode (timer, tally, printable debrief) is not available for this scenario yet.

Prepared with AI assistance. Published by CyberNative AI LLC. Corrections: hello@cybernative.ai.

Quickstart

  1. Before
    Choose Play solo. It works with a mouse, touch or the keyboard alone.
  2. 1
    Read the situation and make the calls with the evidence the team had.
  3. 2
    Run the simplified model on your choices.
  4. 3
    Read what the official record found, passage by passage, with its source tags.
Decisions
Two before cleanup, one during recovery
Record
GitLab postmortem, 10 February 2017

Sources

Source registerGitLab postmortem, 10 February 2017 · tags S1–S4

Primary source

GitLab, Postmortem of database outage of January 31, 10 February 2017. Official postmortem. Accessed 5 October 2026 using native web search and primary-page inspection.

Register

  • S1 — Wrong host and impact scope: Timeline, around 23:00–23:30 UTC; Root cause analysis, problem 1; opening incident summary; Data loss impact. Supports the accidental clearing of the primary data directory, the reported loss window from about 17:20 to 00:00 UTC, and GitLab’s approximate counts of 5,000 projects, 5,000 comments and 700 new user accounts.
  • S2 — Backup health and recovery testing: Broken recovery procedures; Database backups using pg_dump; Root cause analysis, problem 2. Supports the PostgreSQL version mismatch that stopped the logical backup, rejected failure emails and lack of an owner for regular recovery testing.
  • S3 — Copy freshness and purpose: Timeline around 17:20 UTC; LVM snapshots; Recovering GitLab.com. Supports the manually created snapshot about six hours before the outage to refresh staging, the snapshots’ staging purpose rather than disaster-recovery design, the nearly day-old alternate copy and GitLab’s choice to use the newer snapshot to reduce data loss.
  • S4 — Restore duration and affected data: Recovering GitLab.com; Data loss impact. Supports the roughly 18-hour staging-to-production database copy over throttled network disks at around 60 Mbps, and that Git repositories and wikis were unavailable during the outage but were not lost.

Model boundary

All alternative branches, effort and missing-write units are invented teaching devices and labeled illustrative. Units are not hours, bytes or historical loss estimates. The backup-exposure message appears on every assume path as a modeled warning; it does not claim that each such path causes historical loss. Checking recovery pauses deletion; it does not manufacture a working backup. Confirming the secondary preserves the model’s primary copy; it does not guarantee restored replication. The recovery-copy decision is shown only when the model requires restoration, and its two choices change the modelled missing-write window. Equal transfer effort is a simplifying assumption, not a comparison established by GitLab. No fatigue attribution or fast-restore claim is made. No command is executed. No live incident document was needed. No direct quotations are used.

Licence

MIT licence and noticeCovers the original Failure Lab software and documentation only. Third-party excerpts and linked records are excluded.

The MIT licence applies only to the original software and documentation created for Failure Lab by CyberNative AI LLC. Third-party material, including attributed excerpts reproduced in these files and linked inquiry, regulatory and postmortem records, is excluded from that grant and remains subject to its respective rights. The source registers identify the publishers and cited passages. No source PDF or complete third-party record is bundled.

MIT License

Copyright (c) 2026 CyberNative AI LLC

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.