# Debugging stories

Debugging stories are fun! This is a collection of links to various debugging stories.

## Non Fiction stories

[An unprecedented look at STUXNET, the world's first digital weapon](https://www.wired.com/2014/11/countdown-to-zero-day-stuxnet/)

[Atop segfaults](http://rachelbythebay.com/w/2014/03/02/sync/)

[Breakpoint takes 15ms](https://www.jwhitham.org/2015/04/the-mystery-of-fifteen-millisecond.html)

[Bug that hides from breakpoints](https://drewdevault.com/2014/02/02/The-worst-bugs.html)

[C64 Variable Screen Position crash](http://www.linusakesson.net/scene/safevsp/index.php)

[Chrome development branch crashes with x87 FPU stack overflow](https://randomascii.wordpress.com/2016/09/16/everything-old-is-new-again-and-a-compiler-bug/)

[Crash Bandicoot memory card corruption](https://www.gamedeveloper.com/programming/my-hardest-bug-ever)

[Crashes only on Wednesdays](https://gyrovague.com/2015/07/29/crashes-only-on-wednesdays/)

[Custom Rust OS crashes](https://jvns.ca/blog/2013/12/04/day-37-how-a-keyboard-works/)

[Debian package data corruption, but only on sparc64](https://web.archive.org/web/20060612203753/http://vger.kernel.org/~davem/cgi-bin/blog.cgi/index.html)

[Debugging an evil Go runtime bug](https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/)

[Debugging Insights](http://debuggingrules.com/?page_id=46) (collection of stories on separate pages, submitted to Dave Agans, the author of "Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems")

[Digital divider with wrong output](http://danluu.com/teach-debugging/)

[Email only goes 500 miles](http://www.ibiblio.org/harris/500milemail.html)

[How I cut GTA Online loading times by 70%](https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/)

[Impossible random number collision happened in practice](https://medium.com/@betable/tifu-by-using-math-random-f1c308c4fd9d)

[Incorrect sign masking operation](https://labs.spotify.com/2015/08/27/underflow-bug/)

[iOS file corruption](https://engineering.fb.com/ios/debugging-file-corruption-on-ios/)

[First Instance of Actual Computer Bug Being Found](https://www.computerhistory.org/tdih/september/9/)

[FIRST robotics dashboard segfaults due to finalizer](https://www.lukeshu.com/blog/java-segfault.html)

[Fixing A Bug In An 18 Year Old Shockwave Game](https://mattbruv.github.io/ccsr-bugfix/)

[FriendNet / WoW message corruption](https://blog.makandra.com/2010/04/the-greatest-bug-i-never-fixed/)

[FPGA / CPU timing bug](https://eli.thegreenplace.net/2003/10/30/hardware-debugging-is-hard)

[Gdb set follow-fork-mode child doesn't work](https://nanxiao.me/en/use-dtrace-to-diagnose-gdb-issues/)

[git - pack corruption post-mortem](http://git.661346.n2.nabble.com/pack-corruption-post-mortem-td7597765.html)

[Grep segfaults](http://blog.loadzero.com/blog/tracking-down-a-segfault-in-grep/)

[GWT test passes during normal runs but fails during coverage runs](http://ismail.badawi.io/blog/an-obscure-bug-story/)

[Java 8 webapp performance problems](https://engineering.indeedblog.com/blog/2016/09/job-search-web-app-java-8-migration/)

[JVM has 6-7 second GC pauses](https://www.evanjones.ca/jvm-mmap-pause-finding.html)

[Linux kernel CPU contention](http://notes.secretsauce.net/notes/2015/09/05_a-kernel-debugging-story.html)

[Login only works when sitting down (and other stories)](https://books.google.ca/books?id=kse_7qbWbjsC&lpg=PP1&pg=PA56). From Jon Bentley's _Programming Pearls_, section 5.10

[Mandrill stops sending messages](https://www.seancassidy.me/sherlock-holmes-debugging.html)

[Matrix transformation is 10x faster in a test harness than in the game](https://randomascii.wordpress.com/2015/01/19/knowing-where-to-type-zero/)

[Memory-corruption / threading bug](https://nanxiao.me/en/an-experience-of-fixing-a-memory-corruption-bug/)

[Mouse won't move while processes are dying](https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and-i-cant-move-my-mouse/)

[MS Money crashes](https://devblogs.microsoft.com/oldnewthing/?p=6103). Includes reverse engineered binary patch fix!

[Multicast to 237.0.0.1 acts craters performance](https://www.outerthoughts.com/2004/10/perfect-multicast-storm/)

[Network link instability due to DB caching policies](https://engineering.fb.com/production-engineering/solving-the-mystery-of-link-imbalance-a-metastable-failure-state-at-scale/)

[NFS bug hunting on GitLab](https://about.gitlab.com/blog/2018/11/14/how-we-spent-two-weeks-hunting-an-nfs-bug/)

[Node.js memory leak](https://www.tritondatacenter.com/blog/walmart-node-js-memory-leak)

[OpenOffice + Ubuntu|2009-04-28](https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161/comments/28) - "My wife has complained that OpenOffice will never print on Tuesdays" (bug is actually with the [file utility](https://bugs.launchpad.net/ubuntu/+source/file/+bug/248619))

[OS crashes on boot if machine was used recently](https://blog.valerieaurora.org/2013/12/17/heres-my-favorite-operating-systems-war-story-whats-yours/)

[PDP-11 crashes only when live cattle are being shipped from northern Ukraine and western Russia](https://www.jakepoz.com/debugging-behind-the-iron-curtain/)

[Perl can't read from TMPDIR?](https://blog.afoolishmanifesto.com/posts/investigation-into-why-perl-cant-read-from-tmpdir/) [(separate analysis of the same issue here)](https://blog.plover.com/tech/tmpdir.html)

[Polish S doesn't appear on medium](https://medium.engineering/the-curious-case-of-disappearing-polish-s-fa398313d4df)

[PS2 SDK doesn't zero the .bss segment](https://fobes.dev/general/2024/02/29/inline-assembly-dangers.html)

[Python hashes files differently on different days](https://dpb.bitbucket.io/unexpected-behavior-from-the-python-3-built-in-hash-function.html)

[Python/Flask/S3 absurd memory usage](https://www.jamesporter.me/2015/12/09/mysterious-memory-consumption.html)

[Register allocation algorithm in JIT uses extra register](http://bitfunnel.org/debugging-nativejit/)

[Script crashes before 10 a.m](https://darekkay.com/blog/script-crashes-before-10/)

[SIGALRM timer hangs](https://nativeguru.wordpress.com/2015/02/19/why-you-should-avoid-using-sigalrm-for-timer/)

[Signal fails only when returning from page fault handler](https://news.ycombinator.com/item?id=7684824)

[Signal handling on 286 emulator for Xenix](https://news.ycombinator.com/item?id=7684827)

[The Case of Veeerrry Slow Logons](https://techcommunity.microsoft.com/t5/windows-blog-archive/the-case-of-the-veeerrry-slow-logons/ba-p/724275)

[SSH sometimes fails from cron](http://mina.naguib.ca/blog/2012/10/22/the-little-ssh-that-sometimes-couldnt.html)

[Stack corruption when using select()](https://blogs.unity3d.com/2016/04/25/debugging-memory-corruption-who-the-hell-writes-2-into-my-stack-2/)

[Switch crashes machine despite not being connected](http://www.catb.org/jargon/html/magic-story.html)

[The Debug heap that created bugs
](http://lectem.github.io/windows/heap/appverifier/detours/2020/01/02/The-debug-heap-that-created-bugs.html)

[The Wi-Fi only works when it's raining](https://predr.ag/blog/wifi-only-works-when-its-raining/)

[Tmux/vim strangeness](http://www.daniellesucher.com/2014/04/24/my-new-favorite-vim-tmux-bug/)

[Tracking down a CPU bug in Solaris across multiple OS layers](https://gist.github.com/vladak/d7edf765fe2763b698b06488230d5886)

[uint32_t gets truncated to 16 bits](http://kdc-blog.blogspot.com/2008/03/one-day-one-of-my-co-workers-stopped-me.html)

[Unit tests fail when run in Australia](https://github.com/angular/angular.js/issues/5017)

[Windows crash dump of nonsensical instruction](https://devblogs.microsoft.com/oldnewthing/?p=43293)

[HTTP/HTTPS not working inside your VM? Wait for it.](https://rachelbythebay.com/w/2016/03/22/6nat/)

[The big handle gamble](https://x64dbg.com/blog/2017/11/04/the-big-handle-gamble.html)

[We had a unit test once which only failed on Sundays](https://qntm.org/unit)

## Fictional stories

[Coding Machines](https://www.teamten.com/lawrence/writings/coding-machines/)

## Contributors

* Adriano Gil
* Alexandros Tasos
* Arthur Axel fREW Schmidt
* André Luiz
* Bill Smith
* Brad Parker
* Bruce Dawson
* bussiere
* crazymerlyn
* Dan Luu
* Dan Nguyen
* Darek Kay
* Dima Kogan
* Dimitry (unknownexception)
* Florent Viel
* Howard Mao
* Jakub Kądziołka
* Jani Patokallio
* Jessie Weinstein
* José Manuel Díez
* Jouke Waleson
* Jyrki Vesterinen
* Kartik Agaram
* Kevin D. Clark
* Kirtan Gajjar
* Louis Vela
* Matteo Castellini
* Mark Jason Dominus
* michaelx386
* Nan Xiao
* Nathan Isom
* Nikola Brežnjak
* Pietro Menna
* Predrag Gruevski
* Rafik Draoui
* Ramon Fried
* Sean Cassidy
* Ty Lamontagne
* Vladimír Kotal
* James Porter

## TODOs:

* Improve formatting.

* Write up some interesting bugs: 40 hour hang, hardware bug caused by copying behavior of Intel engineering sample found on ebay, etc.