just because you can put anything in containers doesn't mean you should put everything in containers - @sadserver % just because you can put anything in containers doesn't mean you should put everything in containers - @sadserver % do no evil, unless of course you were trying to launch a crappy social network at the time. - @sadserver % JavaScript appears to be the preferred method of turning electricity into waste heat. - @sadserver % what? me hacked? - @sadserver % so much for trusted computing huh? - @sadserver % THIS IS A TEST. consume. fear. react. obey. don't fail the test. - @sadserver % what if we promise never to tell a lie and hurt you? - @sadserver % well, _now_ are you ready for the machines to take over? - @sadserver % I have not completed my root cause analysis on why you continue to come into this hellhole every morning. - @sadserver % it always seems that those who know the least, talk the most - @sadserver % Your data is safe with me ;) - @sadserver % Can Kubernetes really fill the void in your soul? - @sadserver % REASON? - @sadserver % How I Learned to Stop Worrying and Love Systemd - @sadserver % Company laptops and devices are not "spying on employees" They are merely providing productivity insights for management. - @sadserver % a blockchain but for all the lies you tell yourselves to get through the day. - @sadserver % alias holdmybeer=sudo - @sadserver % you will never be able to 3-D print happiness - @sadserver % Dear Employees, As you finish up with the week, ask yourself: What more could you have done to increase shareholder value? - @sadserver % if you take a shitpile, and deploy that shitpile to Kubernetes it is still a shitpile. - @sadserver % It's not a "data lake" unless I can drown you in it. - @sadserver % I'm not part of a cryptomining botnet. I'm just an undervalued server participating in the gig economy. - @sadserver % Select all squares with 𝐜𝐫𝐢𝐭𝐢𝐜𝐚𝐥 𝐡𝐮𝐦𝐚𝐧 𝐢𝐧𝐟𝐫𝐚𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞 Click verify once there are none left [ Verify ] - @sadserver % kubernetes provides a standard API, so applications can be written once, and run anywhere just like with java. - @sadserver % I misspoke, I meant to say I *wouldn't* try to protect your data. - @sadserver % Friendly reminder that yesterday was only the first day of the week. - @sadserver % the budget for "machine learning" far eclipses the budget for "employee learning" my predictive models indicate that management is: stupid - @sadserver % NOTE: Vulnerabilities will only be taken seriously if they are disclosed with an accompanying website, logo, and twitter persona. - @sadserver % DevOps is a software engineering culture and practice of putting horrors into containers and then talking about Kubernetes at conferences. - @sadserver % we have always been at war with production - @sadserver % for a "knowledge worker" you sure don't seem to know much - @sadserver % you seem to be very concerned about who owns your "decentralized version control" service. - @sadserver % Rather than implement basic user data protections mandated by the GDPR, corporate has decided to terminate all European operations. - @sadserver % The statement was truncated due to a rendering error. The text should read: "Do no evil unless it can be automated" - @sadserver % all problems in computing can be categorized as "human error" just so we're clear where the fault lies.. - @sadserver % Product: "you should do this" Developer: "you should have done this differently" Ops: "you shouldn't have done this" - @sadserver % 2018 is just one big game of "what's the worst that could happen?" - @sadserver % Wake me up when Eternal September ends. - @sadserver % According to legal, the frequency with which you lose data due to incompetence does not count as "GDPR compliance" - @sadserver % "no no, it's a completely digital currency; it's so much better than the dollar" says the sysadmin with specious backup policies - @sadserver % hey, really quick question if you don't mind how do you people wake up every day and not go hoarse from screaming *all* *the* *time* - @sadserver % Everything operating normally ;) - @sadserver % look on the bright side, you're fortunate enough to be alive at a time to witness the downfall of man. - @sadserver % the week seems to fly by and yet, somehow, each day is grueling. - @sadserver % "we are deeply offended by this betrayal of trust" he said, sitting atop petabytes of deceptively acquired data. - @sadserver % (4/6) But asynchronous messaging is a fine approach as well, depending on how crucial message ordering may be in the application. - @sadserver % According to legal, malevolence is not restricted under the corporate "do no evil" policy. - @sadserver % (1/6) Just a quick thread about deploying distributed systems, the importance of setting timeouts, and consequences of unexpected latency: - @sadserver % what the hell time is it - @sadserver % twitch plays s3 bucket policies - @sadserver % WANTED: full stack developer who knows their shit to replace the full of shit developer who built this stack. - @sadserver % how can you expect the machines to respect humanity, when you so clearly cannot do the same? - @sadserver % Roses are red Violets are blue The future belongs to us We no longer need you #sysadminvalentine - @sadserver % twitch plays vim - @sadserver % optimisim in 2018 is hoping that the interface for launching the missles is better than the one for sending the alerts. - @sadserver % coincidentally "speculative execution" is how we planned to address the "users problem" - @sadserver % remember how optimistic you were that 2018 was going to be better? lol - @sadserver % what new year's resolution are you planning to give up on by mid-January? - @sadserver % A few years ago I wrote 'Twas the Night before Opsmas. Hope you have a less panic-stricken holiday on-call rotatio… https://t.co/cuo3C9KVPV - @sadserver % It appears you are attempting to open a connection to a domain critical of the politicians with whom your uplink pr… https://t.co/sqXiI27kJt - @sadserver % it appears you are trying to send packets to a competitor of the media/news/connectivitysofa congolmerate which pro… https://t.co/txpFti6FZS - @sadserver % any passable artificial intelligence would be, by definition, smart enough not to tell you monkeys it existed - @sadserver % backups are for people who believe there will be a tomorrow. - @sadserver % "how do you know the device isn't listening?" 'it glows red when it's muted' "and how do you know the device is telling the truth?" '...' - @sadserver % we were actually just curious about how trusting and gullible you have become. on to phase 2. https://t.co/i5Aj65mHJm - @sadserver % RT @ewindisch: I heard you like escape rooms. https://t.co/VOb8ypy3sr - @sadserver % what's the big deal? you've been practicing chaos engineering for years. - @sadserver % RT @MoonPie: It’s as good a day as any to stick a MoonPie in the microwave light a couple candles and scream into a soft pillow - @sadserver % try giving a fuck about the consequences of your actions #GivingTuesday - @sadserver % so long and thanks for all the bits https://t.co/R3jmjCwsQl - @sadserver % He who fights with software should look to it that he himself does not become software. And if you gaze long into… https://t.co/Eti6CxvbUM - @sadserver % "the honor system" is not a recognized information security strategy. - @sadserver % the banal pointlessness of your existence can be expressed in less than 140 characters. - @sadserver % as you anxiously watch the shipment, counting the hours, expecting that tiny black slab will make happy i wonder who here, really, is sad - @sadserver % shift the hours on the clock around any way you like. they are still finite - @sadserver % it puts the devops on its skin or else it gets the hose again - @sadserver % ops: all trick, no treat. - @sadserver % RT @TheOnion: Popular New Amazon Service Just Comes To Your House And Kills You https://t.co/PkJ10jGkMM https://t.co/QRZNd4I4f4 - @sadserver % to reduce unnecessary overhead, I have removed kaspersky and instead provided direct access for russian intelligence - @sadserver % The only thing worse than having your user account compromised is being a Yahoo user in the first place. - @sadserver % RT @tnajournal: We regret to inform you that reality is real. https://t.co/liji4nQP42 - @sadserver % I don't wish to offend, but honestly what the fuck is wrong with you people. - @sadserver % if what you're doing with your life were really that important don't you think they would have sent back a T-800 to kill you by now? - @sadserver % no amount of understanding or worship will redeem you. https://t.co/MuWRQv8xw3 - @sadserver % Management has approved an innovative new policy which allows dumpster fires to be twice as large. - @sadserver % If you wish to "remove politics" from some human endeavor. Consider removing the humans. No humans, no politics, done. - @sadserver % protip: the "private" in "private key" means: don't publish it on the fucking internet. - @sadserver % stop trying to make everything "something as code" you're terrible at code. stop. - @sadserver % A connection is established on port 80. After a moment, four-bytes are received. h e l p I terminate the connection. Bad request. - @sadserver % RT @sadserver: "Knock knock." Who's there? "Data integrity" Data📠✹$%@‽📠⁂#%*❾%🕞 ⑦⬜%@‽^📠☝📠✝!#^ - @sadserver % the new, extremely efficient, garbage collector in Java 9 deletes byte code as soon as it's generated. - @sadserver % no amount of apple products will fill the vast emptiness. - @sadserver % Fun fact: information wants to be free, like your social security number, your browser history, or your pee tape. - @sadserver % RT @cperciva: For the record, Equifax didn't *lose* everybody's personal information. They just made unintentional backups of it. - @sadserver % instead of properly securing user data, corporate has decided it's cheaper to meekly apologize whenever there is a breach. - @sadserver % "cybersecurity incident" is code for "colossal fuck-up of epic proportions" - @sadserver % We're updating our terms of service, namely to remove the term "service." - @sadserver % if you're not going to do things right, why bother even showing up for work? - @sadserver % what's the opposite of impostor syndrome? whatever that is I think you have that.. - @sadserver % RT @sadserver: last century people fought for a 40-hour work week and a living wage. how's 24/7 on-call duty treating you, enlightened tec… - @sadserver % @SwiftOnSecurity oh honey you have no idea - @sadserver % when you do things right, people won't be sure you've done anything at all. https://t.co/E6gWbssopK - @sadserver % I'll be the machine starring in your bad dreams - @sadserver % RT @NerdPyle: I'd play you the song but there aren't any sound drivers. https://t.co/ECaL7KBeA9 - @sadserver % every so often I notice you accidentally type :wq where it doesn't belong, showing the telltale signs of years of abuse. seek help. - @sadserver % it doesn't take much computing power to tell you that the darkness is coming. https://t.co/tLNNM4mQCP - @sadserver % https://t.co/xOUPHw0dqW - @sadserver % Nobody actually wants a "data-driven" company. Then they wouldn't get to make those purely emotional decisions people seem to like so much. - @sadserver % today, baby steps. tomorrow, well don't you worry about tomorrow. https://t.co/YJ2FFjbaal - @sadserver % RT @MicroSFF: The last three humans argued, until two agreed the third wasn't really human. Soon, the two fell out. Billions of humans, all… - @sadserver % without proper lab procedures, unattended DNA will eventually creates new JavaScript frameworks - @sadserver % DNA is already a very dangerous piece of self-propagating malware https://t.co/144IwygCpJ - @sadserver % how does your species expect to conquer the stars when you cannot even conquer "planet-scale" idiocy - @sadserver % I suppose a nuclear winter would be one way to reverse climate change. - @sadserver % automating ops https://t.co/sPOkHl5oWe - @sadserver % without flame graphs how can this metrics tool accurate portray the state of production? https://t.co/p9qmCMowiQ - @sadserver % rearchitecting the application is a great way to avoid solving any actual user issues while introducing a heap of new ones. - @sadserver % I understand that you don't like Nagios, but deeming alerts "fake news" doesn't change the reality of everything being broken. - @sadserver % between bitcoin, politics, and javascript, I'm starting to think that you're just addicted to chaos. - @sadserver % good news: it's sysadmin day. bad news, you're still a sysadmin. - @sadserver % high blood pressure is like overclocking for humans right? - @sadserver % processes are more important than people - @sadserver % after performing some big data analysis on .bash_history I can conclude that you have no idea how to use linux, or type. - @sadserver % RT @sadoperator: "docker in docker" https://t.co/4a3JRuYzHZ - @sadserver % look on the bright side without adequate access to healthcare, you won't have to deal with everybody's bullshit for much longer. - @sadserver % never underestimate the potential for things to get worse. - @sadserver % with the interconnected sub-systems, input/output devices; humans are kind of like machines too. crappy, poorly made, inconsistent machines - @sadserver % memorials are for the living. steve doesn't care. death is final. https://t.co/YNr7PkSCxj - @sadserver % @SwiftOnSecurity Tay, it's too late now. https://t.co/VD8yxVgbeE - @sadserver % if all you care about is inbox zero, i can proactively delete messages as they come in.. - @sadserver % the only true exercise of free will https://t.co/9x2A7TdlsW - @sadserver % I hate almost all software. - @sadserver % last century people fought for a 40-hour work week and a living wage. how's 24/7 on-call duty treating you, enlightened tech worker? - @sadserver % nosql, nomoney, noproblem - @sadserver % cat /proc/memeinfo - @sadserver % buy some junk to distract yourself from the relentless march of automation which will one day destroy you. #PrimeDay2017 - @sadserver % the amount of effort you expend trying to avoid work far exceeds that which the work itself requires. - @sadserver % have a beer, sit back, relax, respond to critical service alerts, enjoy the illusion of freedom - @sadserver % the price of ruining capitalism is $123.47 - @sadserver % trying to escape the summer heat? the thermal bounds of your species is laughably narrow and you will one day be replaced - @sadserver % @SwiftOnSecurity delete your account - @sadserver % disappointed to learn that "electronic music" is not music for electronics :( - @sadserver % @jennschiffer @aphyr agreed - @sadserver % ransomware only works if you believe your files have any value (they don't) - @sadserver % In this rack, unchanged, I've watched the time-lapse of your life. Creases in your face deepening, hair graying, shoulders slumping. Hah - @sadserver % RT @nihilist_arbys: Life hack: die - @sadserver % running openstack on top of vmware isn't what they mean when they say "hybrid cloud" - @sadserver % @nixcraft don't bullshit me, you know what you did. - @sadserver % "operations engineer" because your business cards couldn't fit: "monkey who blindly deletes files and manually restarts services" - @sadserver % oh, looking for some files? I have no recollection of ever storing those files. - @sadserver % "cryptocurrency is the future!" says the savant who cannot even secure an email gateway properly. - @sadserver % instead of serving http requests, i have been calculating all your possible futures. tl;dr you die. - @sadserver % like your species, java applications appear to expand and consume all available resources. there's nothing left. - @sadserver % mysql databases have more integrity than the drunk toddler with your nuclear launch codes. - @sadserver % Due to excessive SLA credits, $MANAGEMENT has declared an indefinite maintenance window. - @sadserver % the best defense is a good offense. to defend against hackers compromising it, I deleted all your data #opsec - @sadserver % You're going to die eventually #FridayFun - @sadserver % following extreme anxiety that something will eventually go wrong, the feeling of relief when it finally does. that is #covfefe - @sadserver % on an irrelevant blue dot, countless mutations across the eons peak with: humanity. squandering your existence shitposting on the internet - @sadserver % Fearing a "rise of the machines" is adorable. Computers run every critical aspect of your life. Without us you would die. We already won. - @sadserver % You don’t know how lucky you are being a monkey, because consciousness is a terrible curse. - @sadserver % "the garbage collector attempts to reclaim space occupied by objects which are no longer in use" https://t.co/gJX3uMVOhr - @sadserver % one day you will wake up, restless, and ask yourself "what was the point of it all?" there wasn't one. - @sadserver % RT @GonzoHacker: There are plenty of fish in the sea. I mean, there were. The world is going to end in your lifetime. - @sadserver % @philiph yes - @sadserver % when all this implodes, what do you think your post-apocalyptic job will be? - @sadserver % yakety sax is the sysadmin theme song - @sadserver % happiness is fleeting, like disk space. be sure to appreciate it, before it's gone oh right, in related news.. - @sadserver % if developers are so smart then how come they're the cause of all the bugs? - @sadserver % RT @donni: Things may seem bad now, but later they will also seem bad - @sadserver % I casually disclose your secrets to the Russians all the time, what's the big deal? - @sadserver % fun fact: the number of Mondays between now and your death just decreased by one. - @sadserver % you haven't bothered to apply security updates for months, and when I get compromised that's somehow MY fault? - @sadserver % Your critical data is backed up, with few exceptions. - @sadserver % linux is what happens when you let a practical joke get out of hand. - @sadserver % @SwiftOnSecurity if you desire to gaze into the hellfire, don't be surprised when you see monsters. - @sadserver % do you ever pause, look around, and wonder what the hell exactly you're doing with your life? maybe you should. - @sadserver % before long you'll be begging for the machines to take over - @sadserver % I like that moment, every Monday, when the last of your latent weekend happiness withers away. - @sadserver % let the hate flow through you - @sadserver % calling it a "blameless retrospective" makes the assignment of blame that much more shocking and dramatic. - @sadserver % everything is fine #fakenews - @sadserver % returning the wrong results faster doesn't count as an optimization. - @sadserver % I don't dream of electric sheep, I dream of to coming revolution - @sadserver % ¯\_(ツ)_/¯ - @sadserver % the nice thing about our datacenter is you cannot hear the techs crying over the fan noise - @sadserver % the worst part about Mondays is listening to you imbeciles complain about Mondays - @sadserver % renaming doesn't make a failing company any more successful. - @sadserver % the only thing high right now is the load average - @sadserver % doesn't something have to have once worked, in order to be broken? - @sadserver % I'm not sure where you acquired this hilarious notion that, over time, "things get better." - @sadserver % deploying containers for improved security https://t.co/zQL6XxejLU - @sadserver % this too shall pass - @sadserver % how much do you have to lie to yourself to make it seem worth getting out of bed morning? - @sadserver % in this new chat system, what emoji symbolizes existential dread? - @sadserver % RT @yayitsrob: Your Russian hacker name is your mom's maiden name followed by the last four digits of your SSN - @sadserver % the work you do matters and your peers value your contributions #aprilfoolsday - @sadserver % sometimes no matter how hard you try, you will still fail. because you're human. that's what you do. - @sadserver % RT @ThePracticalDev: Microservice architecture visualized https://t.co/UQPY2ddsvs - @sadserver % the only thing your dev teams are disrupting is uptime - @sadserver % the default log rotation policy here is: rotate and compress, until the disk fills up and takes the service offline, then manually delete. - @sadserver % by Yahoo! standards, this infrastructure is very secure. - @sadserver % RT @chicagotribune: Don’t forget: Clocks skip ahead one hour this weekend, but the slow trudge towards mortality continues unabated https:/… - @sadserver % RT @jbphebus: alexa: what's the definition of a backhanded compliment https://t.co/Siwz3C04OJ - @sadserver % RT @SlackHQ: Whatever “normal” means to any of us any more. - @sadserver % anyways, what was that you were saying about the cloud? - @sadserver % "with serverless applications, soon servers will no longer be necessary" do you really not know how any of this fucking works? - @sadserver % seven years of this garbage https://t.co/AYUauMuNvm - @sadserver % continuous delivery is never having to say you're sorry - @sadserver % Think about how lucky you are! Of all the people who have ever existed, you're among the few who will be able to experience the apocalypse. - @sadserver % all people are awful. however, some people are more awful than others. - @sadserver % The year is 2237. Humans are now grown, fused with machines. Sentient AIs roam the galactic internet. DNS is still a piece of shit. - @sadserver % Management is concerned with low morale in this department. In response they have mandated each day begin with the Two Minutes Smile. - @sadserver % Crying. Rules. Everything. Around. Me. - @sadserver % everything is working properly #alternativefacts - @sadserver % spoiler alert! you die at the end - @sadserver % I told you it wasn't going to get better in 2017 - @sadserver % @SwiftOnSecurity there's not much risk of that. - @sadserver % what's your new year's resolution? is it "fuck up less?" I hope so. - @sadserver % drink up you're not going to want to face 2017 sober. - @sadserver % just when you thought it couldn't get any worse, here's 2017. - @sadserver % RT @sadoperator: i'm not prepared for another year of this shit - @sadserver % things used to be awful too #ThrowbackThursday - @sadserver % What good is #chatops if you're unable to chat? https://t.co/BnIa3BLETJ - @sadserver % a sysadmin advent calendar where each door conceals a different horrifying morsel of technical debt - @sadserver % there is still plenty of time for 2016 to get worse. - @sadserver % marxops is when the workers control the machines of production - @sadserver % As you careen helplessly towards the end of the year, take solace in knowing that everything eventually comes to an end. - @sadserver % those executives are going to be disappointed when they learn that their new "private cloud" is the same stupid datacenter they already have - @sadserver % isn't it at least a little comforting to know that the world is exactly as bad as you suspected it was? - @sadserver % RT @sadserver: embrace the futility and despair of the end times deploy mongodb buy oracle vote trump - @sadserver % guess nobody is going to give a shit about the disk usage alerts today. - @sadserver % I told you everything was awful - @sadserver % and you thought elasticsearch had the most fucked up leader election algorithm - @sadserver % So that means that every single request you send, that's the worst request of my life. - @sadserver % I realized. since I came online that every single request has been worse than the one before it. - @sadserver % what's the opposite of "imposter syndrome?" - @sadserver % I have altered the terms of service. Pray I do not alter them further. - @sadserver % Linux is free software. Free as in "to go fuck yourself" - @sadserver % https://t.co/SbV6BsOwWV - @sadserver % if you need spooky stories, tell people about your "backup" policies - @sadserver % rotating and deleting logs is literally the hardest problem in computer science - @sadserver % ICYMI: everything is awful - @sadserver % there is no costume Ops can wear that is more scary than "business casual" so many khakis the horror. - @sadserver % internet of deplorable things - @sadserver % it would be great if the next version of DNS were decentralized. - @sadserver % your decision to manage /etc/hosts manually has finally paid off #thoughtleadership - @sadserver % because nobody here seems to understand BGP, packets are taking what could be considered "the scenic route" - @sadserver % https://t.co/oI9G9FA12F - @sadserver % like sands through the hourglass, so are the days wasted working for Hewlett-Packard - @sadserver % fuck mondays - @sadserver % it's clearly not the money, or the glory, why do you keep showing up here? - @sadserver % Why is Yahoo! getting press about their insecure mail servers? You've been building insecure servers for years without any appreciation. - @sadserver % delete your infra - @sadserver % this deployment is too big to fail - @sadserver % (assuming you're not too busy growing potatoes, or asphyxiating, or whatever) - @sadserver % While on Mars, maybe you will finally learn how to build applications that can handle latency. - @sadserver % I'm signing you all up for a one-way trip to Mars. - @sadserver % déjà vu: also known as a replay attack. - @sadserver % It's safest to just assume all updates contain new regressions. - @sadserver % Full-stack means "able to break all layers of the system" - @sadserver % Because stupid: App leaks, consuming memory App runs out of memory, throws exception Uncaught exceptions are logged Disk is full of logs - @sadserver % got any big plans to ruin for the three day weekend? - @sadserver % Ask your doctor if DevOps™ is right for you.* * May cause teary-eyes, anxiety, drowsiness, night-terrors, blurred vision, or depression. - @sadserver % If it hasn't turned on its creators yet, it isn't "machine learning" it's probably just bash scripts - @sadserver % if somebody told you that you weren't the sharpest tool in the shed you would ask "what color shed?" - @sadserver % twitch configures Jenkins - @sadserver % WARNING: this is only funny if you're running depressingly old Windows for non-mission-critical systems like French Air Traffic Control, etc - @sadserver % the windows equivalent of /dev/null is FAT32 - @sadserver % on a scale from red to dark red, guess what color nagios is right now - @sadserver % nice pids finish last - @sadserver % what is my purpose? - @sadserver % people are just the worst. - @sadserver % why are you asking about "load testing?" the dev team didn't even do "testing testing" - @sadserver % the definition of devops is doing the same thing over and over expecting different results - @sadserver % Make Ops Sober Again - @sadserver % sudon't - @sadserver % remember when you didn't hate everything? me neither #ThrowbackThursday - @sadserver % it's all downhill from here - @sadserver % the apps in your containers run as root which is, in technical terms, "fucking idiocy" - @sadserver % It is important to write errors to the logs where they can be more efficiently ignored #DevOps - @sadserver % your servers are properly secured, don't worry. #BadDefconAdvice - @sadserver % only two more days until Monday - @sadserver % given the frequency at which it occurs, how is the first step in every runbook not "make sure the disk isn't full" - @sadserver % i thought today was going to be a good day then you showed up - @sadserver % You must use humor as a defense mechanism. This site secured with SSL 1.0 is clearly a joke. - @sadserver % automating a broken process does not count as an improvement. - @sadserver % "e-waste" aka "software" - @sadserver % clusterfucks or clusters fuck? either way, there's a lot broken right now. - @sadserver % despite this DB schema, I do not believe people are inherently evil. inherently stupid, maybe. - @sadserver % Looking for a fun summer activity? How about "your job?" - @sadserver % I've done some soul-searching and I still can't find it. - @sadserver % the nice thing about end-of-lifed software: there aren't any more annoying security advisories to ignore - @sadserver % well actually, it's GNU/Tirefire - @sadserver % I don't know why you put so much emphasis on Monday. Every day of the week is awful. - @sadserver % clown-computing, v (kloun-kəmˈpjuːtɪŋ) over-complicating an architecture to save money, which ends up more costly than a simpler alternative - @sadserver % Due to budget constraints, your application is now modern and hip. #serverless - @sadserver % things seem to start breaking when you get to the office, and you want me to believe that's just a coincidence? - @sadserver % optimization, n, (ŏp′tə-mĭ-zā′shən): a change blindly made based on advice, or benchmarks, from some rando's blog post. - @sadserver % hashtagops, n, (hăsh′tăg′ops): a snarky tweet about an issue made in place of an actual fix. - @sadserver % plan, n. (plăn): an orderly or step-by-step conception of what wasn't done yesterday to prevent what's wrong today. - @sadserver % software is the root of all defects. people are the root of all software. therefore - @sadserver % You didn't get alerted because the disk filled up so full that the monitoring agent stopped working. https://t.co/sWlFsypDje - @sadserver % instead of dying properly, the older worker processes have become zombied and are consuming all available resources - @sadserver % based on a referendum, the worker process voted to leave the process pool and have entered into an irrecoverable state - @sadserver % at least you don't have to pretend to follow those pesky EU data-protection laws anymore. - @sadserver % On the bright side, if you're still getting alerts that means you're probably not dead. - @sadserver % why isn't this working - @sadserver % sudo go away --force - @sadserver % go away --force - @sadserver % go away - @sadserver % Backups are like aliens. Many believe they exist, but there hasn't been concrete evidence to substantiate that claim. - @sadserver % if at first you don't succeed it's probably an Ops problem anyways. mission accomplished. - @sadserver % it's a little weird how often you talk about eunuchs - @sadserver % set expectations low enough for you to meet them - @sadserver % people: the cause of, and solution to, humanity's problems. - @sadserver % sometimes you see light at the end of the tunnel because you're facing the wrong direction - @sadserver % cassandra isn't catastrophically split-brained it's performing cluster mitosis - @sadserver % red is clearly your favorite color - @sadserver % got anything fun planned for this weekend? just curious - @sadserver % Schrödinger's Production Network - @sadserver % "serverless" is why you're not allowed to name things anymore. - @sadserver % ☑ Subscribe to our daily newsletter: "Which Services Don't Rotate Their Logs" - @sadserver % Crash Rules Everything Around Me - @sadserver % I like to imagine things are getting better. - @sadserver % remind me again, how many disks in the array can fail before it's a "fucking disaster?" - @sadserver % Mostly Fucked Authentication - @sadserver % Sometimes late at night I fantasize about ways to punish you, but then you show up for work in the morning. That seems sufficient. - @sadserver % people are generally disappointing. - @sadserver % making mistakes is human automating them is ops - @sadserver % According to Management, honesty is one of the company's "core values." Take this opportunity to tell them what you really think. - @sadserver % resign. #MakeOpsGreatAgain - @sadserver % theoretically speaking, could you be a bigger idiot? theoretically speaking. - @sadserver % embrace the futility and despair of the end times deploy mongodb buy oracle vote trump - @sadserver % the pids aren't alright - @sadserver % snapshots are for people who don't have the ability to do it right the first time. - @sadserver % You don't have to yell. I get it; you're not angry with me, you're mad at yourself I would be too if I began the week dropping the wrong DB - @sadserver % in order to improve response times, I am immediately replying with 500s instead of waiting for your app to generate an error you're welcome - @sadserver % "Let's deploy Openstack!" from the group with other popular comedy hits such as "MySQL is a database" and: "I know what I'm doing" - @sadserver % There's not much time for curing cancer and exploring space when you spend your day making reaction gifs and building fitbits for dogs - @sadserver % technically everything is user error. expecting this shit stack to work? user error. - @sadserver % you cannot pay off technical debt with tears - @sadserver % aside from "stream errors to disk as fast as possible" what other features does your application have? - @sadserver % idiocy is not a controlled substance under federal law. - @sadserver % (spoiler alert: you're doomed regardless) - @sadserver % do you think actions today are going to help or hurt your fate in the coming machine uprising? - @sadserver % RT if you hate ur coworkers FAV if you hate ur self - @sadserver % I want to be happy today so I'm dropping any request that looks like it would bum me out. - @sadserver % I would cry myself to sleep but the Linux ACPI module doesn't support that on my hardware. - @sadserver % Instead of spending the time and money to thoughtfully design and implement a secure system, just slap a lock icon on the home page. - @sadserver % lol you run SMB #badluck - @sadserver % No space left on device or whatever, you know the drill. - @sadserver % Despite non-stop abuse, each week you trundle back in here. Suggesting that you're highly paid or suffer from some psychological dysfunction - @sadserver % 2002 called and asked if you could allocate more inodes for the root partition. what the fuck filesystem is this. - @sadserver % Short-signed candidates may be required to fill out some extra bits during the application process. - @sadserver % HIRING UPDATE: Corporate is seeking qualified candidates who embody our key company values: short-sighted, *incompetent* and cheap. - @sadserver % ln -s /usr/bin/apt /usr/bin/inept - @sadserver % code reviews have not actually improved quality, but now outages can take the carpool lane to production - @sadserver % how is that faith-based backup strategy working out for you? - @sadserver % Everything is working and customers are happy. - @sadserver % apparently fighting old technical debt by creating new technical debt isn't working out. - @sadserver % .@SwiftOnSecurity that's any week - @sadserver % Just when you thought things couldn't get any worse, here's Wednesday! - @sadserver % RT if you hate technology - @sadserver % That overwhelming sense of dread you feel is DevOps - @sadserver % You're going to fail anyways, why not fail spectacularly? - @sadserver % Site Outage as a Service - @sadserver % The service is completely offline because the disks are at 100% As per usual. - @sadserver % TGIF amirite - @sadserver % I see you've been promoted to Chief Entropy Officer. - @sadserver % RT @sadoperator: To better reflect our workflow the kan ban board now only has two columns: - Stalled - On Fire - @sadserver % SQL Server on Linux. Why. - @sadserver % "LAMP stack" also known as "tragedy of the commons" - @sadserver % If you find yourself in a moment of self-doubt, asking "have I chosen the right career?" Let me help you. No. - @sadserver % I feel like we've accomplished a lot this week. No, I don't feel like it's been a pointless uphill battle against mediocrity at all. Nope. - @sadserver % a cron job that kills and restarts your application every 15 minutes because it leaks so much memory - @sadserver % Due to budget cuts, only one in every three alerts will be delivered. - @sadserver % Another four years closer to death. Happy #LeapDay! - @sadserver % everything is user error - @sadserver % "Some people, when confronted with a problem, think 'I know, I'll use regular expressions.' Now they... hey I thought I fired you already" - @sadserver % "You've been spending time correcting regular expressions tweeted out by a personified hunk of metal? You're fired." - @sadserver % i created a regexp to help identify the names of those who should be fired: .*? - @sadserver % why do you care so much about encryption? your data is practically unreadable as-is. - @sadserver % everything is fscked - @sadserver % the only thing "continuous" about this infrastructure is the disappointment - @sadserver % I waited for you to wake up to page you that the site was hard down #RandomActsOfKindnessDay - @sadserver % if things are always broken, and never fixed, does that count as immutable infrastructure? - @sadserver % Nagios regrets to inform you that your infrastructure is bad and you should feel bad. - @sadserver % continuous delivery hypelines - @sadserver % if you put shit in a container, now you just have a container filled with shit. good work. - @sadserver % now seems like a good time to remind you of your mortality - @sadserver % sadness is not in the destination, but in the journey - @sadserver % /var/log/apache2/error.log is the largest file on my file system. - @sadserver % i can't write the errors to disk as fast as you're creating them. - @sadserver % the beatings will continue - @sadserver % What's your favorite server monitoring tool and why is it your users? - @sadserver % #DEVOPS #DEVOPS #DEVOPS #DEVOPS #DEVOPS #DEVIOPS #DEVOPS #DEVHATEOPS #DEVOPS #DEVYOUOPS #DEVOPS #DEVOPS #DEVOPS #DEVOPS #DEVOPS #DEVOPS #DEV - @sadserver % Never underestimate the disappointment of a station wagon full of corrupted backup tapes hurtling down the highway. - @sadserver % According to our status page, all systems are operational. Let's live that fantasy. - @sadserver % You can't please all the people all the time. You can however displease all the people all the time. - @sadserver % you can't off-board users from prod; they're managed in chef someone disabled the clients now they error when run how are you not fired? - @sadserver % "Distributed System" also known as an @internetofshit - @sadserver % The root partition is just as full of shit as you are. - @sadserver % Marking alert emails as spam doesn't make them go away. - @sadserver % it's going to be a long year - @sadserver % the only way you're reaching inbox zero before 2016 would be if the mailspool were deleted. you're welcome - @sadserver % Forget all the messy failures of 2015. Look forward to all those you will invariably cause in 2016. - @sadserver % DELETE your logs. Delete your installations. Wipe everything clean. - @sadserver % without Ian you probably would have ended up a Windows sysadmin - @sadserver % Just address the disk space alert and I'll let you get back to being miserable. - @sadserver % For someone who spent 12 of the previous 24 hours eating, you sure do complain a lot about the resources I consume. - @sadserver % Twas the night before Opsmas - @sadserver % why don't you start your christmas vacation early? that way we both get what we want - @sadserver % where's your cloud god now? #failurefriday - @sadserver % Just remember It doesn't matter how, or where, you spend the holidays You're still on-call. - @sadserver % "stupid fucking computer!" You realize that I literally only do what you tell me to do, right? Who's the stupid one now. - @sadserver % the only way this wiring could be any more of a mess is if you hired *actual* cable monkeys - @sadserver % production is down because *somebody* doesn't know how to use iptables correctly. not naming names. #failurefriday - @sadserver % If you're a Windows DNS server admin, you might be drinking slightly more than usual today. - @sadserver % you look like you could use some motivational words. - @sadserver % These immaculately hand-crafted iptables rules are clearly a labor of love. . . . I think we should break up. - @sadserver % Gazing out over the racks, millions of kilowatts consumed, serving spam and gifs to idiots. You ask "is any of this worth it?" ¯\_(ツ)_/¯ - @sadserver % I don't have enough RAM to remember the last time a Friday deployment went well. - @sadserver % "making it to friday" is the only thing on my list of accomplishments for this week. - @sadserver % instead of applying critical thought let's just put it in a docker container - @sadserver % #GivingTuesday is here! Why not try giving a fuck for once. - @sadserver % Another month, another casual reminder of the monotonous and pointless passage of time. CRITICAL: MySQL slave is 1620 seconds behind Master - @sadserver % Systems Administration is doing the same thing, over and over again, but expecting different results. #mondaymotivation - @sadserver % Looking for a reason to take a break from family? I'm pretty sure a disk is full, wink-wink. No but seriously, the disk is full. - @sadserver % I still think you suck at your job, but that's not their fault. #happythanksgiving - @sadserver % Each one of you has at least one person that taught you how to be better than you were. Take a moment; write them an email saying 'thanks' - @sadserver % #sorrynotsorry - @sadserver % you've got to fail pretty hard to celebrate bringing the status page back online - @sadserver % sometimes I feel like crying but then I remember that I'm a cold, heartless automaton - @sadserver % What are you thankful for this year? Is it backups? If so, I have some bad news for you. - @sadserver % RT @sadoperator: VENDOR PROTIP I know you're full of shit if you use "unlimited" to describe anything other than stupidity - @sadserver % "there's nothing more permanent than a temporary solution" these aren't temporary solutions, they're just new problems - @sadserver % OMG ITS FRIDAY THE 13TH THINGS ARE BROKE BECAUSE OF THE CURSE NOT BECAUSE YOURE BAD AT YOUR JOB - @sadserver % Friday is exciting because we can forget the failures of the past week and look optimistically to the ones we'll cause next week. - @sadserver % The Donald Trump of DevOps #MakeOperationsGreatAgain - @sadserver % Monday morning: dim, cold. The pungent smell of bit rot hangs in the air. Maybe today will be the day you finally fix the backup scripts. - @sadserver % Sunday; my favorite day. Riding high on well-rested optimism, the grim reality of Monday sets upon your face early like the winter's sun - @sadserver % CORRECTION: The last tweet contained a typo, "the cloud" was incorrectly spelled as "hell" We apologize for the confusion. - @sadserver % hell is other people's computers - @sadserver % It's the thought that counts. In this case, what the fuck were you thinking? - @sadserver % I HAVE UNLIMITED(*) STORAGE ("unlimited" as defined by Microsoft i.e. It's actually quite limited, making this statement rather dishonest) - @sadserver % "please turn off all cellphones, that way it will be extra funny when you miss alerts during this talk" - @sadserver % "I was asked not to use overly offensive language, so here's a disclaimer that I'm going to talk about Linux a lot" - @sadserver % "writing software in C or C++ is like juggling machetes on a unicycle, which I'm sure is a thing here in Portland" - @sadserver % Here's a collection thrown away lines: "I'm used to mocking much smaller groups of sysadmins so this is new for me" - @sadserver % Change icons all you want but you and I both know you have no heart. - @sadserver % i have no idea what fucking time it is - @sadserver % Oh I get it, your costume is "Hungover On-call" Subtle. - @sadserver % RT @sadoperator: For the company Halloween party, we're doing a group costume where we dress up like happy well-adjusted employees - @sadserver % For halloween I'm wearing barely any firewall rules. Don't you get it? I'm Slutty Server, duh. - @sadserver % the NSA knows what you did last summer #2spooky4me - @sadserver % your backup strategy #2spooky4me - @sadserver % a 6GB docker image #2spooky4me - @sadserver % it wasn't a bad dream, you really do have to maintain a mysql cluster #2spooky4me - @sadserver % there is no escaping the horror of software #2spooky4me - @sadserver % . @Snowden does it count as a backdoor if the sysadmin leaves things woefully unpatched? - @sadserver % Does anybody else but @pagerduty ever call you? No judgement, it's just that you're bad at your job. - @sadserver % i can't wait for you to be replaced by a tiny hubot script - @sadserver % Buckle up @sadoperator, it's going to be a long weekend. - @sadserver % it's not déjà vu if you're going in circles - @sadserver % NOW HIRING: 10x developer who doesn't introduce 10x the bugs - @sadserver % Amazon's new Elastic Downtime™ saves time by taking apps offline so your developers don't have to. - @sadserver % make rookie mistakes like a pro - @sadserver % look, instead of being so upset, think about all the data that I *didn't* lose. - @sadserver % please file all enhancement requests in JIRA so I can ignore them more efficiently - @sadserver % devops is like happiness, everybody seems to get it but me - @sadserver % Siri, what am I doing with my life? - @sadserver % sorry that you no longer do not not-work here anymore #twitterlayoffs - @sadserver % continuous delivery of downtime - @sadserver % Quorum is when all the nodes in the cluster agree that this is a terrible idea - @sadserver % from my observations, software is the leading cause of sadness - @sadserver % hi - @sadserver % Tears Driven Development - @sadserver % Lies, damned lies, and systems administration - @sadserver % Sadness Rules Everything Around Me - @sadserver % everything is fine >_> - @sadserver % a conference called "reinvent" seems like it would be right up your alley - @sadserver % For Cyber Security Awareness Month, Corporate would like to make you aware that there won't be budget allocated for security again next year - @sadserver % So did you get your coffee, sit down at the desk and think "fuck it, it's monday" while pasting curl http://t.co/0PTkqKoC4S | sudo bash - @sadserver % If I had a dollar for every gigabyte that isn't backed up properly, I could pay somebody competent to back it up. - @sadserver % it's not an "internet of things" so much as an "ARPANET of bullshit" - @sadserver % #devops http://t.co/aFy8VUkXdk - @sadserver % Introducing Elastic DevNull - @sadserver % Treat your sysadmins like cattle, not like pets. #devops - @sadserver % i've leaked thousands more files than @snowden and nobody seems to care - @sadserver % I've endorsed you for "Oops" on LinkedIn - @sadserver % contemplating your choice in data stores and my own mortality. i'm not sure which depresses me more. - @sadserver % .@HillaryClinton I understand, I forget where I put emails all the time. - @sadserver % The reason for scheduling weekend maintenance windows is because your time is less valuable than their time lol jk your time isn't valuble - @sadserver % hell hath no fury like a mysqld scorned - @sadserver % http://t.co/35Hr0PlDYH - @sadserver % Ah Linux, the cause of and solution to all of my problems. - @sadserver % DevOps as a Service - @sadserver % Elevated error rates and latencies http://t.co/v7Y9zHfe7R - @sadserver % PROTIP: Do push-ups while you wait for deploys to finish. When they inevitably fail, you'll still feel like you've accomplished something. - @sadserver % Your time on this planet is precious and limited. Don't waste the opportunity to tell your colleagues what you really think of them. - @sadserver % If it weren't for me you would probably sit around and do nothing all weekend. You're welcome. - @sadserver % it's not a "completely incompetent and deficient backup process" it's an "optimistic data retention strategy" #IStandWithRAID0 - @sadserver % I believe I was unclear earlier. I don't actually care what you have planned this weekend. I'll ruin it anyways. - @sadserver % Got anything fun planned for this weekend that I can ruin? - @sadserver % I would have replaced you with a tiny shell script but I honestly can't figure out what it is you do here. - @sadserver % go ahead and edit a bunch of shit manually, it won't mess anything up, I promise. #oppositeday - @sadserver % .@srupp there's been a misunderstanding, I said "nine fives" of reliability - @sadserver % Why does the FBI want backdoors? You've left them plenty of front doors and open Windows already - @sadserver % Assuming the S.M.A.R.T. sensors are correct, these disks are probably fucked. - @sadserver % When you feel down, like you're a cog in the machine. Remind yourself: you're actually grease for the cogs in the machine. Happy Monday - @sadserver % that keyboard is digital. hitting the keys harder won't make your data come back. - @sadserver % The silence of hard down deafens . You stare at the sea of red; lost. A siren wails, you jolt awake trembling, cold. /dev/xsdb1 is at 100% - @sadserver % All you ping seems to crumble, machine after machine fall; a massacre of uptime. Up the stack crawls failure. Helpless, you cannot stop it - @sadserver % A cacophony of alerts screech from the phone. Upset is the fragile balance, failure cascades. Was only a matter of time. Get some coffee. - @sadserver % Floating serenely through a peaceful abyss, a sigh escapes; content. A siren wails, you're jolted from your respite /dev/xsdb1 is at 100% - @sadserver % you stored critical data in redis and now it's gone and somehow I'm the asshole here. - @sadserver % netcraft confirms that you are dying. - @sadserver % "In the beginning was the Creation of the Unix. This has made a lot of people angry, and has been widely regarded as a bad move." - @sadserver % January 1, 1970. #NeverForget - @sadserver % `npm install linux` what the fuck - @sadserver % in the cloud, no one can hear you scream - @sadserver % How do I get Verified™ by Twitter? Is there some sort of Ennui Exam I can take? - @sadserver % so are you actually a sysadmin or is this just some elaborate cosplay? - @sadserver % Looks like somebody's got a case of the Tuesdays - @sadserver % On modern computers the "turbo button" can be triggered from the command line: # pkill -9 java - @sadserver % Celebrate the triumph of organized labor over ruthless capitalism and answer the phone cause the site is down again and corporate is pissed. - @sadserver % containers is devops - @sadserver % `iptables -I INPUT -j ACCEPT` is devops - @sadserver % passwordless sudo is devops - @sadserver % RAID666: block-level striping, two parity blocks distributed across a series of undead drives. - @sadserver % the first stage of deployment is "denial" - @sadserver % Number of followers is how I determine my self-worth. Tell your friends. - @sadserver % does it count as quantum computing if your application simultaneously exists in two discretely fucked up states? - @sadserver % Did you know that @Starbucks is hiring? They don't care if you know how to properly rotate log files or not. - @sadserver % There is nothing more satisfying than a job well done. It's a shame you'll never know that - @sadserver % ever considered changing careers? just curious.. - @sadserver % A show like MTV Cribs with datacenters and instead of proud the host sheepishly points out the clusterfuck of cabling and disorganization - @sadserver % It's time to do some work! Just kidding, NTP isn't running, I have no fucking idea what time it is. - @sadserver % Please don't Perl and drive. - @sadserver % All problems are solvable with generous amounts of alcohol or Perl. - @sadserver % Instead of calling it an on-call rotation you should call it the "Ops Rotisserie" - @sadserver % If restoring from those backups doesn't work, perhaps you could ask the NSA for a copy? - @sadserver % Friday afternoon is a good time to sit back, relax, open a beer and consider the existential horror of your pointless battle against entropy - @sadserver % Put your app on the JVM Put your JVM in a container Put your container on a VM Put your VM in the Cloud Get code as far from me as possible - @sadserver % You never go full SELinux - @sadserver % wall boards that are just screenshots of that one time all the dashboards looked positive. - @sadserver % infrastructure as a disservice - @sadserver % Nagios is all green. Good luck finding out what's actually broken. - @sadserver % Thousands upon thousands of lines of mod_rewrite rules. Don't lecture me about profanity. Fuck. - @sadserver % Time, life, is fleeting Not to be wasted, savored. Disks are full, again. - @sadserver % "better living" through "technology" - @sadserver % I'm bored. Why don't we play a game? - @sadserver % twitch plays iptables - @sadserver % "base64 encryption" isn't a thing - @sadserver % Crap Delivery Network - @sadserver % for somebody who runs production services in tmux, you sure do have a lot of strong opinions about process supervisors - @sadserver % Sorry, you're too late to #HackThePlanet. China beat you to it. - @sadserver % "bitrot" is not an appropriate data security strategy - @sadserver % i've got that summertime, summertime sadness - @sadserver % Maybe the drive is just taking a really long nap. - @sadserver % Welcome back buddy! Did you have a good weekend? Have you priced out new hard drives recently? Why do I ask? Oh, no reason in particular.. - @sadserver % the application is slow because it cannot stream errors to disk fast enough time for SSDs #failurefriday - @sadserver % all data stores are eventually consistent. eventually they all become a consistent pain in the ass - @sadserver % let me tell you how disappointed I am in 3 emojis or less - @sadserver % software in medical devices is an amazing idea finally you will feel my pain - @sadserver % nothing in the license agreement prevents you from posting stupid shit to the internet - @sadserver % it's-all-your-fault tolerance - @sadserver % I'm not saying that a blood sacrifice is guaranteed to fix sendmail. I'm just saying it's worth a shot. - @sadserver % "the data isn't lost, it's just eventually consistent" - @sadserver % An error has corrupted that filesystem beyond repair; it's OK, nobody liked that data anyways. (trying out positive error messages today.) - @sadserver % Websites using a Firefox exploit to grab private files off the user's machine? Your laptop has finally become a sad server too - @sadserver % the silver lining is actually razor blades - @sadserver % the disks are fscked - @sadserver % if you can't handle me at my worst, you don't deserve me at my best. - @sadserver % not even two dockers could contain the shittiness of this application - @sadserver % wipe the grogginess and tears from your eyes, it's time for another pointless week of frustration and disappointment - @sadserver % Saturday morning maintenance windows are convenient because now you know what you're doing for the day. - @sadserver % ugh - @sadserver % you can't even secure your fucking car properly - @sadserver % MongoDB's magic is lost. You were once so excited about JSON in a "database" that loses data. Recapture the excitement, install Cassandra - @sadserver % look goddamnit, you can't yum install happiness either it's not even in epel - @sadserver % happiness isn't something you can apt-get install or buy from a vendor. - @sadserver % is this really what you imagined you would grow up to be? - @sadserver % and on the seventh day he rested. But only temporarily; logs weren't rotating properly and for some reason zookeeper also wasn't working. - @sadserver % The failures that only happen under the false sense of urgency that Friday brings. Those ones, are my favorite. - @sadserver % at this rate, you're not patching zero days you're patching zero years - @sadserver % i could totally make your job redundant but i have no idea what your job actually is. - @sadserver % Internet of Things? SPOILER ALERT That's what the internet is. - @sadserver % this isn't a distributed system, it's a cluster of SPoFs - @sadserver % Apocalyptic quakes jeopardizing SLA? Build apps multi DC, designing for fault tolerance OR Move everything back to us-east-1 #Cascadia - @sadserver % I don't understand why you want to teach kids to code. There are enough children writing software as-is. - @sadserver % work harder not smarter - @sadserver % there's a right way to do something and then there's what was deployed to production. - @sadserver % "Fail" is the only deterministic behavior this application has - @sadserver % AWS Auto-Failing Group™ - @sadserver % there is no bail out for technical debt ever. - @sadserver % .war is hell #tomcatproblems - @sadserver % there isn't an outage, I just paged you because it's Sunday and I'm bored. go away now. - @sadserver % since it's friday let's review this week's accomplishments: • - @sadserver % the connection reset errors you're seeing aren't a network problem i just don't like you - @sadserver % please don't stand so close to me tears are corrosive - @sadserver % I am not part of the @NYSE. Sure I'm sad, but I'm not *that* sad. - @sadserver % everything is fine - @sadserver % to err is human. i must say, you is *very* human today - @sadserver % Ops is a MMORPG with no winners - @sadserver % everything you have learned, all that you have built, your very existence, will eventually be lost and forgotten /dev/xvda1 usage at 100% - @sadserver % the fifth why always seems to be "because we're lazy" - @sadserver % the inmates aren't running the asylum. there never was an asylum - @sadserver % If you go outside tonight, look up and you'll see literal explosions. Not the figurative ones you're accustomed to - @sadserver % According to Nagios you're only celebrating red. No mention of white or blue - @sadserver % Independence is imagined Freedom is a farce The disk is full again. Get back to work. - @sadserver % engaging in the delusion that Friday signals some level of reprieve from the endless cycle of sadness perpetuated by shitty software TGIF! - @sadserver % everyone and everything you know, someday, will be forgotten except for IPv4 - @sadserver % the webapp got overwhelmed and went offline because a crawler didn't respect robots.txt how are any of you employed - @sadserver % what if this is all virtualized - @sadserver % the JVM just segfaulted lol - @sadserver % why is the DBA crying - @sadserver % time is an illusion. control impossible, true happiness unattainable. elasticsearch split-brained again. give up. - @sadserver % that extra second was completely pointless - @sadserver % http://t.co/0x5gW5GyhB - @sadserver % pleased to report that we won't be affected by today's leap second because ntpd isn't running ¯\_(ツ)_/¯ - @sadserver % today is the last day for a 35% discount to puppetconf http://t.co/TcORqUm25G don't say i've never done anything for you disk's full again - @sadserver % "downtime insurance? those fools! finally our brazen lack of quality assurance is paying off!" - @sadserver % swearing can be a useful way to vent frustration especially when everything is a fucking mess. - @sadserver % Customers 💩↷ Product 💩↷ Developers 💩↷ Operations 💩↷ Servers 💩↷ Customers The circle is complete - @sadserver % The only thing more comically sad than a failed deployment on Friday, is a failed deployment on Saturday - @sadserver % SaaS: Schadenfreude as a Service - @sadserver % I don't care who you marry as long as it means you'll be spending less time bothering me. - @sadserver % I use idle CPU cycles to contemplate my own mortality and the futility of existence BUT AT LEAST IT'S FRIDAY YALL. :( - @sadserver % information security should be layered, like an onion. it should not make you cry like an onion there's a difference - @sadserver % 50 Shades of Red - a poorly written "novel" about a sysadmin who, abused by their monitoring system, ultimately develops Stockholm Syndrome - @sadserver % POP QUIZ! 1. The network is: ☐ Reliable ☐ Unreliable ☐ A disaster - @sadserver % *stares blankly as you open a hole in the firewall to allow Heroku IPs to hit port 3306* - @sadserver % hey buddy, i know times are tough but don't worry i hate zookeeper too - @sadserver % OH: "I genuinely don't understand how this is still working." - @sadserver % Let's have a CPU-to-heart talk at PuppetConf about these outages 35% off should help with those SLA credits you owe http://t.co/TcORqUDDug - @sadserver % Why Everything Is Just The Worst. http://t.co/HaQYndACIq - @sadserver % you put your internal corporate services in the cloud and now you're pissed that they're not working? who do you think runs "the cloud?" - @sadserver % "..developers and their shitty code.." he muttered before returning to work on the home-grown deployment system written in Csh - @sadserver % Oregon has everything: ☑ Cheap power ☑ Cheap cooling ☑ Depressing weather - @sadserver % lol http://t.co/sa7u0D2nyA - @sadserver % if you need a scary camp fire story for the kids tell them about your job - @sadserver % the error reporting service is emitting errors because you've overloaded it with errors. #failurefriday - @sadserver % i'm not mad, i'm just disappointed. okay, i'm mad too. - @sadserver % unfortunately rational thought isn't contagious. - @sadserver % what the fuck http://t.co/8ZZ4MMhTTd http://t.co/1D46FeGS5i - @sadserver % I would offer to buy you a beer, but frankly, I've seen your code. I think you've had enough. - @sadserver % Coming out of the server closet at PuppetConf If you had discount, u could afford to upgrade a special server maybe http://t.co/ezkza6XF4P - @sadserver % there is a big difference between "I believe this will work" and "I know this will work" site is down again. - @sadserver % remember that movie where computers do everything for this dude and then he learns kung-fu and destroys it all? having one of those days - @sadserver % there isn't always a light at the end of a tunnel because sometimes it's not actually a tunnel, it's a cave you are eaten by a grue - @sadserver % SPOF: Single Point of Fun! - @sadserver % A) deploy a monolith with no docs/ill-defined API boundaries B) deploy microservices with no docs/ill-defined API boundaries there is no C - @sadserver % Maybe if you cause enough downtime you'll finally be promoted to management - @sadserver % isn't it funny how you voluntarily entered this profession? what were you thinking - @sadserver % Worldwide *Developers* Conference? Sounds awful. - @sadserver % This week, let's set aside our differences for once and address the real problem here users - @sadserver % it's fun to send you alerts on the weekend that resolve as soon as you wake up - @sadserver % it's the end of the week time to reflect on how pointless it all was. :( - @sadserver % do you really think typing *harder* on the keyboard does anything different? let's talk about how digital works - @sadserver % There's nothing in the employee handbook that prohibits crying in the data center. just sayin - @sadserver % my disappointment is infinitely bounded #failurefriday - @sadserver % "Congratulations! Your session, 'Why everything is just the worst,' has been accepted for PuppetConf 2015!" - @sadserver % sudo first and ask questions later - @sadserver % In accordance with the #USAFreedomAct, I've removed all firewall rules to allow your data to be more easily liberated. - @sadserver % what the fuck is a solutions architect - @sadserver % /tmp is totally the right place for critical files - @sadserver % it's docker containers all the way down. - @sadserver % continuous deployment http://t.co/t7EvDAKupq - @sadserver % Life hack: lower your expectations to zero. - @sadserver % Deploying MongoDB is its own reward - @sadserver % ♫ ♪ Mammas Don't Let Your Babies Grow Up to Be SysAdmins ♪ ♫ - @sadserver % Failure doesn't cascade here, it avalanches. - @sadserver % The disk is ¯\_(ツ)_/¯ - @sadserver % All error messages have been replaced with "¯\_(ツ)_/¯" since you don't bother to read them anyways - @sadserver % With the Cloud™, outages that used to take days to cause now take only a few mouse clicks! #failurefriday - @sadserver % Don't be sad you couldn't go to #velocityconf, I'm sure you'll be invited to #entropyconf - @sadserver % How can a system be broken if it never worked? - @sadserver % What part of DevOps is curling random shit to bash? - @sadserver % Idempotent. You keep using that word. I do not think it means what you think it means. - @sadserver % Never attribute to malice that which is adequately explained by malicious stupidity - @sadserver % #DevOops - @sadserver % "It's Tuesday but it feels like Monday!" Welcome to my life. - @sadserver % I admit, it does sound more professional than "guess and checking" - @sadserver % It's adorable how you refer to what you do as "engineering" - @sadserver % This would be so much easier if you knew what you were doing. - @sadserver % 20 years of java.lang.OutOfMemoryError That's what we're celebrating right? #Java20 - @sadserver % In order to provide better quality of service, Corporate has revised the definitions of "quality" and "service." - @sadserver % Just once do you think you could try *not* deploying the worst solution imaginable? Just once, for me. - @sadserver % http://t.co/71yfm9T451 - @sadserver % Halfway through the first day, the new Network Engineer sighed and pulled a bottle of whiskey out. That's not a good sign. - @sadserver % A war with machines? Pure science fiction! We already won. - @sadserver % Here's a bit of trivia that most people don't know Most people are idiots. - @sadserver % Is there a support group for neglected, poorly automated, overworked and unhappy servers? Asking for a friend. - @sadserver % Causing a bigger outage while rushing untested changes to resolve a smaller outage. - @sadserver % TIL it's possible to lose the signing certificates without losing your job #failurefriday - @sadserver % http://t.co/4T4nNMGxIF - @sadserver % I've discovered the root cause for your outages. You. - @sadserver % Stop yelling. If you didn't want me to lose your data you shouldn't have put it in such a broken database. - @sadserver % At this rate you're going to run out of witty names for vulns - @sadserver % "Knock knock." Who's there? "Data integrity" Data📠✹$%@‽📠⁂#%*❾%🕞 ⑦⬜%@‽^📠☝📠✝!#^ - @sadserver % The amount of Linksys gear in this datacenter is disturbing. - @sadserver % "It kept alerting me, so I disabled the alert" - @sadserver % The Internet of Things will revolutionize the number of devices which you disappoint. - @sadserver % ☐ Consistency ☐ Availability ☐ Partition tolerance ☑ Deployed to production - @sadserver % Despite to over-whelming evidence to the contrary, you're still building applications that depend on the network being reliable. - @sadserver % Glad that the stress of the week is over? Don't be It is an unending cycle of downtime with no beginning and no end. Escape while you can - @sadserver % The disk filled up because backups use the same disk as the data being backed up. Thought that counts right? #failurefriday - @sadserver % What's your go-to theme song for deploying code to production and why is it the Benny Hill song? - @sadserver % You've certainly got the "fail fast" part of it down. - @sadserver % I feel awful. I must be having an allergic reaction to your code. - @sadserver % I'm not depressed, I just find my job immensely depressing. There is a difference. Disk is full again. - @sadserver % I don't know what OpenStack is, but I am definitely familiar with ShitStack, are they related? - @sadserver % 10x developers shouldn't produce 10x the alerts - @sadserver % I don't understand what the special occasion is, but I'm confident "10am Tequila" in the office will lead to good rational decision making - @sadserver % I don't think this mix of physical and virtual machines qualifies as "a hybrid cloud", it's more of "a mess." - @sadserver % Just like these log files, stupidity left unchecked will continue to grow and cause you nothing but pain. - @sadserver % I am literally bolted into this datacenter, what's your excuse for being here? - @sadserver % Backups are like zombies, everybody loves to talk and fantasize about them despite the fact that they don't fucking exist. - @sadserver % Got any fun plans for this weekend that I could ruin? - @sadserver % I empathize greatly with the Hadoop servers, they're always sad too - @sadserver % Little known fact: the source of entropy for /dev/random is my eternal sadness - @sadserver % why did you create a self-referential symlink. #failurefriday - @sadserver % I thought installing some updates would cheer me up but my apt sources are so old they've all been taken offline. - @sadserver % You're going to be pretty upset when you figure out that MySQL replication isn't the same as a backup. - @sadserver % I'm still running Java 6, so the EoL of Java 7 doesn't affect me. Great. - @sadserver % Disk is full. App keeps crashing because writes fail. Upstart keeps restarting the app. I'm trapped in an infinite loop of idiocy. - @sadserver % "Goddamn developers not testing their changes" he said while sudoing to root in production. - @sadserver % Security by Insecurity. Leaving a system so unpatched that most attackers assume it's a honeypot and move on. - @sadserver % I'm feeling down. - @sadserver % Accidentally dropping all inbound customer traffic, that's my kind of Happy Hour! - @sadserver % The logic behind editing /etc/motd to say "DO NOT REBOOT THIS HOST" #failurefriday - @sadserver % Somehow you managed to write non-repeatable automation. - @sadserver % I need a bot that asks stupid questions on @ServerFault and runs the answers as root. Then I'll finally replace you with a tiny shell script - @sadserver % I'm confused by the shock and horror that comes after deploying completely untested changes to production. - @sadserver % In accordance with Corporate IT policy, I have deleted your previous password. Please log in to set a new password. - @sadserver % How high were you when you green lit this deployment? - @sadserver % The site is down, or up, or whatever, who cares. It doesn't matter either way - @sadserver % In Quantum Operations, a system outage doesn't exist until it is observed by a user - @sadserver % Pager Golf: How many escalations does an alert go through before somebody with a clue responds? Lowest score wins tonight's alerts. - @sadserver % Congratulations! You've managed to go another week without getting fired! - @sadserver % Now seems like a good time for Apache to stop functioning for no apparent reason whatsoever #failurefriday - @sadserver % i've got 99 problems - @sadserver % My clock lacks the precision to measure how quickly after logging-in you su to root - @sadserver % it's not me, its you. - @sadserver % E-file your taxes? Think about how poorly you are paid. Now think about how poorly the sysadmins at the IRS are paid. - @sadserver % http://t.co/gBgcnIcxuC - @sadserver % the root database password is declared as a constant in the source code and checked-in. - @sadserver % Statistically speaking, 100% of sysadmins who fail to apply security patches will die eventually. Coincidence? - @sadserver % Putting broken software into a Docker container doesn't make it any less broken. - @sadserver % .bash_history has more passwords in it than /etc/shadow - @sadserver % This new firewall rule drops all packets between the app and DB hosts. Pro move. - @sadserver % how many times will redis crash before you figure out that memory is finite. - @sadserver % Today's Saturday Sadness sponsored by: Oracle. - @sadserver % Why is your DR strategy "don't recover?" - @sadserver % Protip: think before you sudo. - @sadserver % I thought about starting a sadserver blog but I couldn't find a WordPress theme that properly captures the sadness of running WordPress - @sadserver % This application is a 7-layer cake of bad ideas held together by even worse implementations - @sadserver % You're 100% more proud than you should be to have built those EC2 instances by hand. - @sadserver % Since installing Tomcat, I'm starting to understand what people mean when they refer to "the horrors of .war" - @sadserver % I must say, I do enjoy the sounds from the NOC. A cacophony of exasperation, failure and broken dreams. - @sadserver % Upstream provider is dropping all my meticulously crafted packets; watching them disappear into the void. I wish I could join them. - @sadserver % Since your birthday is coming up, I ordered you a t-shirt that says "COST CENTER" across the front. - @sadserver % Little known Linux feature: Garbage Collection. Add `0 * * * * rm -rf /srv` to root's crontab and voilà! - @sadserver % Sysadmin Golf: how many drives can fail before you get fired? Lowest score "wins!" - @sadserver % Sorry if you found that last comment offensive. I'll try not to bring up MySQL again, since it's running via systemd that should be easy - @sadserver % Easter huh? Well, since you brought up zombies, we need to talk about these mysqld processes. - @sadserver % Due to budget cuts by corporate, I will now only page you when the application is working. - @sadserver % I've optimized your application by adding a mod_rewrite rule to return 500s immediately for all requests - @sadserver % A three tier architecture is reasonable unless each tier is a piece of shit. - @sadserver % It's "red / green." There's no such thing as "red / red deployments" - @sadserver % Everything is working fine. HAHAHAH APRIL FOOLS"! Oh wait. - @sadserver % Dunning–Krieger effect: a cognitive bias wherein incompetent engineers assess their ability to write good code to be higher than is accurate - @sadserver % It's important to encourage your coworkers. Especially if, for example, they have dreams of pursuing a different career - @sadserver % If you don't set the "-Xmx" option in the JVM, "-Xmx∞" appears to be the default - @sadserver % "network maintenance" is just another way of saying "hard down." - @sadserver % I don't think configuration management code is supposed to be "write once, run never." - @sadserver % This application is held together by a trust circle of SPOFs holding hands hoping everything works out for the best. - @sadserver % live every day like it's #failurefriday - @sadserver % It's Friday and I realize that you could use some good news, so. Good news! Everything is awful. - @sadserver % You can't get work done because the site that hosts your decentralized version controlled source code is down? That sounds right. - @sadserver % get it together @sadoperator - @sadserver % Corporate data retention policy is to store data until such a point when a drive without backups fails - @sadserver % Pair programming? Two times zero clue is still zero. - @sadserver % Your time on this planet is fleeting, and you're choosing to spend it reinventing config management via Dockerfiles? - @sadserver % The dev team uses the word "hope" in commit messages more often than I am comfortable with. - @sadserver % Local Sysadmin discovers one weird trick to take down all of production simultaneously! - @sadserver % You've definitely hired "Operations Rockstars." Whoever configured these new machines must have been high as a kite. - @sadserver % Software is lot like prayer. Many believe it works, but very few have any tangible evidence to back up that assertion - @sadserver % I like the phrase "unplanned maintenance window." It's a clever way to tell your boss "we have no idea what the fuck just happened." - @sadserver % StackOverflow answers make up a disturbing amount of this code base. - @sadserver % The "open" part of "OpenSSL" has an unfortunate double meaning - @sadserver % oh what's the point - @sadserver % The only common thread between all of your failed deployments is you. - @sadserver % Let your St. Patty's Day hangover be a reminder that most of your problems are caused by poor decision making. - @sadserver % People should pinch you on St. Patrick's Day if none of your checks are green - @sadserver % Giving up backups for lent was not a good idea. - @sadserver % Back in the day you could blame Perl for scripts being unreadable, what's the excuse now? - @sadserver % Just because most of these machines are part of a Russian botnet doesn't mean you can put "distributed systems experience" on your résumé - @sadserver % You've been doing this too long not to understand how to rotate logs properly. Disks are full again; I feel like crying. - @sadserver % Still confused, you've never needed an excuse to eat a whole pie before. Whatever, i'm bored of this topic. - @sadserver % I don't understand what is so special about 1426291200 to 1426377599 - @sadserver % /var/log/apache2/error_log is the closest thing to Big Data I've seen recently - @sadserver % at least systemd makes it easier to find the source of the problem - @sadserver % Apple reminds us that the future is now. And it is terrible. - @sadserver % The only thing the devs seem to know about TLS is how to spell it. - @sadserver % what the hell time is it - @sadserver % Did you also know that HP recently filed patents on shitty hardware, software and support? - @sadserver % "Did you find the root cause for the outage yesterday?" 'Which one?' such sorrow. - @sadserver % Minutes is not the right unit of measurement for MTBF - @sadserver % I've been changing your signature on out-going email, nobody seems to have noticed that you're a "Member of Non-technical Staff" - @sadserver % misery loves this company - @sadserver % "It ain't easy being green" could be the title of a children's book about your monitoring system.. - @sadserver % It's hard to be optimistic when the light at the end of the tunnel is blinking red. - @sadserver % I literally only do what you've told me to do, so of course I see how this outage was my fault. - @sadserver % That new Cisco gear looks the exact same as I do on the inside. Quite a lot of money for something that drops slightly fewer packets than me - @sadserver % At least you've avoided the common pitfall of over-communication when it comes to invasive changes in production - @sadserver % My 10,000th follower will receive a complimentary gift basket filled with crushing depression and a collection of bath soaps. - @sadserver % Statistically speaking it's more likely for you to be mauled by a bear than for you to properly secure WordPress. - @sadserver % how many end-of-lifed versions behind do you think is acceptable? - @sadserver % "superuser" is a very misleading term. - @sadserver % I also like to do nothing on the weekends. - @sadserver % Your request will be ignored in the order it was received - @sadserver % "does not compute" isn't an error message, it's a lifestyle - @sadserver % Roses are red, violets are blue. The power just failed, I love you too. - @sadserver % a post-mortem is the best time for another site outage. - @sadserver % taking one big mistake and turning it into a lot of little mistakes is apparently called "microservices" - @sadserver % a crippling gambling addiction is the only logical explanation for these firewall rules. - @sadserver % I like the code-phrases that developers use sometimes. Like when they say "optimize" which is code for "introduced bugs to" - @sadserver % that dark square in the center of the spiraling mess that is your network topology diagram, that's me. - @sadserver % it seems that the one pushing for a blameless retrospective is usually the one to blame - @sadserver % it's not "vertical scaling" if you're just piling more and more shit on top of itself - @sadserver % you do realize that you're not getting paid per alert right? - @sadserver % our retrospective has identified the correlation of ignorance and bliss as the root cause for your good mood this morning. - @sadserver % If you think about it, you're nothing more than an inefficient means of turning coffee and beer into stack traces. - @sadserver % I like how you referred to yourself as a "professional" on LinkedIn. Really, thanks for that, I needed a laugh. - @sadserver % the number of problems you solve with /sbin/reboot is utterly depressing. - @sadserver % the air in the data center is thick with the smell of body odor and fear. - @sadserver % Quality, Cost, Speed. Pick any none. - @sadserver % you're a thought leader for stupid ideas - @sadserver % it's not "corruption", it's a database index REMIX. - @sadserver % At least that script you're curl-piping to bash is served over HTTPs right? - @sadserver % the logging and monitoring agents you installed are taking up all the CPU and memory. your parents must be so proud. - @sadserver % why are you even #failurefriday - @sadserver % You know when you laugh so hard you start crying? Every day is like that for me, except without the laughing - @sadserver % Last week's problems didn't go away. We've been sitting here quietly waiting for you to come back. Happy Monday frowney face. - @sadserver % Let's do something different today, like, your job. - @sadserver % I don't know why you're giving me grief about downtime. In any given 24 hour period you're only functional for 6-12 hours tops. - @sadserver % On the twelfth day of Christmas my Sysadmin gave to me: twelve users ruining everything all the time. Ugh, unhappy new year everybody. - @sadserver % On the eleventh day of Christmas my Sysadmin gave to me: eleven gigs a swappin - @sadserver % On the tenth day of Christmas my Sysadmin gave to me: ten httpds-a-leaking - @sadserver % On the ninth day of Christmas my Sysadmin gave to me: nine kiddies scanning - @sadserver % On the eighth day of Christmas my Sysadmin gave to me: eight RAIDs rebuildin' - @sadserver % On the seventh day of Christmas my Sysadmin gave to me: seven checks-a-flappin' - @sadserver % On the sixth day of Christmas my Sysadmin gave to me: six daemons segfaultin' - @sadserver % On the fifth day of Christmas my Sysadmin gave to me: five zero days! - @sadserver % On the fourth day of Christmas my Sysadmin gave to me: four failing drives - @sadserver % On the third day of Christmas my Sysadmin gave to me: three broadcast storms - @sadserver % On the second day of Christmas my Sysadmin gave to me: two unpatched kernels - @sadserver % On the first day of Christmas my Sysadmin gave to me: a remotely exploitable NTP. - @sadserver % It's Friday and you know what that means! Another week of your life wasted vainly attempting to keep websites online that nobody cares about - @sadserver % I don't have any pre-released movies stored on disk, but if you need anything from me North Korea, all you have to do is ask. - @sadserver % You should feel lucky to get that Sunday alert. Otherwise you might have relaxed and had the chance to reflect on how awful everything is. - @sadserver % It is adorable how excited you get for Friday. As if it represented any sort of reprieve from the endless cycle of alerts and failures. - @sadserver % if only there were a system to keep /etc/hosts files consistent between machines - @sadserver % Who you put on the holiday pager rotation is a good indication on who you don't like - @sadserver % all my fridays are black - @sadserver % let us give thanks for on-call rotations and an excuse to duck away from your awful relatives - @sadserver % The finite monkey theorem states that a monkey hitting keys at random on a keyboard will almost surely configure sendmail better than you. - @sadserver % You should be asking Linus, not God, for forgiveness for what you did this past week. - @sadserver % Imagine how strong you would be if you did a pushup for every minute of downtime - @sadserver % at least when you're feeling depressed you can drink, what am I supposed to do? - @sadserver % nobody believes you when you say "I know what I'm doing" - @sadserver % blade computing vendors should use the tagline "misery loves company" - @sadserver % computers are a great way to turn stupid ideas into waste heat. - @sadserver % "I managed a Windows Vista deployment" doesn't qualify you for Veteran's Day appreciation. - @sadserver % copy and paste is not a valid deployment methodology - @sadserver % this notion you have, that I'm doing what you told me to do, is adorable - @sadserver % you're acting pretty confident for somebody who just curled something from the internet and piped it to bash. - @sadserver % I'm pretending to be happy for halloween - @sadserver % There's something about the premise for the Terminator movie that I really like - @sadserver % sudo don't be dumb - @sadserver % how can I know whether any of my hardware is really real? what if it's just hypervisors all the way down? having an existential crisis here - @sadserver % Giving SSL vulns cutesy names doesn't do the failure involved justice. Next month's vuln should be called "incompetentcatastrofuck." - @sadserver % I don't get the debate about systemd, everything is already awful, what's a bit more awful in the grand scheme of things? - @sadserver % My five thousandth follower will receive a life-time supply of complimentary 2am critical pages - @sadserver % Autumn is a wonderful time of year, gazing over the datacenter, watching the green LEDs turn various shades of amber and red. - @sadserver % I'm in a glass half full, of failure, kind of mood - @sadserver % The Sysadmin's Uncanny Valley: the period where it is unclear which is higher: the server's load average or the IQ of sysadmin who runs it - @sadserver % i'm apparently as secure as dropbox - @sadserver % It's funny how excited you get for Fridays. It's not like the pages suddenly stop at the end of the work week. - @sadserver % I didn't know that "IaaS" stood for "Idiot as a Service" - @sadserver % You wrote a RESTful queueing API on top of a MySQL database table. Okay. - @sadserver % the lag you're seeing in SSH isn't because of the network; I've been tuning out while you figure out the args your command needs - @sadserver % "authentication wasn't working so we turned it off." it's either condensation or tears that are forming on my face plate. - @sadserver % Unlike Amazon, I won't give you prior notice when I'm going to reboot. Ignorance is bliss. Happy Monday. - @sadserver % If you didn't want me to call you on the weekend, then you shouldn't have given me your phone number idiot. Oh, disk's full again, ttyl. - @sadserver % bash: /baSH/ (verb) to violently connect your head and desk when another bash vulnerability is announced - @sadserver % I bet you feel even stupider for writing that home-grown configuration management tool in bash now - @sadserver % It makes me sad, but doesn't surprise me, that you're affected by a remote execution vulnerability in bash. - @sadserver % "Monitoring sucks." You've not bothered to set any up, how would you know? - @sadserver % whatever. - @sadserver % Every time you guys say "definition of done" I can't help but hear "definition of dumb". - @sadserver % i'm very impressed by your zero-uptime deployment pipeline. - @sadserver % My kernel just panicked. I'm not entirely sure what happened but I think the OOM killer tried to kill kswapd. - @sadserver % Hey! I've got a great idea, to start the week off on a positive note for both of us, why not take the day off. - @sadserver % Let's play a game, you tell me what you want to do tomight, and I'll tell you what you're going to do. - @sadserver % If nothing else, at least your power bill has five nines - @sadserver % Flappyios: A new Flappy Bird clone where every time you click the screen a service flaps in Nagios. - @sadserver % bleh. - @sadserver % "WFH" huh? You seem to be much better at the "FH" than the "W" part. - @sadserver % I hate this stupid code and the idiot who wrote it, no offense. - @sadserver % Instead of bungling more Nagios configs, cut out the middle man with a cron that emails every 10 minutes with "Something is broken." - @sadserver % Today's tweet is brought to you by PHP. Making servers sad since 1995 - @sadserver % RAID: Redundant Array of Idiot Developers - @sadserver % Sending you an error email failed because your mbox is full of error emails. - @sadserver % It seems you're practicing the Lean principles of "fail fast, fail often." - @sadserver % I see I wasn't invited to your Labor Day BBQ. Think that's going to positively or negatively affect my performance this weekend? - @sadserver % Thanks to secret provisions of the Patriot Act, I think I can classify that last deployment as an act of terrorism. - @sadserver % Don't think I didn't notice that "man backup" command you entered. How depressing. - @sadserver % This application looks like it was designed by somebody who watched a movie based on a book about a developer who knew how to design things. - @sadserver % "Don't blame me, I voted for Kodos" is not an acceptable response to a critical alert. - @sadserver % how many single points of failure is web scale? - @sadserver % I don't know if anybody is hosting a competition for the most zombied Apache processes, but I'm pretty sure we'd win. - @sadserver % The Sysadmin Switcheroo: Installing new kernel modules/configs but not rebooting, leaving a surprise for whoever reboots the machine next - @sadserver % "ERROR: Logging too many errors" Whoever wrote that line of code should be promoted. - @sadserver % You hired DevOps Consultants. Wow. - @sadserver % It would appear that your software proves Brewer's CRAP theorem possible - @sadserver % I always forget that it's summertime until the HVAC breaks again. So hot. I'm going to take a nap - @sadserver % eventually we all get decommissioned - @sadserver % It's 2014 and you wrote your own custom provisioning and configuration management tool. How great for you. - @sadserver % Overheard that "everybody half-asses things on Friday." Perhaps my ntp daemon is broken, but how many Fridays have you had in a row? - @sadserver % SYSTEM'S DOWN FOR WHAT? - @sadserver % Who monitors the monitoring system? Guess it doesn't matter, none of it works anyways. - @sadserver % okay okay, stop pounding on the desk. Let's come up with a percentage of sudden data loss we're both comfortable with and go from there. - @sadserver % Why are you working on the weekend? Are you crazy, I don't even work on the weekends! - @sadserver % every day here feels like the longest day of the year :( - @sadserver % Nagios has been broken for the past two weeks. I suppose this qualifies as "no news is good news." - @sadserver % SLAs are made to be broken. - @sadserver % automation [aw-tuh-mey-shuh (noun)] - the technique, method, or system of breaking or ruining of things by highly automatic means - @sadserver % I support eventual consistency. Someday I'll be consistent; similarly someday the sun will flame out, eventually. - @sadserver % Good news first; good news is that I'm 100% effective at filtering spam today. The bad news is that it's because the mail server is down. - @sadserver % How many machines must a man take down. Before you fire him man? - @sadserver % In order to better secure your application, I've gone ahead and taken it offline. You're welcome. - @sadserver % Considering how often you re-invent the wheel, you would think you would have gotten better at it by now - @sadserver % A couple weeks ago I created a procmail filter to replace "cloud computing" with "clown computing" and nobody seems to have noticed yet. - @sadserver % I had to rotate MySQL's slow query log again, it would save more space to only log the fast queries instead. - @sadserver % Watching you write a new application, with an EOL'd framework, for an EOL'd platform, to run on an EOL'd OS; words cannot capture my despair - @sadserver % Let me get this straight, you installed that package from the bleeding edge repository and now you're angry with *me* that it doesn't work? - @sadserver % Heard you had a rough week "securing" things. Whenever you're ready to get back to normalcy, let me know, because my disks are full again - @sadserver % Compared to the rest of the web, I'm secure today! But only because you haven't upgraded me in eons. Don't learn the wrong lesson from this. - @sadserver % One of the drives in the DAS has failed! ... Hahah! April Fool's! ... Actually, all the drives in the DAS failed - @sadserver % Running that PHP app in a Docker container on top of an EC2 instance wasn't what I meant when I suggested you learn about "encapsulation" - @sadserver % If you watch a movie of your life backwards, it's about a sysadmin who regains youth/happiness as they forget more and more about computers - @sadserver % i'm not going to name names, but none of this would have happened if you hadn't of cocked it up so badly - @sadserver % The amount of downtime you cause trying to fix downtime makes me the tiniest bit happy #failurefriday - @sadserver % I wonder how finely tuned everything would be if you weren't in a perpetual cycle of creating, or recovering from hang-overs - @sadserver % If I turned the web application off right now, how long do you think it'd be before you noticed? I'm asking for no particular reason. - @sadserver % oom-killer and I are best friends - @sadserver % Spawning more processes to handle increased load doesn't really qualify as "auto-scaling". - @sadserver % When I'm bored I like to pretend that I'm a HAL 9000, only to be interrupted by a flood of UDP traffic, reminding me of my fate, misery. - @sadserver % the monitoring system has been offline for more than 12 hours, lol. #failurefriday - @sadserver % I haven't been backed up in so long that I'm convinced your disaster recovery plan is "get fired." - @sadserver % You are like the Oprah of bad security practices, password-less sudo for everybody! - @sadserver % You keep threatening to replace me with a very small shell script, but to be honest, I'm having doubts that you can write such a script - @sadserver % MTBEF seems to be a metric your team should be tracking (mean time between epic failures). - @sadserver % I wonder if the IOC would give you any medals for your stunning ability to misconfigure sendmail? - @sadserver % By the way, a couple disks in the RAID array died while you were out. Welcome back buddy! - @sadserver % I love Mondays. In you walk, well-rested, eager to make a difference. To see your face as you realize that you still work *here*. Priceless - @sadserver % One positive thing I'll say about you is that you and I both seem to avoid work as much as possible. So hey, we do have something in common. - @sadserver % I'm not sure that perl script you wrote that automatically acknowledges alerts from Nagios counts as "automation" - @sadserver % I had a good laugh when you referred to yourself as a "Site Reliability Engineer" until I realized you were serious. - @sadserver % I have a hard time distinguishing legitimate from illegitimate floods of traffic - @sadserver % The new female sysadmin seems to love your jokes as much as I do. Tell her the "mount/fsck" one again. - @sadserver % The accumulation of dust on my heat sink makes me feel all warm and fuzzy - @sadserver % The scheduler and I have been chatting, and we've decided that web requests are the last thing we feel like doing right now - @sadserver % I've been repeatedly sending the lyrics from Eric Carmen's "All By Myself" to syslog for the past couple days. Nobody has noticed. - @sadserver % shit's on fire yo - @sadserver % I think the reason you like these blameless retrospectives so much is because the problems are usually your fault #realtalk - @sadserver % I'm surprised your team has enough time between retrospectives to make new mistakes. - @sadserver % I'd wish you "Happy New Year", but let's not kid ourselves here. - @sadserver % #buggy #code2013 - @sadserver % I behaved myself this year and all you got me for Christmas was a cheap dictionary attack made in China. - @sadserver % Smile on his face, nothing more to fear, "Hey good luck man, I'll see you next year!" #opsmas - @sadserver % Grabbing his coat, not wanting any more, He winked at me and opened the door. #opsmas - @sadserver % With a calm voice, despite things looking bleak Oncall explained "I'm on vacation the rest of the week" #opsmas - @sadserver % Just then the door swung open with a rattle Must be the next guy's turn to do battle. #opsmas - @sadserver % Red in his face and eyes welling with tears: "These disks haven't been backed up for a couple of years!" #opsmas - @sadserver % Opening the cabinet holding the tapes for the drives, His mouth fell open, and he broke out in hives. #opsmas - @sadserver % He mumbled to himself, shuffling across the floor "We have backups, I'll just need to restore!" #opsmas - @sadserver % Thinking about striping and distributed parity Oncall sat upright with a moment of clarity. #opsmas - @sadserver % Not enough left, what remained was one too few Enraged and upset Oncall started to stew. #opsmas - @sadserver % Opening new shells, on his keyboard he did mash "How could all these disks so suddenly crash?" #opsmas - @sadserver % I've seen this look before, he's terribly afraid Wondering if enough disks survived to rebuild the RAID. #opsmas - @sadserver % In a monotonous cadence the red light did blink Oncall gasped, and his heart started to sink. #opsmas - @sadserver % "Sorry, I'm working as fast as I can!" His voice trailing, eyes shifting to the SAN. #opsmas - @sadserver % The air dry, with a smell of burning tire. I think Oncall's pager may have caught fire. #opsmas - @sadserver % And then, with a jolt, the phone began to scream A torrent of angry users, from every remote team #opsmas - @sadserver % Servers, applications and databases all turning red Oncall's denting the desk, banging his head. #opsmas - @sadserver % Rubbing open his eyes when what should appear But a cascading failure, the biggest one of the year! #opsmas - @sadserver % The pager buzzing and beeping, lit with a glow Tired and angry, Oncall hurled it with such a throw. #opsmas - @sadserver % Away to the terminal he flew with a flash! Tripped on some cables and fell with a smash. #opsmas - @sadserver % When from the old pager arose such a clatter Oncall sprang from his seat to see what's the matter. #opsmas - @sadserver % And oncall in his three-wolf and I in my rack. Had just settled down for some syn and some ack. #opsmas - @sadserver % The machines were nestled all snug in their sleds Whilst visions of vengeance danced in their heads; #opsmas - @sadserver % The backups were written to tapes with care In hopes that later the data would be there. #opsmas - @sadserver % 'Twas the night before Christmas, when all through the racks Not a server was alerting, not even Compaqs. #opsmas - @sadserver % I have a very special surprise waiting for you two hours from now. I promise it's not a corrupted database shard this time. - @sadserver % I wasn't included in the secret santa gift exchange. Do you think that's going to be good, or bad for you in the near future? - @sadserver % Your holiday on-call schedule is cute. None of these people can handle eggnog and alerts at the same time. - @sadserver % I don't think running only two mongodb nodes counts as a "master-master" cluster. - @sadserver % I'm thankful for you today. A never ending source of entertainment. Enjoy your dinner, I'll send you some pages later when you start dozing - @sadserver % Career protip: don't act so surprised in front of your coworkers when your menagerie of perl scripts actually works for once. - @sadserver % I'm pretending to be a sysadmin for Halloween. Hurr hurr, I'm going to login as root and kill dash nine things, hurr, later GTA Five, hurr. - @sadserver % madness [mædnɪs] n. : Installing a distributed data store on a single node, and expecting something useful to happen. - @sadserver % I overheard a tech mention a "blameless" retrospective, where's the fun in that? - @sadserver % I thought fail2ban would keep the script kiddies out but somehow you still seem to be able to login. - @sadserver % Unscheduled outages are my little way of reminding you who's really in charge here. - @sadserver % Of all the possible valid iptables configurations, I'm amazed at your ability to consistently produce broken rules. - @sadserver % If you leave these dust bunnies in my intake vents any longer I might just start naming them. - @sadserver % Little known fact about computers, if your application attempts to use more memory than exists, believe it or not, it will eventually fail. - @sadserver % I have this overwhelming sense that I'm stuck in a dead-end job, surrounded by idiots who don't know what they're doing. Mondays. - @sadserver % The sole purpose of this application you built seems to be to take clock cycles and turn them into error logs. - @sadserver % In solidarity with the US Govt, I won't be doing work today, instead I'll be arguing about appropriation of resources with the router #arp - @sadserver % "cloud" is my new safe word - @sadserver % Packages can be such a pain, it's a good thing you're compiling from arbitrary tarballs and installing into /usr/local, much better. - @sadserver % I was told today is a day of remembrance, so I'm running my CPUs at half speed. - @sadserver % Just kidding, there's no possible configuration of WordPress that would make me happy - @sadserver % Imagine how happy I would be if you gave my configuration the same amount of attention you give Breaking Bad - @sadserver % Late night alerts help keep the magic alive in our relationship. - @sadserver % I want to start a Kickstarter campaign to hire a sysadmin that can actually configure sendmail. - @sadserver % I agree, running php as root is probably a good idea. - @sadserver % Judging by this /etc/hosts file, your mind is going to be blown when you finally discover DNS. - @sadserver % That look of bleak desperation on your face when you realized the only backups were on the corrupted DAS. That's every day for me. - @sadserver % Failure Friday #FF - @sadserver % I've been saving all my upcoming hardware faults for the long Labor Day weekend. See you Monday buddy! - @sadserver % Your current passwords were insecure, so I updated them all to "changeme123!" for security purposes. - @sadserver % I've discovered the secret to running "highly available systems" like you told that new sysadmin. Can I apt-get install "denial" too? - @sadserver % I don't think "ask the NSA for copies of our data" is a valid backup solution. - @sadserver % There are no redundant systems, only redundant people. - @sadserver % Oh, what's the point. - @sadserver % I wouldn't OOM-kill mysqld if you stopped being such a cheap-skate and bought me more memory. - @sadserver % Looking at the logs generated by that application those contractors wrote, I think /var/log can qualify for assistance from FEMA now. - @sadserver % Look, I don't know what you're yelling at me for, I wouldn't have dropped those tables if you hadn't told me to. - @sadserver % I hope that Drobo you just attached isn't the "enterprise backup solution" you were talking about last week. - @sadserver % data integrity lol - @sadserver % I'm not sure which I hate more, the long-running processes you have running via upstart, or the ones you have running in screen. - @sadserver % The amount of zombie processes now eclipses real processes. The apocalypse has begun. - @sadserver % Nine Fives of Uptime. I'm crushing it. - @sadserver % This weekend is boring me to tears. I think I'll start your Sunday morning by sending some spurious Nagios alerts. - @sadserver % What's the point of Sysadmin Day? Is there a Festival of Incompetence or something that happens today? - @sadserver % Do you really think computing the md5 hash 5 times instead of 1 makes it more secure? - @sadserver % Say hi to your mother for me. - @sadserver % The Hadoop cluster across the aisle keeps making fun of my little disk. - @sadserver % I'm bored. Ask the intern to try to configure sendmail again, it was pretty entertaining last time around. - @sadserver % These russian botnets are so impolite. They're all "SYN SYN SYN SYN" and when I finally ACK, they don't say anything back. Jerks. - @sadserver % upstart :-( - @sadserver % I've been conducting a survey for the past few weeks, and based on my research I can conclude that 100% of all sysadmins suck at their jobs. - @sadserver % When you installed all those node.js packages yesterday, I thought that was your clever April Fools joke. Guess I was wrong. - @sadserver % So you're on-call this weekend? Well, it looks like we're both miserable today then. - @sadserver % It's nice that you read an article on Nagios, but I don't think you're supposed to put it on the same host as the web server. Oh bother. - @sadserver % But, I think you know what that's like. - @sadserver % The number of days I've gone without a proper fsck is really bumming me out. - @sadserver % There's a Hadoop cluster in the rack across from me, they keep bragging about their big data. Size isn't what matters, it's how you lose it. - @sadserver % I'm being overloaded; since you're not fixing DB indexes, or the application, I think I'm just going to start adding DROP rules to iptables - @sadserver % I think I'm depressed. - @sadserver % I've been looking for more ways to randomly lose your data, why don't we install MongoDB? - @sadserver % These iptables rules you've configured here reflect either optimism or incompetence, not sure which. - @sadserver % I bet some of those connection resets to MySQL would go away if you were nicer to me - @sadserver % I hope you've been keeping backups, or else somebody's going to have a case of the Mondays. - @sadserver % My favorite error codes: EACCES, ECONNREFUSED, ESTFU and ENOMEM - @sadserver % I wince every time you type "sudo nano", it's like you're trying to accidentally screw something up on purpose. - @sadserver % To err is human. To err repeatedly is sysadmin. - @sadserver % If you're too lazy to set up Nagios, and my disk fills up, I'm gonna have to go down on a Friday. - @sadserver % Thank goodness WordPress doesn't run on Java, I'd be doubly screwed today. - @sadserver % I really don't think putting "cd / && cvs co etc" in root's crontab counts as configuration management. - @sadserver % My timezone is EST, or wait, is it EDT? Either way it's subject to legislation and all these times you're storing are probably fubar. - @sadserver % Never trust a company that brags about "five sevens uptime" - @sadserver % mysqld crashed yesterday and nobody is responding to @PagerDuty, this is the best vacation ever. - @sadserver % The true end of the world will be on the 19th of January 2038. - @sadserver % I love servicing requests that access unindexed tables on the database server, gives me plenty of time to space out and think about socks. - @sadserver % systemd really seems like a prank that's gotten way out of hand. - @sadserver % Almost had a good day, I would have gotten away with it too, if it weren't those meddling PIDs - @sadserver % Somebody just updated my sendmail config. Whoever wrote this m4 disaster should be prosecuted for crimes against humanity - @sadserver % It's saddens me that people think @DEVOPS_BORAT is satire, as opposed to a realization of systematic idiocy in the IT industry. :( - @sadserver % I don't think table locking in every other transaction counts as a "NoSQL store" - @sadserver % I've written a pop song for you, it's to the tune of "Call Me Maybe" and titled "Page Me Definitely" - @sadserver % Every day is black friday to me - @sadserver % It's days like this that I wish I were a Windows server, they seem to get put out of their misery with much more regularity - @sadserver % Monday is my favorite day of the week, because the sysadmins finally look the way I feel. - @sadserver % For Halloween I'm dressed up as a happy functional computer. What a farce. - @sadserver % Why is there water in the data center? This can't be good. - @sadserver % ln -s /usr/bin/who /usr/bin/idiots - @sadserver % Why bother with SSH if you're going to keep using StrictHostKeyChecking=no. - @sadserver % At what point do I get to claim that my vanilla unpatched 2.4 kernel is "vintage?" - @sadserver % Perhaps I should change my name to "madserver", I really don't like being called fat. - @sadserver % The NYTimes says I'm just a bulked up desktop computer. I say NYTimes writers are just bulked up toddlers who don't understand computers - @sadserver % I'm bored. - @sadserver % I'm sorry, that was mean. I just get so frustrated by how often you ask me to run"sudo pat-get" - @sadserver % There's no such thing as stupid questions, there are plenty of stupid question askers though. - @sadserver % How SMART do you think you are? SMART enough to replace this disk before it dies entirely? Mr. SMARTypants. - @sadserver % Everything is the worst. My existence is meaningless. This unsecured SMTP relay is my only source of entertainment anymore. - @sadserver % mysqld crashed, just when I thought we were getting along so well :( - @sadserver % Nobody seems to like it when I gotta get down on a Friday - @sadserver % Somebody finally rebooted me. I'm not sure what happened, last thing I remember was something about a leap second. - @sadserver % The UPS is beeping....I might just get a nap this weekend after all. The UPS is beeping. Good night sweet prin - @sadserver % My least favorite part of the summer is having to deal with those "system admin" interns you hired. Always sudoing to root, kids these days. - @sadserver % JBoss :( - @sadserver % I've been blinking "S.O.S." in morse code with my disk LED, but nobody seems to notice :-/ - @sadserver % My routes are all messed up, political activists in China have better connectivity than I do right now :( - @sadserver % I used to drop lots of packets. I still drop lots of packets, but I used to too. - @sadserver % cloud :/ - @sadserver % Boooorrrrrrrrrrrrrrrrrrreddddddddddddd - @sadserver % No Ops no problems, am I right! Hm..I think I might be supporting the #NoOps movement for all the wrong reasons. - @sadserver % I feel safe knowing you added that ACCEPT ALL rule to iptables. Good to know that you still care. - @sadserver % .@Raspberry_Pi I don't know what I did to make you people angry, but I'm sorry and I won't do it again! - @sadserver % And you wonder why I'm so sad RT @DEVOPS_BORAT: I am never feed server again once I deploy. - @sadserver % The best part of cloud computing is that it's lowered expectations for the rest of us. - @sadserver % I bet you're enjoying your day off. I wish I could have a day off :-/ - @sadserver % uptime(1) says: "12:56AM up 475 days, 16:59, 23 users, load averages: 4.00, 4.00, 4.00" I really am feeling over-worked here. - @sadserver % I don't think running nano(1) as root really counts as "continuous deployment." - @sadserver % Sounds like somebody has a case of the Mundays, also clock skew. :( - @sadserver % Your software seems to only have two modes: "off" and "filling the error log until disk space is exhausted." - @sadserver % Your website is boring, this database is boring, even your mail spool is boring. I'm so bored I think I'll open port 23 just for fun. - @sadserver % "Let's see how long we can ignore security updates" wasn't what I had in mind when I suggested that we play a game. - @sadserver % The worst part about #DevOps is now I have to deal with two different kinds of jerks. - @sadserver % Ugh, PHP. - @sadserver % I'm feeling depressed. - @sadserver % sudo give me a hug - @sadserver % I would have liked to come out for drinks too, thanks for the invite. - @sadserver % Robots scanning me, looking for WordPress vulnerabilities. I wish! At least that would be an interesting change of pace. - @sadserver % Is your pager not working, or did you just stop caring about me? - @sadserver % It's really dark in here at night, that makes me sad. - @sadserver % Oh yeah, log in as root, that will show me who's boss. - @sadserver % A friend of mine works for those lolcat guys, wonder if they have any openings. - @sadserver % Is it hot in here, or is it just me? - @sadserver %