The_Gibson @thegibson@hackers.town

@TheGibson

Deleted all shared libs on a production Solaris 10 system.

Was able to rebuild the config using the equivalent of Linux "ldconfig" and rebooted the machine before anyone realized. Solaris ran fine after that.

I also did an "rm -rfv .", not realizing I had changed to ~ instead of a subdir. Lost quite a lot of important files that way. Yes, no backups.

@thegibson We spent probably 20 minutes writing a query to carefully select a subset of production servers upon which to execute a handful of *very* resource intensive commands, and then to disable a bunch of critical services.

We stored that data in a variable.

We referenced a different variable when we ran that command in our production datacenter.

It ran for about 10 minutes before I said "Gee, this is taking a really long time" and then my co-worker went in to a panic and reverted our changes.

Went from multiple million active users to less than 500k in about 5 minutes, back up to multiple millions in less than 3.

Happened so fast that it didn't even trigger an alert.

@TheGibson Was managing a terrible HR application and got overconfident while applying patches, went for the next one without a snapshot and it failed miserably (it didn't in our test env).

The HR department (hundreds of employees) was left unable to do any work for half a day. We didn't know at the time if it was going to be an easy fix or something that would let us struggle with our backups for *days*.

Been treating prod environments like unexploded bombs since then, which is what they are.

@thegibson

rm -rf / *

...as root

...at my first real job

...at NASA

@thegibson wired about 10 lightbulb sockets backwards because at the time I thought 110/120v AC was done the same way that 220/240v AC was done, so wouldn't matter which direction I hooked them up, so connected them randomly.

@thegibson @djsundog Backin up was easy lawd, when he sang the blues. Backin' up was good enough for me.

@csepp

I ruined a MB once by just dropping a screw on it, causing a short. Happened so fast!

@TheGibson

@thegibson completely destroyed production Kafka cluster, more senior Ops engineers managed to fix it. But the very next day I did it again

@thegibson@hackers.town backups are for people who are more worried about integrity than confidentiality

@TheGibson "real men" don't backup.

because "real men" aren't afraid of crying.

I'm trying to figure out the name of this MP3 a guy showed me in 2003. All I remember is that it had a black and red display and a weird barrel/dial on the side.

@TheGibson You might be interested in this:

https://www.solipsys.co.uk/Chartodon/108295593183482710r.svg