this post was submitted on 15 Sep 2023
215 points (97.8% liked)
Asklemmy
43945 readers
1039 users here now
A loosely moderated place to ask open-ended questions
Search asklemmy π
If your post meets the following criteria, it's welcome here!
- Open-ended question
- Not offensive: at this point, we do not have the bandwidth to moderate overtly political discussions. Assume best intent and be excellent to each other.
- Not regarding using or support for Lemmy: context, see the list of support communities and tools for finding communities below
- Not ad nauseam inducing: please make sure it is a question that would be new to most members
- An actual topic of discussion
Looking for support?
Looking for a community?
- Lemmyverse: community search
- sub.rehab: maps old subreddits to fediverse options, marks official as such
- !lemmy411@lemmy.ca: a community for finding communities
~Icon~ ~by~ ~@Double_A@discuss.tchncs.de~
founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Not me personally, but one of my career mentor's friend's took down the entirety of Google Ads as an intern for like 10 minutes. Apparently it was a multi-million dollar mistake, but they fixed the issue so it couldn't happen again and all was well afterward.
In my first couple months, I broke Amazon so that no-one in Europe could buy video for a few hours. On a Friday, right before going on a week's vacation.
The way that the ensuing investigation and response was carried out - 100% blame-free, and focused on "how did these tools let him down? How can we make sure no-one ever makes that same mistake again?" - gave me a career-long interest in Software Resiliency and Incident Management.
Source: https://twitter.com/CarlaNotarobot/status/1481458190722207747
Yep. And every time there's a thread about an Internet service having an outage, there's some kid saying "oh, someone's getting so fired for this one!"
Yeah, the competent business folks know that if you fire people for outages, you lose everyone who even stands a chance of preventing outages. And you tell the rest of your staff to hide problems. Businesses that do that kind of thing tend to end up with a valuation in the single digits.
Not always.
Sometimes the internet service outage is due to a car taking out a green box or a pole.
If they fire the backhoe driver, then backhoe drivers will never learn.
Oh yeah, That too. Plus trenchers, and hole boring for like fences or billboards.
If an intern (or damn near any employee) can be in a position to single handedly take down that scale of system itβs not the intern that should be fired - itβs the architect that baked that kind of weakness in the first place.
You're not a real SRE until you've caused at least a $100K outage. You're not a good SRE until you've fixed it so nobody can ever make that particular one again.