this post was submitted on 05 Sep 2024
906 points (99.0% liked)

Programmer Humor

19570 readers
1631 users here now

Welcome to Programmer Humor!

This is a place where you can post jokes, memes, humor, etc. related to programming!

For sharing awful code theres also Programming Horror.

Rules

founded 1 year ago
MODERATORS
 
you are viewing a single comment's thread
view the rest of the comments
[–] nick@midwest.social 43 points 2 months ago (4 children)

Just had to restart our main MySQL instance today. Had to do it at 6am since that’s the lowest traffic point, and boy howdy this resonates.

2 solid minutes of the stack throwing 500 errors until the db was back up.

[–] xmunk@sh.itjust.works 19 points 2 months ago (2 children)

If you have the bandwidth... it is absolutely worth it to invest in a maintenance mode for your system, just check some flat file on disk for a flag before loading up a router or anything and then, if it's engaged, just send back a static html file with ye olde "under construction" picture.

[–] nick@midwest.social 4 points 2 months ago* (last edited 2 months ago)

That’s not really… possible at this point. We have thousands of customers (some very large ones, like A——n and G—-e and Wal___t) with tens or hundreds of millions of users, and even at lowest traffic periods do 60k+ queries per second.

This is the same MySQL instance I wrote about a while ago that hit the 16TiB table size limit (due to ext4 file system limitations) and caused a massive outage; worst I’ve been involved in during my 26 year career.

Every day I am shocked at our scale, considering my company is only like 90 engineers.

load more comments (1 replies)
load more comments (2 replies)