Lemmy.ca's Main Community

3125 readers

198 users here now

Welcome to the lemmy.ca/c/main community!

All new users on lemmy.ca are automatically subscribed to this community, so this is the place to read announcements, make suggestions, and chat about the goings-on of lemmy.ca.

For support requests specific to lemmy.ca, you can use !lemmy_ca_support@lemmy.ca.

founded 4 years ago

MODERATORS

131

Overnight downtime =( (lemmy.ca)

submitted 1 day ago by Shadow@lemmy.ca to c/main@lemmy.ca

19 comments fedilink hide all child comments

Sorry for the downtime! Unfortunately our secondary firewall took over for some reason, and haproxy failed to properly come up.

I'll be scheduling a maintenance window in the next few days to do some further digging, so I can make sure this is fully resolved.

you are viewing a single comment's thread
view the rest of the comments

[–] whoisearth@lemmy.ca 18 points 1 day ago (3 children)

Wearing my Ops hat (what I do for a living can't help it) have a few questions I'm more than happy to assist with depending on the answers.

What TZ are you in?
Are you open to volunteer support staff in other TZ?
What monitoring solutions are you using? Nagios, Zabbix, etc.
PagerDuty has a free tier do you have alerting and escalation setup?
Do you have run books on the associated issues that come up?
Could access be limited for a volunteer support staff?

I woke up just before 7am EDT and noticed the status page makes reference to PDT so I can only assume you're based out of BC. Given Canada has so many TZ it may be worthwhile investigating a more mature support model and give you some breathing room?

[–] Shadow@lemmy.ca 14 points 1 day ago* (last edited 1 day ago)

Myself, Otter and the server are all in Vancouver. Smorks is out east and mp3 is in the middle of all of us. Between us we actually have pretty good coverage most days.
Yes, if someone has professional SRE experience and wants to help out I'm open to it. There's very little on-going maintenance as things just run smoothly 99.999% of the time, but some additional eyes can't hurt and if someone wants to build new stuff there's things I'd like to do =)
Betterstack at the moment, combined with some custom healthscript scripts which write to our discord and an improperly configured alert config on my phone that didn't wake me up.
Oh neat, I didn't realize pagerduty had a free tier, I'll check it out.
Nope. There haven't been issues that come up, execpt for this one stupid opnsense issue everything has been amazingly stable.
To some degree yes.

[–] Sturgist@lemmy.ca 9 points 1 day ago

Not admin team, but I think I remember them saying the servers are in Vancouver.

[–] skankhunt42@lemmy.ca 6 points 1 day ago

I'm personally not willing to be on call but I was up around the same time and willing to help if they need it. I've been a Linux Admin for 13+ years and have my own k8s cluster in the basement.

I also don't mind the downtime. Thank you guys for everything you're doing!