It’s almost caught up. My apologies. I am trying to get it fixed permanently.
:( I'm working on it
ok - the queues are processing again. I will work on a more permanent fix after dinner
Working on it
ok - rabbitmq started having prroblems with the delivery queue again. I got it going again. Those messages should be delivered soon.
I will check.
I am not 100% sure. Fedia.io is running on a beast of a server, and so long as it’s working correctly, it should be able to deliver it instantly. But that doesn’t mean that the receiving servers are able to consume and render them that fast.
Once it is more reliable, then I’ll agree, but thank you
there isn't an inteval per se - I don't yet know why it's not immediate. It's possible that the delay is on the receiving side - the server that fedia.io is quite substantial and unless there is some sort of bug, the processing should happen immediately.
Outbound federation was indeed broken. I fixed it, but there was a huge backlog the server had to work through, which took about 12 hours to complete. I just checked and everything appears to be working ok. I am going to create some automation that will detect and alert on this (or other issues) happening in the future.
Not entirely. It looks like the rabbit issue was only impacting one of the queues (“deliver”), though I would have expected that to impact things like microblog too. All I can say with clarity is that the instance was operating in a very unhealthy state.
The queue appears like it’ll take several hours to flush, but it’s working.
Note that even once I get this fixed, there will inevitably be another problem crop up. Posting here is fine, but an email to jerry@infosec.exchange or a ping to @jerry@infosec.exchange (my mastodon account) would probably be faster (note, when federation breaks here, messages to @jerry@infosec.exchange wouldn't get through from fedia.io...