Keep in mind that the upcoming Lemmy update will probably fix this I think. (Replacing websockets)

Just hopping into the chain to say that I appreciate you and all of your hard work! This place—Lemmy in general, but specifically this instance—has been so welcoming and uplifting. Thank you!

At least the "reply" button goes away so I don't end up double- triple- or even duodecuple-posting! Thanks for all the hard work that must be going on behind the scenes right now!

maybe related, but I've noticed that upvoting/downvoting has similar lag delays

Been noticing this in the app I’m working on. Pretty much all POST requests fail to return a response and just timeout after 60 seconds. A quick refresh shows that the new items do successfully get created though.

I assume that there is something that is O(N), which explains why wait time scales with community size (amount of posts, comments)

Again, thank you for the outstanding work! You are awesome!

Also, the new icon for lemmy world is great!

Have you tried enabling the slow query logs @ruud@lemmy.world? I went through that exercise yesterday to try to find the root cause but my instance doesn’t have enough load to reproduce the conditions, and my day job prevents me from devoting much time to writing a load test to simulate the load.

I did see several queries taking longer than 500ms (up to 2000ms) but they did not appear related to saving posts or comments.

Thank you for your hard work and keeping us up to date.

Does this behaviour appear on other big instances? E.g. lemmy.ml?

Thanks for your and the other Lemmy devs work on this. These growing pains are a good thing as frustrating as it can be for users and maintainers alike. Lemmy will get bigger and this optimization treadmill is really just starting.

In my case, the page keeps spinning but the post is not submitted, regardless of reloading the page or waiting for a long time. There was one case where I cut down significantly on the amount of characters in the post and then it posted, but I have been unable to replicate this.

@ruud@lemmy.world Yo dude, first off huge props and a big thank you for what you have setup. I’ll be donating monthly while I am here. I appreciate that we have an alternative to Reddit at this critical moment in time.

I do have a question on your long term plans, do you want to continue to expand and upgrade the server, as funding allows, or is there a cap that you will close off the server to new members? Or perhaps make it more of a process to join?

ok, so it's not just me. Hope it gets resolved soon!

Thanks for posting the workaround and for working to resolve the issue. Lemmy is a great place, and a real breath of fresh air after Reddit.

its def more hung up today, oddly its only first level replies for some reason

Thank you so much

I noticed that too, page keeps spinning but comments are posted immediately anyway.

I’ve done this twice in the last 20 minutes and the content is not there. This workaround was working earlier today though.

I noticed this, thanks for the clarification

This is the biggest issue I have run into. Thanks for looking into it.

One of the large applications I was working on had the same issue, to solve it we ended up creating multiple smaller instances and started hosting a set of related API's in each server.

for example read operations like list posts, comments etc could be in one server. write operations can be clusered in one server.

Later, whichever server is getting overloaded can be split up again. In our case 20% of API's used around 3/4th of server resources, so we split those 20% API's in 4 large servers and kept the remaining 80% API's in 3 small servers.

This worked for us because the DB's were maintained in seperate servers.

I wonder if a quasi micro-services approach will solve the issue here.

Edit 1: If done properly this approach can be cost effective, in some cases it might cost 10 to 20 percentage more in server costs, however it will lead to a visible improvement in performance.

Is the slowdown that it the instance has to send out updates about the comment to every other instance before returning a successful response? If so, is anyone working on moving this to an async queue?

Sending out updates seems like something that’s fine being eventually consistent