myersguy

Love Wonderboy, and am glad to see he is looking out for himself.

This isn't really contradictory to what I said. I only wished to express that the two don't scale exactly the same (in terms of execution)

I've seen this happening a lot, so I would assume there are some issues federating comments.

Wow, that is surprisingly not bad given the size of the instance!

Interesting indeed. I spent some time combing through the DB and pictrs folder to try to figure it out, but I'm at a loss so far.

Again, my knowledge of the Lemmy codebase is very small, and we could possibly host the monolith in microservices style. The point I am making is this (when it comes to scaling monolith vs microservices):

If the federation logic were split out, we could configure it to run on super tiny docker instances on Google Cloud or AWS. Any time we needed it to, it would autoscale to handle the traffic. The configuration for these dockers could be super minimal memory, no storage, and multiple weak CPU cores. This would be super affordable while still being able to handle as much traffic from federation as we ask it to. One of the cool things with Google Cloud Run is that it handles load balancing between docker containers for you (just point the federation traffic at the necessary URL)

IF Lemmy has things like background services, scheduled tasks, etc, this would significantly muddy the water (we would need each service to be able to handle being run on a multitude of instances, or we would need to be able to disable each one instance by instance). And if we just scaled by spinning up more instances of Lemmy, we would also need to ensure that only federation traffic is heading to the weaker instances that we spun up for such purpose, or we would need to ensure that each spun up instance has enough resources to handle federation traffic along with the main application.

I feel like I need to state once more: I don't necessarily think Lemmy needs to move to microservices. Only that scaling monolith vs microservices is not necessarily the same.

I'd be less concerned with memory (of which Lemmy seems to use very little), and much more concerned with CPU core count. I touched on it in my other comment, but I don't understand how a few cores is supposed to handle the ridiculous number of federation workers people are setting their instances to.

Well, I'm going to start by repeating that I don't necessarily agree that it being monolithic is necessarily a problem right now.

The immediate thought in my mind would be all of the federation logic. That's where all of the instances seem to be lagging behind, and it seems the common fix is "just increase the workers to one billion". Apparently that does something meaningful, but the developer in me wants to know how a few cores can put so many workers to use.

Spinning federation off into a microservice means you could deploy it on something like Cloud Run or AWS ECS, and have it autoscale as the workload demands it. Seems like a pretty prime candidate to me.

I'm going to have to disagree. I just remoted in to check it out. I have 2560x1440 resolution images weighing 2.9 MiB in there. It's pretty clear that these are full size images.

I'm not so sure this is the case. The images folder on my single user instance is over 1gb, and I haven't posted any images.

I disagree that it being a monolith is immediately a problem, but also

In fact you scale a monolith the same way you scale micro services.

This is just not true. With microservices, it is easy to scale out individual services to multiple instances as demand requires them. Hosting a fleet of entire Lemmy instances is far more expensive than just small slices of it that may require the additional processing power.

Yes, sex is a lot like sweeteners in that respect.