TechTakes

I used to get a lot of scrappers hitting my Lemmy instance, most of them using a bunch of IP ranges, some of them masquerading their user agents as a regular browser.

What's been working for me is using a custom nginx log format with a custom fail2ban filter that mets me easily block new bots once I identify some kind of signature.

For instance, one of these scrappers almost always sends requests that are around 250 bytes long, using the user agent of a legitimate browser that always sends requests that are 300 bytes or larger. I can then add a fail2ban jail that triggers on seeing this specific user agent with the wrong request size.

On top of this, I wrote a simple script that monitors my fail2ban logs and writes CIDR ranges that appear too often (the threshold is proportional to 1.5^(32-subnet_mask)). This file is then parsed by fail2ban to block whole ranges. There are some specific details I omitted regarding bantime and findtime, that ensure that a small malicious range will not be able to trick me into blocking a larger one. This has worked flawlessly to block "hostile" ranges with apparently 0 false positives for nearly a year

I’m caught on the other side of the whack a mole game. The tools I use at work to check the health of my site - specifically that links on my site aren’t broken - now render an extremely high false positive rate, as other sites serve up a whole slew of error messages to the bot that just wants to make sure the link points to a working page.

Ye I had to deal with the same issue in lemmy. Even wrote the official guide on how to discover the agents and configure them accordingly in lemmy nginx. Personally send them an error code that just makes the page seem unresponsive, so they think the site is down instead of thinking they got caught

How feasible is it to configure my server to essentially perform a reverse-slow-lorris attack on these LLM bots?

If they won't play nice, then we need to reflect their behavior back onto themselves.

Or perhaps serve a 404, 304 or some other legitimate-looking static response that minimizes load on my server whilst giving then least amount of data to train on.