this post was submitted on 21 Aug 2024

318 points (100.0% liked)

196

16489 readers

2111 users here now

Be sure to follow the rule before you head out.

Rule: You must post before you leave.

^other^ ^rules^

founded 1 year ago

MODERATORS

greembow@lemmy.blahaj.zone

moss@lemmy.blahaj.zone

moss@lemmy.world

queue@beehaw.org

funky_rodent@lemmy.blahaj.zone

PeachyMcPeachface@lemmy.blahaj.zone

threegnomes@lemmy.blahaj.zone

greembow@lemmy.world

remotelove@lemmy.ca

Roflmasterbigpimp@feddit.de

qaz@lemmy.world

A_Very_Big_Fan@lemm.ee

qaz@lemmy.blahaj.zone

A_Very_Big_Fan@lemmy.world

qaz@lemmy.dbzer0.com

qaz@sh.itjust.works

qaz@lemmy.sdf.org

318

rulebots.txt (lemmy.world)

submitted 2 months ago by GroupNebula563@lemmy.world to c/196@lemmy.blahaj.zone

33 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] GroupNebula563@lemmy.world 9 points 2 months ago (1 children)

Perhaps, likely depends on the crawler though

[–] unexposedhazard@discuss.tchncs.de 12 points 2 months ago

Yeah i dont think ignoring robots.txt is even illegal. They can ofcourse just block your crawlers IP but that would be a cat and mouse game that they would lose in the end.