this post was submitted on 26 May 2022
28 points (93.8% liked)
Privacy
31874 readers
551 users here now
A place to discuss privacy and freedom in the digital world.
Privacy has become a very important issue in modern society, with companies and governments constantly abusing their power, more and more people are waking up to the importance of digital privacy.
In this community everyone is welcome to post links and discuss topics related to privacy.
Some Rules
- Posting a link to a website containing tracking isn't great, if contents of the website are behind a paywall maybe copy them into the post
- Don't promote proprietary software
- Try to keep things on topic
- If you have a question, please try searching for previous discussions, maybe it has already been answered
- Reposts are fine, but should have at least a couple of weeks in between so that the post can reach a new audience
- Be nice :)
Related communities
Chat rooms
-
[Matrix/Element]Dead
much thanks to @gary_host_laptop for the logo design :)
founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I don't think that a robots.txt file is the appropriate tool here.
First off, robots.txt are just hints for respectful crawlers. Go proxies are not crawlers. They are just that: caching proxies for Go modules. If all Go developers were to use direct mode, I think the SourceHut traffic would be more, not less.
Second, let's assume that Go devs would be willing to implement something to be mindful of robots.txt or
retry-after
indications. Would attackers do? Of course not.If a legitimate although quite aggressive traffic is DDoSing SourceHut, that is primarily a SourceHut issue. Returning a 503 does not have to be respected by the client because the client has nothing to respect: the server just choose to say "I don't want to answer that request. Good Bye". This is certainly not a response that is costly to generate. Now, if the server tries to honor all requests and is poorly optimized, then the fault is on the server, not the client.
I have not read in details the Go Proxy implementation, to be truthful. I don't know how it would react if SourceHut was answering 503 status code every now and then, when the fetching strategy is too aggressive. I would simply guess that the server would retry later and serve the Go developers a stale version of the module.
(I would appreciate if the down voters were able to express their disagreement with words. Maybe I'm wrong, but then, please do me the favor of explaining me how. Also, I'm not a SourceHut hater; I even give money to Drew every month, because I like the idea of SourceHut. I just think Drew is wrong on that matter)
https://github.com/gomods/athens/blob/723c06bd8c13cc7bd238e650a559258ff7e23142/pkg/module/go_get_fetcher.go#L145-L148 https://github.com/gomods/athens/blob/723c06bd8c13cc7bd238e650a559258ff7e23142/pkg/module/go_get_fetcher.go#L163-L165
So two infos: