this post was submitted on 15 Jun 2023
6 points (71.4% liked)

FREEMEDIAHECKYEAH

15 readers
10 users here now

๐Ÿฟ ๐Ÿ“บ ๐ŸŽต ๐ŸŽฎ ๐Ÿ“— ๐Ÿ“ฑ


๐Ÿดโ€โ˜ ๏ธ Wiki / ๐Ÿ’ฌ Chat


Rules

1. Please be kind and helpful to one another.

2. No racism, sexism, ableism, homophobia, transphobia, spam.

3. Linking to piracy sites is fine, but please keep links directly to pirated content in DMs.

founded 1 year ago
MODERATORS
 

How do you scrape stuff, I tried wget but it didn't work. A login + a button press is required to see anything. I can do this manually if required. There are many different slides and videos though.

you are viewing a single comment's thread
view the rest of the comments
[โ€“] TootSweet@latte.isnot.coffee 10 points 1 year ago (1 children)

If you login in a browser, it'll most likely give you a "session cookie" that you should be able to see in the developer tools. (If you're using Firefox's developer tools, it'd be under the "storage" section.) The name of the cookie will generally have the word "session" in the name. After logging in, that cookie identifies you to the server, letting the server know that "this particular request is from CucumberSalad" (or whatever your user is named on that service.) Wget probably hasn't been working because the requests from wget don't include that cookie like the requests from your browser do.

(Just looking at my developer tools while using Lemmy, it seems like the Lemmy web ui doesn't use session cookies but rather a JSON web token in a cookie named "jwt", but I think that cookie would suffice if I was trying to scrape the Lemmy web ui.)

Once you have the proper cookie name and value, you can have wget send the cookie to the server with each request by adding the flag --header 'Cookie: <cookie name>=<cookie value>' (but replace the values in angle brackets. Example: --header 'Cookie: JSESSIONID=ksdjflasjdfaskdjfaosidhwe'.)

Also, if you can provide more info as to what you're trying to scrape, folks can probably help more. Hopefully what I've given you is enough to get you going, but it's possible there might be more hurtles to jump to get it working.

Lovely comment tgankdfor tge help