Next time post python-related questions on programming.dev/c/python
Programming
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities !webdev@programming.dev
c/learn_programming would also be a good fit for this.
If I understood correctly crossposting is somewhat frowned upon to avoid duplocate posts on lemmy (and I agree with that) so I will remember for next time.
Thanks!
Sorry, thanks for sharing the link. Will do that for next time.
No problem, I hope you will find help there, when you need it
Not 100% sure if this is helpful, but here is what I'm importing and a snippet initiating webdriver. Sorry on mobile. The hardest part was getting the right chromium installed and getting the path right. I believe this is the one I used. sudo apt-get install chromium-chromedriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver")```
Thanks! I appreciate your response, and love the username! :) Fortunately getting the right chrome driver working was one of the very few things that just worked for me. Have you gotten proxies working without proxy detection with this setup, and if so, what did you do?
Oh, sorry haven't tried proxies. Just scraping a single site.
You can do it with plain selenium all you need to set the proxy in the browser options.
from selenium import webdriver
PROXY_WITH_PORT= "111.222.333.443:8080" chrome_options = webdriver.ChromeOptions() chrome_options.add_argument(f'--proxy-server={PROXY_WITH_PORT}')
chrome = webdriver.Chrome(chrome_options=chrome_options) chrome.get("http://google.com")
In general you can pass any command like argument to the browser using options. For chrome you can find all the proxy related options here: https://www.chromium.org/developers/design-documents/network-settings/
Also if you are using the latest selenium (4.10 I think) you don’t need to point to the chromedriver executable, selenium will automatically download it if it can’t find the driver
Edit: more information about chrome arguments Edit: info about driver download
Thank you. The problem with this is that it does not support authorized proxies though. You cannot pass user:pass through it. My janky solution was to uae something like pproxy to relay the connection through my own peoxy server without user:pass. Has not been fully effective though.
I will look into doing this and using IP authorization instead of user:pass. Thank you for the help.
How is vanilla selenium with WebRTC or DNS leaks?
You might need to install your own proxy on the selenium PC and then chain that proxy to the authenticated one. Then configure the driver to use the local unauthenticated proxy
Also since Selenium just drives an actual browser the WebRTC and DNS leaks will be the browser’s responsibility not selenium. As long as you can locate elements on a page your will be ok
have you considered using something else? like playwright/puppeteer?
I have considered it but since I only know Python I have yet to try it. Might be time to learn javascript :)
Wow I had no idea! Thanks for sharing I'll look into it.
After getting it working with proxies I realize they have the exact same webRTC issue. I assume it's for the same reason as Selenium having it. Unfortunate but there you have it.