Arc717
Newbie
- Aug 7, 2018
- 17
- 10
Hello whoever is reading this:
I've been coding (mostly webscraping) in Python for about a year and haven't run into any problems until now. I've been webscraping this site for a while (BHW won't let me post urls. Content marked as inappropriate), but then the site owners hired Distill Networks and blocked bots.
No logins or captchas are required, however the site requires cookies and JavaScript to be enabled. I am trying to make a request, and then just grab the html. Here is what I learned so far:
My normal Chrome works fine, but I am blocked before I make one request when I used selenium. They can detect selenium whether you browse by hand or not.
So far in Selenium I have:
- hex edited out the $cdc_ in chromedriver and turned it into like $wall or something
- set navigator.webdriver to false manually (I noticed navigator.webdriver is undefined in my normal Chrome, but I can't figure out how to undefine it in selenium. I figured false is better than true)
- Selenium automatically set my user agent to match my current browser
- installed adblock to show I have plugins
- changed browser window size to something not the default when Selenium is opened
- I didn't bother using a proxy since my IP isn't banned.
- googled the page I wanted and clicked the google link manually instead of making the request directly
...and, no dice.
If anyone has a working solution, we can negotiate a price, or if someone could help me out for free that'd be amazing (But not likely.)
Anyways, thanks for reading.
I've been coding (mostly webscraping) in Python for about a year and haven't run into any problems until now. I've been webscraping this site for a while (BHW won't let me post urls. Content marked as inappropriate), but then the site owners hired Distill Networks and blocked bots.
No logins or captchas are required, however the site requires cookies and JavaScript to be enabled. I am trying to make a request, and then just grab the html. Here is what I learned so far:
My normal Chrome works fine, but I am blocked before I make one request when I used selenium. They can detect selenium whether you browse by hand or not.
So far in Selenium I have:
- hex edited out the $cdc_ in chromedriver and turned it into like $wall or something
- set navigator.webdriver to false manually (I noticed navigator.webdriver is undefined in my normal Chrome, but I can't figure out how to undefine it in selenium. I figured false is better than true)
- Selenium automatically set my user agent to match my current browser
- installed adblock to show I have plugins
- changed browser window size to something not the default when Selenium is opened
- I didn't bother using a proxy since my IP isn't banned.
- googled the page I wanted and clicked the google link manually instead of making the request directly
...and, no dice.
If anyone has a working solution, we can negotiate a price, or if someone could help me out for free that'd be amazing (But not likely.)
Anyways, thanks for reading.
