Python + Selenium Leaving Footprints?

[TiagoS]

Hey guys! I'm developing a bot that scrapes prices from a specific website and i'm running into a problem. Somehow, the website is able to tell that I'm using selenium/chrome webdriver. It displays higher prices when I scrape trough my bot and when I search manually, it shows fake/higher prices. (I've done a search and it is confirmed that the website does that, other people mentioned it). Here what I have tried so far:

- Clearing cookies/cache
- Random delays
-Changing User agent
-Using proxies

I'm looking for ideas of what I might be doing wrong, what kind of footprints am I leaving? What's even worse is that sometimes it works and sometimes is does not (it works 4 out of 10 times).

I am even willing to pay if someone manages to come up with a solution, if that's the case (Pm me if you can).

 

[GoDesain]

That new for me.. can you share the url with me ?
will try to check..
about clear cookies and chache (start private mode):

profile = webdriver.FirefoxProfile()
profile.set_preference("browser.cache.disk.enable", False)
profile.set_preference("browser.cache.memory.enable", False)
profile.set_preference("browser.cache.offline.enable", False)
profile.set_preference("network.http.use-cache", False)

Random delay:

from random import randint
from time import sleep
sleep(randint(10,100))

Random useragent:

from fake_useragent import UserAgent
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", UserAgent().random)
profile.update_preferences()

Proxy :

profile = webdrive.FirefoxProfile()
profile.set_preference('network.proxy_type',1)
profile.set_preference('network.proxy.http',"xxx.xxx.xxx.xxx")
profile.set_preference('network.proxy.http_port',3128)
profile.update_preference()

 

[LostLife]

Are you sure website is not changing prices randomly? Please check manually first for 20-30 times. Please make sure deleting cookies are successful. Can u share the url please?

 

[living2xl]

why not put a filter that scrapes the same url x times and discards those scrapes with prices higher than x times the median

 

[Google Prince]

It's most likely Chromedriver, which is known to leave footprints.

Change the webdriver to firefox, that might help.

 

[jamie3000]

https://www.distilnetworks.com are able to detect selenium, not sure if they ever elaborated on how. Think it was through some sort of js probing.

 

[jamie3000]

https://www.reddit.com/r/webdev/comments/5q1ypx/what_is_your_approach_on_scraping_distil_networks/

 

[akio_daichi]

Can you share the site's url? I think this site is not using distill-networks services, because they block you out immediately.

 

[pasdoy]

This is one of the reason https://github.com/Valve/fingerprintjs2

 

[pasdoy]

And the list of most footprints https://github.com/Valve/fingerprintjs2/wiki/List-of-options

 

[Tozzy]

It is not so hard to detect auto browser even with plain JS and no rocket science.
Who knows how this or that website does it but one can instantly think of 10 or 20 ways to maintain this.

 

[B.Shahin]

It is the 1st time I see some thing like this
I am using selenium for fb, and it is working for months with no error
I am also using chromedriver!
So just following here to see if there is some thing required to be added!

 

[jamie3000]

You can always recompile phantom and other headerless browsers from source if you can find what js enumeration is used for detecting

 

[MrPenguin]

Use Firefox. Install the extension random user agent. This will randomize alot of different settings. Should work.

 

[rootjazz]

Lots of things could be at play.

When you use the browser normally (and the website will know (if they chose to know) what is normal behaviour within some standard deviations). Time reading / clicking etc.
Also, JS events will not be triggered, no mouse action could be a flag against you, just programmtically clicking on things would not give a realistic even pattern. Should be able to check this in a browser debug app.

I've had to automate some sites where had to control the actual mouse in order to get around the detections - and then move the mouse in a realistic way (you can look at curve generation / bezier curves

 

[rolax]

Lots of things could be at play.

When you use the browser normally (and the website will know (if they chose to know) what is normal behaviour within some standard deviations). Time reading / clicking etc.
Also, JS events will not be triggered, no mouse action could be a flag against you, just programmtically clicking on things would not give a realistic even pattern. Should be able to check this in a browser debug app.

I've had to automate some sites where had to control the actual mouse in order to get around the detections - and then move the mouse in a realistic way (you can look at curve generation / bezier curves

did you just have some movement of the mouse.. or did the mouse actually navigate to the pixel location and hover over the buttons you wanted to click?

 

[itz_styx]

there is a good fingerprinting method that i wont name here (no need for whitehats to get ideas). that same method also allows for detecting what type of browser you are using i.e. selenium.

 

[BloodyNinja]

I will only say here that no one has a good solution to this yet.
There are some known or less known half-measures but a robust solution doesn't exist.
Most people in this topic have no idea about the problem.

 

[Lothric]

That new for me.. can you share the url with me ?
will try to check..
about clear cookies and chache (start private mode):

profile = webdriver.FirefoxProfile()
profile.set_preference("browser.cache.disk.enable", False)
profile.set_preference("browser.cache.memory.enable", False)
profile.set_preference("browser.cache.offline.enable", False)
profile.set_preference("network.http.use-cache", False)

Random delay:

from random import randint
from time import sleep
sleep(randint(10,100))

Random useragent:

from fake_useragent import UserAgent
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", UserAgent().random)
profile.update_preferences()

Proxy :

profile = webdrive.FirefoxProfile()
profile.set_preference('network.proxy_type',1)
profile.set_preference('network.proxy.http',"xxx.xxx.xxx.xxx")
profile.set_preference('network.proxy.http_port',3128)
profile.update_preference()

This
UserAgent().random looks not good enough. I just tested them and they spits out garbage like chrome 29, 41.. it doesn't make sense to me. If they use real world stats, they should return version like 59, at least 57, am i missing something here?

 

[tNK]

I suggest using multilogin to automatically handle all your browser fingerprints. That's what I'm using at least after wasting time trying to do it manually. I've been experimenting using it with selenium gecko driver and it's been working well. If you need help integrating to your software let me know.