Selenium and Playwright both detected by Tripadvisor's bot detection tools

picsou_k6 · Apr 8, 2024

Hello there,

i have been working for couple of months on a project to get data from TripAdvisor.

I am able to easily get comments from any facilities using a location ID, but I still get a lot of trouble to login automatically using either Selenium or Playwright with python.
When I try to access the website, I get this:

I am using IP rotation from a lot of different website (Iproyal, Brightdata, Smartproxy). I guess the problem comes from my browser fingerprint or/and IPs I am using and I haven't found a workaround..

My main objective is to get the cookies so I can make API calls.

Amy idea ?

Nabil Sikder · Apr 9, 2024

Did you tried Undetected chrome driver/ no driver from ultrafunkamsterdam?

paleksic · Apr 9, 2024

Playwright and selenium dont do a lot for your fingerprint, as it for the most part remains the same throughout all of the sessions. I mostly use plugins like playwright-extra and their stealth addon and playwright-afp(I am the plugin author) that was created just for the purpose of changing the browser fingerprint. Another way is using gologin or some similar service, that would give you the option of creating browsers with unique fingerprints

picsou_k6 · Apr 9, 2024

Nabil Sikder said:
Did you tried Undetected chrome driver/ no driver from ultrafunkamsterdam?

Nop I didn't I will try this right now

paleksic said:
Playwright and selenium dont do a lot for your fingerprint, as it for the most part remains the same throughout all of the sessions. I mostly use plugins like playwright-extra and their stealth addon and playwright-afp(I am the plugin author) that was created just for the purpose of changing the browser fingerprint. Another way is using gologin or some similar service, that would give you the option of creating browsers with unique fingerprints

Yes I was currently looking at gologin solution, but I'm still trying to figure out how to make the API works. I will also check plugins you recommend, thanks!
In the meantime this code is producing this error:

Python:

import time
from sys import platform
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from gologin import GoLogin
from gologin import getRandomPort

# random_port = get_random_port() # uncomment to use random port

gl = GoLogin(
    {
        "profile_id": "jolly-darkness",
        "token": "my.very.very.secret.jwt.token",
        # "port": random_port
    }
)

if platform == "linux" or platform == "linux2":
    chrome_driver_path = "./chromedriver"
elif platform == "darwin":
    chrome_driver_path = "./mac/chromedriver"
elif platform == "win32":
    chrome_driver_path = "chromedriver.exe"

debugger_address = gl.start()
chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", debugger_address)
driver = webdriver.Chrome(executable_path=chrome_driver_path, options=chrome_options)
driver.get("http://www.python.org")
assert "Python" in driver.title
driver.close()
time.sleep(3)
gl.stop()

But I get this error:

Bash:

downloadProfileZip
b'<html>\n<head><title>307 Temporary Redirect</title></head>\n<body>\n<center><h1>307 Temporary Redirect</h1></center>\n<hr><center>nginx/1.25.4</center>\n</body>\n</html>\n'
extracting profile
exception File is not a zip file
uploadEmptyProfile
createEmptyProfile
no proxy
empty profile name
profile= {'statusCode': 400, 'message': [{'target': {'id': 'jolly-darkness'}, 'value': 'jolly-darkness', 'property': 'id', 'children': [], 'constraints': {'isLength': 'id must be longer than or equal to 24 characters'}}], 'error': 'Bad Request', 'profile_id': 'jolly-darkness'}

paleksic · Apr 9, 2024

You are using the wrong profile id. The id is not the name, click on 3 dots in the profile you want to start and then go to copy profile id

picsou_k6 · Apr 9, 2024

paleksic said:
You are using the wrong profile id. The id is not the name, click on 3 dots in the profile you want to start and then go to copy profile id

Way better, thanks mate

whatisworthbuying · Apr 18, 2024

paleksic said:
Playwright and selenium dont do a lot for your fingerprint, as it for the most part remains the same throughout all of the sessions. I mostly use plugins like playwright-extra and their stealth addon and playwright-afp(I am the plugin author) that was created just for the purpose of changing the browser fingerprint. Another way is using gologin or some similar service, that would give you the option of creating browsers with unique fingerprints

any version for c#?

Ashk881 · Apr 18, 2024

picsou_k6 said:
am using IP rotation from a lot of different website (Iproyal, Brightdata, Smartproxy). I guess the problem comes from my browser fingerprint or/and IPs I am using and I haven't found a workaround..

Your IPs aren't the reason. If you are using default playwright and selenium configurations on a standard chromedriver install,it will expose a cdp_ref_id key and some other devtools protocol artifacts.

These artifacts can be tested by the site with javascript to check if CDP is enabled to access the site. That's most likely why you are getting detected.

Use an modded chromedriver like https://github.com/ultrafunkamsterdam/undetected-chromedriver with the above stealth plugins mentioned by @paleksic

Accessing from an undetected chromedriver will most likely change your fingerprints but run it from a different IP and on a different machine if you can just to be safe.

swaydee_gk5 · Jun 8, 2024

Ashk881 said:
Your IPs aren't the reason. If you are using default playwright and selenium configurations on a standard chromedriver install,it will expose a cdp_ref_id key and some other devtools protocol artifacts.

These artifacts can be tested by the site with javascript to check if CDP is enabled to access the site. That's most likely why you are getting detected.

Use an modded chromedriver like https://github.com/ultrafunkamsterdam/undetected-chromedriver with the above stealth plugins mentioned by @paleksic

Accessing from an undetected chromedriver will most likely change your fingerprints but run it from a different IP and on a different machine if you can just to be safe.

But Playwright specifically doesn't use chromedriver only websockets,

I had something similar where when I ran a raw chromium browser to access a site everything was okay, but the minute I initialized the same browser using Playwright especially and tried to access the same site, I got CAPTCHA'd immediately

There is something obvious within the Playwright initialization process which makes it clear that the browser is a bot, I doubt any enterprise bot detector relies on navigator.webdriver as it's a very rudimentary check that can easily be bypassed

But I don't know, maybe through some command-line argument that gets passed, or through the way playwright sends messages to the browser, they are able to pick up on something odd and deny access based on that

Some poor guy had a similar problem to me and to OP and got shot down by the repo owners when he tried to ask them about it https://github.com/microsoft/playwright/issues/12538

Anyways the above had me thinking and now I'm curious, how could we check via JS or DOM whether the browser is running in debug-mode or whether CDP is enabled? As from the little I know I thought it wasn't possible to deduce the flags that a browser was initialized with or its debug status as JS wont have scope access

Pahbwe · Aug 7, 2024

If anyone offers tripadvisor scraping as a service that works in August 2024 please ping me a DM

tom_beacon · Jun 15, 2026

picsou_k6 said:
Way better, thanks mate

Hey! Any feedback on the success with GoLogin?

CTRWizard7 · Jun 16, 2026

Yeah that error is basically what @paleksic already flagged... "jolly-darkness" is the profile name, not the id. Look at the bottom of your traceback, it literally says id must be longer than or equal to 24 characters. So GoLogin is trying to download a profile that doesnt exist and falling back to creating an empty one, which is why you see the 307 and the "not a zip file" stuff. All downstream of the bad id.

Open the profile, three dots, copy profile id, paste that 24 char string in place of the name and it should connect.

One more thing, executable_path got removed in selenium 4, so even after you fix the id that line will throw. swap it for a Service:

Code:

from selenium.webdriver.chrome.service import Service
driver = webdriver.Chrome(service=Service(chrome_driver_path), options=chrome_options)

had the same empty-profile mess when i first started with their api, was always the id. once that connects you can grab the debugger address and go from there.

Selenium and Playwright both detected by Tripadvisor's bot detection tools

picsou_k6

Newbie

Nabil Sikder

Registered Member

paleksic

Newbie

picsou_k6

Newbie

paleksic

Newbie

picsou_k6

Newbie

whatisworthbuying

Newbie

Ashk881

Elite Member

swaydee_gk5

Newbie

Pahbwe

Newbie

tom_beacon

Junior Member

CTRWizard7

Registered Member

Main Menu

Marketplace

Making Money

BlackHat World