Change useragent Chrome headless chromium mode

:


---

I'm trying to scrape data from a website using ChromeDriver and Selenium, but the site has anti-bot detection. I successfully scraped data once on my PC, but after that, I can no longer scrape due to the anti-bot mechanism, even with a 4G mobile IP. I tried using a new PC and a 4G connection, and it worked again, but after that, the anti-bot measures kicked in again.

To work around this, I tried using a VPS online, but I still can't scrape because of the anti-bot detection. The issue seems to be how to change the user agent in headless mode. Unfortunately, I don't have access to the source code of the scraping bot, only the executable (.exe).

I found this solution online for changing the user agent: {chrome:headless:userAgent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36}. How can I implement this if I don’t have access to the source code?


---

Which website you are trying to scrape? you can try proxy rotation also can use stealth.

Use Undetected Chromedriver or Seleniumbase UC, should do the work most of the time,
All depends on the website(s) detection system, you could also be rate limited if you're scraping aggressively,
Just for reference, you can use AI to generate the code after all, it will do the work in no time

hideath said:

Use Undetected Chromedriver or Seleniumbase UC, should do the work most of the time,
All depends on the website(s) detection system, you could also be rate limited if you're scraping aggressively,
Just for reference, you can use AI to generate the code after all, it will do the work in no time

Click to expand...

This

vittorianicolosi1994 said:

:


---

I'm trying to scrape data from a website using ChromeDriver and Selenium, but the site has anti-bot detection. I successfully scraped data once on my PC, but after that, I can no longer scrape due to the anti-bot mechanism, even with a 4G mobile IP. I tried using a new PC and a 4G connection, and it worked again, but after that, the anti-bot measures kicked in again.

To work around this, I tried using a VPS online, but I still can't scrape because of the anti-bot detection. The issue seems to be how to change the user agent in headless mode. Unfortunately, I don't have access to the source code of the scraping bot, only the executable (.exe).

I found this solution online for changing the user agent: {chrome:headless:userAgent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36}. How can I implement this if I don’t have access to the source code?


---

Click to expand...

If you don't have access to the source code but still want to change the User-Agent in headless mode, you can try the following methods:

1. Use Chrome launch parameters (for executable .exe files)
If you are running a standalone executable file instead of Python code, you can try to change the User-Agent through command line parameters.

Windows Method
Run your executable in CMD or PowerShell and add the --user-agent parameter:

sh
Copy
Edit
your_scraper.exe --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
2. Use environment variables
Some Selenium robots support reading environment variables, you can try setting the CHROME_USER_AGENT variable and then run your .exe file:

sh
Copy
Edit
set CHROME_USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
your_scraper.exe
On Linux/Mac:

sh
copy
edit
export CHROME_USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
./your_scraper
3. Modify Chrome configuration directly
If your scraper uses ChromeDriver, you can try to modify Chrome's configuration file to use a new User-Agent by default.

Find the Chrome profile path (usually C:\Users\your username\AppData\Local\Google\Chrome\User Data)
Edit the Local State file (open it with a text editor)
Add/Change User-Agent:
json
Copy
Edit
{
"user_experience_metrics": {
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
}
}
Save and rerun your .exe file
4. Try GUI mode (non-headless mode)
Some anti-bot mechanisms will detect headless browsers, you can try to run Chrome in GUI mode:

sh
Copy
Edit
your_scraper.exe --disable-headless
Or, try to start a normal Chrome in the code browser, instead of headless mode (if you can access the code):

python
copy
edit
options = webdriver.ChromeOptions()
options.headless = False # Turn off headless mode
driver = webdriver.Chrome(options=options)
5. Other possible solutions
Use residential proxy (residential IP proxy) instead of server IP on VPS
Use Puppeteer + Stealth plugin (if Selenium is still detected)
Use undetected_chromedriver (can bypass detection under Python)
Try navigator.webdriver = false method (if you can inject JS code)
Summary If you can't modify the code, you can try: Method 1: Command line parameters (easiest)
Method 2: Environment variables (if Scraper supports)
Method 3: Modify Chrome configuration (for ChromeDriver)
Method 4: Turn off headless mode (if the website detects headless browser)