How to detect puppeteer with 100% accuracy

greydingo · May 27, 2020

I just posted this in another thread, but I thought it may be better as it's own thread.

I will disclose a method to detect puppeteer with 100% accuracy.

I have never told anyone about this. I've never seen it discussed anywhere, I discovered this on my own a while back.

I'm guessing anti-bot companies already know this trick.

I just uploaded a demo repo.

https://github.com/digitalhurricane-io/puppeteer-detection-100-percent

If your replace a function that puppeteer uses, such as document.querySelector, and throw an error inside of it, you can see the string "__puppeteer_evaluation_script__"

In order to get around this, you need to fork the library and change that string.

ADETILEHIN TAIWO · May 27, 2020

I really can't understand, can you please explain more in this.

TheVigilante · May 27, 2020

ADETILEHIN TAIWO said:
I really can't understand, can you please explain more in this.

Same here

greydingo · May 27, 2020

Check the explanation in the readme on github.

I'm sorry, no disrespect at all, but if you don't have a thorough understanding of javascript, it just may not make sense.

Check the main.js file in the folder named public in the repo. It's only 32 lines.

Ideas taken from this blog post:
https://antoinevastel.com/javascript/2019/06/10/monitor-js-execution.html

lotus13 · May 27, 2020

Bookmarked, awesome. I would rather use nightmare js, but I might come back to this.

sockpuppet · May 27, 2020

greydingo said:
In order to get around this, you need to fork the library and change that string.

That string is the sourceURL, it would be the best to remove it instead of renaming.

greydingo · May 27, 2020

sockpuppet said:
That string is the sourceURL, it would be the best to remove it instead of renaming.

Not sure about that. I don't remember everything I tried. But if it was better to remove it, or could be removed, I'm guessing I would have done that.

What I did do, was rename it to the same name as common scripts run by common browser plugins.

Legit scripts running on the page have a name anyways.

BlackDesign · May 27, 2020

if its so easy, why big companies like IG don't block it that way?
is the page.click() function is also using the querySelector?

greydingo · May 27, 2020

BlackDesign said:
if its so easy, why big companies like IG don't block it that way?
is the page.click() function is also using the querySelector?

I don't have an answer for you. But if you don't believe me, clone the repo and try for yourself.

And who's to say big companies aren't looking at that? They may have reasons for letting you get by with things some of the time. I won't claim to have all the answers.

I worked on a bot for a site before that would purposefully let you get away with something sometimes, in order to trick you into thinking the changes you were making to the bot were working.

SprintBoog · May 27, 2020

Or you can handle the exception with a try catch blocks

greydingo · May 27, 2020

elhaloui_hicham said:
Or you can handle the exception with a try catch blocks

No you can not.

I'm not throwing an error. Just creating one.

I'll edit the original post to say 'creating' instead of 'throwing'

Edit: NVM. I can't edit the original post.

sockpuppet · May 29, 2020

You could avoid the detection by running the javascript in it's own context with createIsolatedWorld
But unfortunately it's not in puppeteer yet, see #2671

So you need to send raw devtools-protocol commands to get it working

Code:

const mainFrame = page.mainFrame()
const iso = await page._client.send('Page.createIsolatedWorld', {frameId: mainFrame._id, worldName: 'iso'});
await page._client.send('Runtime.evaluate', {
  expression: 'document.querySelector("#text-field").focus()',
  contextId: iso.executionContextId
});
await page.keyboard.type('hi');

greydingo · May 31, 2020

sockpuppet said:
You could avoid the detection by running the javascript in it's own context with createIsolatedWorld
But unfortunately it's not in puppeteer yet, see #2671

So you need to send raw devtools-protocol commands to get it working

Code:

const mainFrame = page.mainFrame() const iso = await page._client.send('Page.createIsolatedWorld', {frameId: mainFrame._id, worldName: 'iso'}); await page._client.send('Runtime.evaluate', { expression: 'document.querySelector("#text-field").focus()', contextId: iso.executionContextId }); await page.keyboard.type('hi');

That does the trick. Great work sockpuppet!

Sebastiann · Jun 1, 2020

sockpuppet said:
You could avoid the detection by running the javascript in it's own context with createIsolatedWorld
But unfortunately it's not in puppeteer yet, see #2671

So you need to send raw devtools-protocol commands to get it working

Code:

const mainFrame = page.mainFrame() const iso = await page._client.send('Page.createIsolatedWorld', {frameId: mainFrame._id, worldName: 'iso'}); await page._client.send('Runtime.evaluate', { expression: 'document.querySelector("#text-field").focus()', contextId: iso.executionContextId }); await page.keyboard.type('hi');

This is awesome

Do you need to instantiate a new isolated context for each task, or can you just initialise it once?

For example:

Code:

const puppeteer = require('puppeteer-extra');

class Actor {

   browser;
   page;
   iso;
 
   async init() {
      this.browser = await puppeteer.launch();
      this.page = await this.browser.newPage();
      const mainFrame = await this.page.mainFrame();
      this.iso = await this.page._client.send('Page.createIsolatedWorld', { frameId: mainFrame._id, worldName: 'iso' });
   }

   async evaluate(expression) {
      await this.page._client.send('Runtime.evaluate', {
         expression: expression,
         contextId: this.iso.executionContextId
      });
   }

   async foo() {
      // ...
      await this.page.goto('https://foo.com', { waitUntil: 'networkidle2' });
      let expression = 'document.querySelector("#text-field").focus()';
      await this.evaluate(expression);
      await this.page.goto('https://bar.com', { waitUntil: 'networkidle2' });
      await this.evaluate(expression); // Will this fail?
   }
}

Or would this fail each time the page changes via navigation?

Sebastiann · Jun 1, 2020

Or would it need to look like

Code:

async evaluate(expression) {
   const mainFrame = await this.page.mainFrame();
   const worldName = 'iso' + process.hrtime.bigint();
   const iso = await this.page._client.send('Page.createIsolatedWorld', { frameId: mainFrame._id, worldName: worldName });

   await this.page._client.send('Runtime.evaluate', {
      expression: expression,
      contextId: iso.executionContextId
   });
}

sockpuppet · Jun 1, 2020

The context gets destroyed in the navigation.
I would do something like this:

Code:

async createContext() {
   const mainFrame = await this.page.mainFrame();
   const worldName = 'iso' + process.hrtime.bigint();
   this.iso = await this.page._client.send('Page.createIsolatedWorld', { frameId: mainFrame._id, worldName: worldName });
}

async evaluate(expression) {
   await this.page._client.send('Runtime.evaluate', {
     expression: expression,
     contextId: this.iso.executionContextId
   });
}

Sebastiann · Jun 2, 2020

sockpuppet said:
The context gets destroyed in the navigation.

Hmmm ok, so would a listener on each navigation change be reliable?

Code:

this.page.on('framenavigated', async frame => {
   if (frame !== this.page.mainFrame()) {
      return;
   }
   const worldName = 'iso' + process.hrtime.bigint();
   this.iso = await this.page._client.send('Page.createIsolatedWorld', { frameId: frame._id, worldName: worldName });
});

I've heard that there's a "lag" on some listeners, but that seems like it would run each time the mainFrame's navigation changes right?

Thanks for replying!

Wendy logan · Jun 2, 2020

Thanks for this

sockpuppet · Jun 3, 2020

Sebastiann said:
Hmmm ok, so would a listener on each navigation change be reliable?

No idea if that works, you have to try it. But that looks overcomplicated, what is your use case?
I would just call createContext on every navigation step.

Sebastiann · Jun 3, 2020

sockpuppet said:
No idea if that works, you have to try it. But that looks overcomplicated, what is your use case?
I would just call createContext on every navigation step.

It seems to work but it's hard to know if it will be stable, I think I will just call it manually each time as you suggest to be sure. Use case was more just code brevity (ie not having to add a line to reinitialise each time it navigates haha)...

Loving your posts on this btw, do you currently use Puppeteer or something else?

I'm always trying to harden my Twitter actor as much as possible, so there's been a ton of improvements I've been able to make from you and greydingo's info so far...

How to detect puppeteer with 100% accuracy

Junior Member

Registered Member

Banned - Multiple Rules Violations

Junior Member

Regular Member

Junior Member

Junior Member

Power Member

Junior Member

Junior Member

Junior Member

Junior Member

Junior Member

Junior Member

Junior Member

Junior Member

Junior Member

Junior Member

Junior Member

Junior Member

Main Menu

Marketplace

Making Money

BlackHat World