[GUIDE] How Ad Networks (and everyone else) Knows You're A Bot

TasDePixels said:

i personally use nightmare.js (based on electron) i switch useragent signature & proxy and bunch of other stuff as a paramaters in every call for full stealth.

it works.

Click to expand...

TasDePixels said:

i also tested the link you provided.

What's sweet about nightmare is that you can easly access a bunch of methods that define ur emulated navigator such as WebRTC etc..

Click to expand...

Really interesting, thanks for sharing. I'd never heard of nightmare.js before. You should share your script if you're open to it.

BrazilianBusinessman said:

OP, entering into the tech savy aspect, as sellenium can be seen as a automated tool, this means any tool that use it can be flagged, right?

I think I realized why many creation tools are leaving footprints that arent related to proxies but the tool itself

Click to expand...

I honestly don't know. If you're turning off / deactivating navigator.webdriver I think it would be OK but I don't know for sure.

TasDePixels said:

i personally use nightmare.js (based on electron) i switch useragent signature & proxy and bunch of other stuff as a paramaters in every call for full stealth.

it works.

Click to expand...

Follow-up here. If you have any cool / easy tutorials on using Nightmare.js, it might be fun to play around with. I mostly use Python for scraping but this looks like it might be a neat alternative.

AdvertisingGuy said:

I mostly use Python for scraping but this looks like it might be a neat alternative.

Click to expand...

i prefer node.js for any bot related matter.
example from the documentation page: https://github.com/segmentio/nightmare/blob/master/example.js

Pistacho said:

Yeah I always tell people that if you really must use a browser then puppeteer and Object.defineProperty are your friends. This is a good example of how to do it properly: https://gist.github.com/nicoandmee/7d7dc2e79e2d553a22645d312f39bf3c

Click to expand...

I've never been able to do canvas on chrome via injection. That example is detected for the same reasons.

Herion said:

I've never been able to do canvas on chrome via injection. That example is detected for the same reasons.

Click to expand...

It is pointless now a days sites like Google and the major Social Networks are finger printing to fight fraud and spam. They are mostly checking for feature detections so they are not checking for unique canvas images, those sort of finger prints have died down a lot in the last few years. The main things they check are features found in each User-Agent family to catch you using a fake a User-Agent so people using tools like Ghost browser and Multilogin are paying a shit load of money in vain because those tools are useless at evading feature checks.

Only a few things like your web gl renderes and a plugins list are the few things they will check. Which the above script already covers it, that’s how I found it because I needed a list of the web gl thingy for my bot.


So to those spending money on stupid stuff like multilogin app ask yourself this: do you have any idea which feature checks the site you are botting has on their browser fingerprint script and if you can’t answer that question then I would seriously ask you to have someone look at the site’s JavaScript code to find the feature checks. You will see how all that money you were paying for some snake oil solution was nothing but money down the drain, because chances are tools like multilogin and ghost browser have 0 of the feature checks spoofed.

These are some features:https://browserleaks.com/features

Trust me there are a tons more and even Youtube/Google uses some that are not there that will quickly make your multilogin app detected as a bot once you switch to a non chrome User-Agent

BOT FLAG #2 -- DATA CENTER IP ADDRESSES

The most common way of identifying the physical location of a website visitor is by analyzing their ip address. To see an example, open up a web browser, and go to https://whatismyipaddress.com. It should return your ip address and your physical location.

If you click on the map, you'll receive additional details gleaned from your ip, such as your ISP, and your connection type. My example is below--I am using a residential IP address from Verizon, the most popular Internet and phone carrier in the US. I’ve blocked any sign of my actual IP address because I don’t want to get hacked.

This is exactly how it should be--you visit a page with your normal IP address, and the website owner will believe that you are a normal website visitor. And so will the advertisers buying the ad space.

There is nothing suspicious here, nor should there be. But do a search for the ip address 18.223.106.225...

You'll see the ISP for that IP is Amazon.com. Amazon, and other cloud service providers such as Microsoft, Google, among others, own entire blocks of IP addresses they use to connect their remote servers to the Internet.

Nearly all bot users / manufacturers use cloud service companies to run their bots--cloud providers have "five nines" uptime, ultra-fast Internet that’s 400 MBPS or better and have multiple operating systems available. One can deploy multiple instances of a bot across dozens of computers instantly and send visits to a website. For an example of this in action, see this video:



While not every visitor from a data center IP address is a bot, nearly all bots come from data center IPs. Thus, 100% of data center IP addresses that visit a website and load an ad will be flagged as a bot or invalid traffic. The makers of this video would get flagged for every visit and if it were a real site, it would get blacklisted from the exchanges within days if this is where most of their traffic came from.

So if you’re buying traffic, and seeing it come in from a data center IP address such as AWS, Microsoft, Google, DigitalOcean, or any other cloud provider, it’s bot traffic and it will get flagged. Likewise, if you’re visiting a site, and using a highly anonymous proxy from a data center IP, or surfing through a data center VPN, this will also get flagged.

If you want to know in real time what IP addresses belong to what ISP / ASN there are dozens of api services where you send an IP address via a get request, and the API will send you back a JSON response with the ISP / ASN, geographic location and other parameters. ipstack.com and ipapi.co are two services that I’ve used and they work fine.

I've attached some data from my own tracking pixel below:

Cogent Communications is a provider of dedicated servers which are often used for scraping and VPNs. Those two visits would be flagged as data center traffic as well.

Really interesting post.

So as i've reading here seems like it is somethig like this: selenium < puppeteer < Nightmare

whiteaxe said:

Really interesting post.

So as i've reading here seems like it is somethig like this: selenium < puppeteer < Nightmare

Click to expand...

I don't think it's possible to use a browser automation framework like the ones you just mentioned to bypass ad verification services. Or at least it's not possible to do at a large scale or for a long time.

You can also speculate which os someone is using from a proxy i believe.

Which may trick some out if they are using a 4g proxie but appear to be coming from a windows OS

AdvertisingGuy said:

BOT FLAG #2 -- DATA CENTER IP ADDRESSES

While not every visitor from a data center IP address is a bot, nearly all bots come from data center IPs. Thus, 100% of data center IP addresses that visit a website and load an ad will be flagged as a bot or invalid traffic. The makers of this video would get flagged for every visit and if it were a real site, it would get blacklisted from the exchanges within days if this is where most of their traffic came from.

So if you’re buying traffic, and seeing it come in from a data center IP address such as AWS, Microsoft, Google, DigitalOcean, or any other cloud provider, it’s bot traffic and it will get flagged. Likewise, if you’re visiting a site, and using a highly anonymous proxy from a data center IP, or surfing through a data center VPN, this will also get flagged.

If you want to know in real time what IP addresses belong to what ISP / ASN there are dozens of api services where you send an IP address via a get request, and the API will send you back a JSON response with the ISP / ASN, geographic location and other parameters. ipstack.com and ipapi.co are two services that I’ve used and they work fine.

Click to expand...

Hey great posts and also thanks for your comment in my thread. When you query the ISP/ASN by IP, how do you know that they are a data center or not? Those api services above doesn't give such information.

The question is if they know I'm a dog??!?

AdvertisingGuy said:

I am an AdvertisingGuy after all.





I can't comment beyond what I've already said. If you have the right answer and can confirm, put it in this thread. Some of these things can't be gotten around, however.



Some of those things I'll cover in this series. I don't know of any "permanent cookies" that track users--Verizon used to do this and they were hit by a $1.35B fine. According to the below link it was pretty prevalent once upon a time, but I'm not sure if it's done now. A billion dollars is a good way to dissuade people.

https://qz.com/634294/a-short-guide-to-supercookies-whether-youre-being-tracked-and-how-to-opt-out/

As for the freshly wiped / installed browser, I know it's done with cookies. If you were to close Chrome, delete the Chrome cookie folder(s) and then open them back up, a bunch of new cookie folders would be recreated, especially after you visit your first site. But beyond the cookie, ad tech companies have additional ways to ID users. There are advanced fingerprinting techniques which take your graphics card, ip address / dma, browser version/user agent and can reliably ID people to a high degree of accuracy. I'll get into that stuff as well.

Click to expand...

t




this all thinning can be spoof.

What if you develop bot in C++ and it works completly outside of webbrowser?

velkan said:

What if you develop bot in C++ and it works completly outside of webbrowser?

Click to expand...

With http requests. Yeah its done a lot for things which need max efficiency and speed (ticketing and sneaker bots).

But you have to be on point with your anti bot detection and headers.

Btw it is better to create a bot using a webdiver like selenium or something like this, or go with http requests?

whiteaxe said:

Btw it is better to create a bot using a webdiver like selenium or something like this, or go with http requests?

Click to expand...

http requests if you have the skills you use far far less resources.

British Botter said:

You can also speculate which os someone is using from a proxy i believe.

Which may trick some out if they are using a 4g proxie but appear to be coming from a windows OS

Click to expand...

This I don't think is correct. There are all different types of connections, operating systems, etc. So this doesn't help with fraud (IMO).

British Botter said:

http requests if you have the skills you use far far less resources.

Click to expand...

Correct me if I'm wrong, but requests doesn't load JavaScript / etc. I think you need a real browser if the content is dynamically generated (think of Facebook where you scroll down the page and see additional content). I don't think requests has a way to handle this.

peakaboo said:

Hey great posts and also thanks for your comment in my thread. When you query the ISP/ASN by IP, how do you know that they are a data center or not? Those api services above doesn't give such information.

Click to expand...

They do give that information, as I included in my post. They will tell you if it's Google, MSFT, Amazon, etc. There is no flag that says "datacenter / not" but they will give you the ISP. You just need to know which companies are data centers.

velkan said:

What if you develop bot in C++ and it works completly outside of webbrowser?

Click to expand...

I don't know enough about C++ or programming to comment, but this is content is generally aimed towards ad networks / companies trying to ID bots. Unless it's in a real web browser, I'm sure the bot will be identified.

AdvertisingGuy said:

I don't know enough about C++ or programming to comment, but this is content is generally aimed towards ad networks / companies trying to ID bots. Unless it's in a real web browser, I'm sure the bot will be identified.

Click to expand...

What if we automate windows/OS rather than browser? (scalability is a issue but still...)

AdvertisingGuy said:

This I don't think is correct. There are all different types of connections, operating systems, etc. So this doesn't help with fraud (IMO).

Click to expand...

Windows and linux process tcp slightly differently you can do certain timing attacks to work out if someone is using a proxy and also whether or not its a linux or windows server I'm no expert on this but am very aware its possible.

AdvertisingGuy said:

Correct me if I'm wrong, but requests doesn't load JavaScript / etc. I think you need a real browser if the content is dynamically generated (think of Facebook where you scroll down the page and see additional content). I don't think requests has a way to handle this.

Click to expand...

Headless browsers are bulky and resource heavy at scale. For javascript heavy sites, developer tools are your friend. But if you're writing bots you will already know this.

So does tools like Jarvee get around this problem for the footprints or doesn't instagram care?