Detect Bot
- Thread starter blogjomar
- Start date
0xxi3
Regular Member
- Mar 25, 2016
- 358
- 303
I saw your previous thread here and look, if you're asking this question here instead of fiddling with it on your own then you have an incredibly steep learning curve ahead of you. Don't take it as me shitting over your idea and telling you not to try. Do try. But know ahead that it's a rabbit hole that goes deep.
I, myself, have not done full on cloaking but I have done some noob-level snooping to see how Adwords verifies your website.
Let's say you start off with a """simple task""" of detecting a PPC network crawler and displaying alternative content to it.
Step 1. Set up a website where you log visitors' UA, IP address and set a visits count cookie
Step 2. Create a campaign and start collecting data.
Step 3. Realize that, hey, all of those Google and Adwords crawlers have a UA that announces them as crawlers.
At this point you're thinking that you're done, you set up your server to show version A of the website to bots and version B to real visitors. You take in account IP addresses that you've detected and blacklist their subnets as potential crawlers, you also figure out whether there were any real browsers visiting your website before ads started running and blacklist those IPs and their subnets as well. Simple, right? Let's go through some possible threats (to your cloaking system) and respective solutions. Remember that crawlers will likely be visiting your page from time to time even after your campaign launches.
Threat 1. Crawler changes its UA to impersonate a browser
Solution 1. Detect whether it's a "dumb" crawler by executing JavaScript
Threat 2. Crawler can parse and execute your JavaScript
Solution 2. Verify whether actual JavaScript execution environment matches its stated browser as seen in UA
Threat 3. Crawler is actually a Puppeteer, headless Chromium browser
Solution 3. Check whether it supports all features that a real-real Chrome is supposed to support
Threat 4. Puppeteer was set up in a way that bypasses your checks
Solution 4. Measure its behaviour to see if its human-like e.g. if it dispatches correct events and moves its mouse
Threat 5. You have no fucking clue how to measure human-like behaviour because people are unpredictable, sometimes worse than bots and this would require client-server communication which would slow down redirect to version B of your website for real people.
Solution 5. ??? Do it anyway?
As you can see, in 5 simple steps you've been if not pwned then at least crippled. Many real visitors will bounce because any complex behaviour checks will require you to collect at least some data, communicate it to your server (which takes time) and as we all know a few seconds of wait is already too slow for your average user. Now, there is no guarantee that PPC crawlers go beyond level threat 3 above. So you can try. In fact, if you want to learn more about this topic you should try. Just don't expect crazy results.
I, myself, have not done full on cloaking but I have done some noob-level snooping to see how Adwords verifies your website.
Let's say you start off with a """simple task""" of detecting a PPC network crawler and displaying alternative content to it.
Step 1. Set up a website where you log visitors' UA, IP address and set a visits count cookie
Step 2. Create a campaign and start collecting data.
Step 3. Realize that, hey, all of those Google and Adwords crawlers have a UA that announces them as crawlers.
At this point you're thinking that you're done, you set up your server to show version A of the website to bots and version B to real visitors. You take in account IP addresses that you've detected and blacklist their subnets as potential crawlers, you also figure out whether there were any real browsers visiting your website before ads started running and blacklist those IPs and their subnets as well. Simple, right? Let's go through some possible threats (to your cloaking system) and respective solutions. Remember that crawlers will likely be visiting your page from time to time even after your campaign launches.
Threat 1. Crawler changes its UA to impersonate a browser
Solution 1. Detect whether it's a "dumb" crawler by executing JavaScript
Threat 2. Crawler can parse and execute your JavaScript
Solution 2. Verify whether actual JavaScript execution environment matches its stated browser as seen in UA
Threat 3. Crawler is actually a Puppeteer, headless Chromium browser
Solution 3. Check whether it supports all features that a real-real Chrome is supposed to support
Threat 4. Puppeteer was set up in a way that bypasses your checks
Solution 4. Measure its behaviour to see if its human-like e.g. if it dispatches correct events and moves its mouse
Threat 5. You have no fucking clue how to measure human-like behaviour because people are unpredictable, sometimes worse than bots and this would require client-server communication which would slow down redirect to version B of your website for real people.
Solution 5. ??? Do it anyway?
As you can see, in 5 simple steps you've been if not pwned then at least crippled. Many real visitors will bounce because any complex behaviour checks will require you to collect at least some data, communicate it to your server (which takes time) and as we all know a few seconds of wait is already too slow for your average user. Now, there is no guarantee that PPC crawlers go beyond level threat 3 above. So you can try. In fact, if you want to learn more about this topic you should try. Just don't expect crazy results.