Bypass Google scrapping detection

[content spinner]

Hi deers seo-men and seo-women,
I'm working on a Google scrapper to perform a rankchecker.

And i would know how to scrap Google without get bust by their anti scrapping detection.
Indeed in the case i get caught i would know if it's possible to break the captcha (if it's not expensive, of course)

To bypass detection, i switch my user agent, waiting a random time beetween two query, delete google cookie


Thank you very much

 

[puneetas3]

I hate to break the secret but here it is. Get any cheap $5 vps that provides /64 or smaller IPv6 block. Install debian/ubuntu and setup a random 10K to 25K IPv6 addresses from your assigned block in your network interface. Now create a php script which can read from the interface and get all the 10K to 25K IPv6. Now use curl in your php script to randomly use any one of the IPv6 you just read. Make outgoing calls with curl to google and scrape as you like. With each request a random IPv6 is used. I personally use this setup and scrape at speed of 2 page/second and it doesn't get me banned on google. You can generate new random IPv6 from your block once a week and replace the old ones in the interface.

You can use it to scrape any IPv6 enabled site. This works great for scraping on server side. If you are using a software to scrape then you will need to do following. Edit windows host file and make it so that google.com is associated with your server's IPv4. Now whenever you make request to google.com all files will be served through your above php curl script which is listening at IPv4 address and making outgoing request with random IPv6.

If you do lots of scraping you can just add multiple cheap vps with different /64 block.

 

[content spinner]

Aw what an awesome trick !
Thank you very much for your answer puneetas3

 

[SEO FOX]

I hate to break the secret but here it is. Get any cheap $5 vps that provides /64 or smaller IPv6 block. Install debian/ubuntu and setup a random 10K to 25K IPv6 addresses from your assigned block in your network interface. Now create a php script which can read from the interface and get all the 10K to 25K IPv6. Now use curl in your php script to randomly use any one of the IPv6 you just read. Make outgoing calls with curl to google and scrape as you like. With each request a random IPv6 is used. I personally use this setup and scrape at speed of 2 page/second and it doesn't get me banned on google. You can generate new random IPv6 from your block once a week and replace the old ones in the interface.

You can use it to scrape any IPv6 enabled site. This works great for scraping on server side. If you are using a software to scrape then you will need to do following. Edit windows host file and make it so that google.com is associated with your server's IPv4. Now whenever you make request to google.com all files will be served through your above php curl script which is listening at IPv4 address and making outgoing request with random IPv6.

If you do lots of scraping you can just add multiple cheap vps with different /64 block.

Nice way dude. thanks for info.

 

[Asterixpro]

I hate to break the secret but here it is. Get any cheap $5 vps that provides /64 or smaller IPv6 block. Install debian/ubuntu and setup a random 10K to 25K IPv6 addresses from your assigned block in your network interface. Now create a php script which can read from the interface and get all the 10K to 25K IPv6. Now use curl in your php script to randomly use any one of the IPv6 you just read. Make outgoing calls with curl to google and scrape as you like. With each request a random IPv6 is used. I personally use this setup and scrape at speed of 2 page/second and it doesn't get me banned on google. You can generate new random IPv6 from your block once a week and replace the old ones in the interface.

You can use it to scrape any IPv6 enabled site. This works great for scraping on server side. If you are using a software to scrape then you will need to do following. Edit windows host file and make it so that google.com is associated with your server's IPv4. Now whenever you make request to google.com all files will be served through your above php curl script which is listening at IPv4 address and making outgoing request with random IPv6.

If you do lots of scraping you can just add multiple cheap vps with different /64 block.

How do you take care of latency and errors?

 

[puneetas3]

How do you take care of latency and errors?

Curl can handle error and output the error. If there is error make the last request again. There shouldn't be any latency issue. Its just like browsing normal site.

 

[Ramse]

I hate to break the secret but here it is. Get any cheap $5 vps that provides /64 or smaller IPv6 block. Install debian/ubuntu and setup a random 10K to 25K IPv6 addresses from your assigned block in your network interface. Now create a php script which can read from the interface and get all the 10K to 25K IPv6. Now use curl in your php script to randomly use any one of the IPv6 you just read. Make outgoing calls with curl to google and scrape as you like. With each request a random IPv6 is used. I personally use this setup and scrape at speed of 2 page/second and it doesn't get me banned on google. You can generate new random IPv6 from your block once a week and replace the old ones in the interface.

You can use it to scrape any IPv6 enabled site. This works great for scraping on server side. If you are using a software to scrape then you will need to do following. Edit windows host file and make it so that google.com is associated with your server's IPv4. Now whenever you make request to google.com all files will be served through your above php curl script which is listening at IPv4 address and making outgoing request with random IPv6.

If you do lots of scraping you can just add multiple cheap vps with different /64 block.

This neat little trick came out of nowhere. Thank you for sharing it

 

[sergey007]

Puneetas3, wow, have been looking for a way to solve the problem and it seems that you have just solved it. Thanks a lot!

 

[puneetas3]

Thought to create a video showing scraping on the vps using the above method, ofcourse if I get time.

 

[Mister Don]

Great solution you have there. I would be eager to see that video!

 

[M4DM4X]

I'm searching for a coder who can implement this on my server for a few bucks

 

[viliche]

I hate to break the secret but here it is. Get any cheap $5 vps that provides /64 or smaller IPv6 block. Install debian/ubuntu and setup a random 10K to 25K IPv6 addresses from your assigned block in your network interface. Now create a php script which can read from the interface and get all the 10K to 25K IPv6. Now use curl in your php script to randomly use any one of the IPv6 you just read. Make outgoing calls with curl to google and scrape as you like. With each request a random IPv6 is used. I personally use this setup and scrape at speed of 2 page/second and it doesn't get me banned on google. You can generate new random IPv6 from your block once a week and replace the old ones in the interface.

You can use it to scrape any IPv6 enabled site. This works great for scraping on server side. If you are using a software to scrape then you will need to do following. Edit windows host file and make it so that google.com is associated with your server's IPv4. Now whenever you make request to google.com all files will be served through your above php curl script which is listening at IPv4 address and making outgoing request with random IPv6.

If you do lots of scraping you can just add multiple cheap vps with different /64 block.

I wanted to scrape Google using a wide IPv6 range. I didn?t tested it with 10K or 25K but "only" 500 random addresses. I?ve started scrapping Google (3 firsts serps, 1 query for each) using specific useragent and headers (well, simulating a browser as much I can with Curl). Despite using a different IPv6 for each request, and using a random query interval of 2-10 seconds, I got always blocked after scraping 80 URLS or so. I really wonder what made your solution worked so fine? Any clue? (Or any question, please feel free to ask).

 

[epitome]

2-10 secs probably isn't enough delay on a single ip to keep it alive for very long

 

[viliche]

2-10 secs probably isn't enough delay on a single ip to keep it alive for very long

I agree, it is just that puneetas3 said he scraped at speed of 2 page/second without getting banned on google using this setup. My guess is that using thousands of IPv6 equals to use a single IPv4. On my side, running a query every 15sec looks optimum when using a single IP.

 

[DSNYC]

You just need to take this method, turn it into some bullshit SAAS platform, and sell it to BHW users for $39.99 / month.

 

[razzaguhl]

The method http://www.blackhatworld.com/members/puneetas3.254804/ mentioned is not working for me.
I scraped with a 45sec delay, but after a few minutes google bans the hole /64 IPv6 block and just send me errors (503)

 

[puneetas3]

No wonder, still working in 2018. Damn you Google.

 

[GregFromMoonsy]

The method mentioned by @puneetas3 working fine in situation, where around your block nobody else trying to do same.
If, so - Google quickly reduce ban limit.

For this reason for some people it working for other not.
The same situation is with IPV4.

The best results giving use exotic vps, not known by too much other scrapers.

Greg.