Idea about improving private proxies longevity in Google, what you think?

[innozemec]

OK, guys i have an idea in my mind since couple of days and wanted to see what you guys think about it..

First of all i am talking about private proxies only!

We all know G is getting pretty good at proxy detection and flagging.. I am thinking that for sure they must be using Javascript to help in identifying real people and bots, since most scrapers we use don't process javascript.. no matter how we limit our scraping requests and spread them randomly throughout the day private proxies are still detected sooner or later..

So, my idea is throughout the day several random search accesses to be done with javascript enabled browser control through each proxy as that might help in G thinking "ok, this IP is loading our javascript"

Ofcourse, i think of trying to code some basic vb.net automated bot to rotate the proxies and do random searches and different page visits in G using the Webbrowser control, spread throughout the day.. i have some basic vbnet coding skills..ofcourse this won't be with selling purpose.. don't get me wrong..

but mainly i wanted to share my thinking and see what you guys think about the idea and will it help in getting my private proxies more hard to flag..

 

[Gogol]

Yupp I also think they use JS to track each of the visits. Unfortunately it is something that is really tough for a web developer (not saying it's impossible, but I never tried it LOL). Any desktop app can do it easily. But then, you lose speed per thread if you do that!

 

[innozemec]

Yupp I also think they use JS to track each of the visits. Unfortunately it is something that is really tough for a web developer (not saying it's impossible, but I never tried it LOL). Any desktop app can do it easily. But then, you lose speed per thread if you do that!

why it would be tough for a web developer and why would i loose speed per thread?

Maybe you didn't understand me right..

For example i am using a bot to scrape G wit my private proxies
I make a totally separate bot with javascript capability that is set to make 3 searches with each proxy in a day. It spreads them by random throughout the hours in a day and each time picks a random keyword from a big list to search for in G. There is no need of threading or speed in here, as it will make just 3 requests per proxy for the whole day in order not to eat from the proxy allowed requests in G before a captcha is triggered..

 

[Gogol]

Oh ok sorry for misunderstanding that. Yeah it would be an unique idea for sure. Never heard of anyone doing it actually Indeed, that's very smart!

why it would be tough for a web developer and why would i loose speed per thread?

Maybe you didn't understand me right..

For example i am using a bot to scrape G wit my private proxies
I make a totally separate bot with javascript capability that is set to make 3 searches with each proxy in a day. It spreads them by random throughout the hours in a day and each time picks a random keyword from a big list to search for in G. There is no need of threading or speed in here, as it will make just 3 requests per proxy for the whole day in order not to eat from the proxy allowed requests in G before a captcha is triggered..

 

[Gogol]

Sorry for double posting , but one more thing I would like to point out, that I believe that google also traces what is being searched. As for example, when I search with queries that look like footprints (I have seen this a lot when I search for vulnerable sites using footprints ), I sometimes get Captcha after 5-6 trials in-spite of the fact that I have javascript enabled and I am not a bot. So, that's something that we can add to it too. Like may be searching for some normal random terms with those bots.

 

[proxygo]

biggest problem with the Google bans came from gscraper.
gscraper scrapes so aggressively even i locked my 2 sales threads
supporting that tool... its a 100% quick fire ban tool

 

[thedorf]

I follow you, good thoughts. I'll have my developer look at this thread, we need to increase our private proxy longevity.

 

[troofer]

odd. i have custom scraper that was developed with a piece of software (i bought the whole thing...and please don't ask about the soft ). and i never was hounded by G, i tried payday loans scrape and whatever, and not even a peep from G. and the odd thing is, no freaking proxies. I voiced my concerns with the developers regarding the proxies, and they just laughed it off. only recently i began seeing corruption in certain scraping results, which i was told it's being fixed. but my long tail keyword scrapes are never corrupted.

 

[innozemec]

Oh ok sorry for misunderstanding that. Yeah it would be an unique idea for sure. Never heard of anyone doing it actually Indeed, that's very smart!

yeah, i haven't seen someone doing it, so i think it is worth to try..

odd. i have custom scraper that was developed with a piece of software (i bought the whole thing...and please don't ask about the soft ). and i never was hounded by G, i tried payday loans scrape and whatever, and not even a peep from G. and the odd thing is, no freaking proxies. I voiced my concerns with the developers regarding the proxies, and they just laughed it off. only recently i began seeing corruption in certain scraping results, which i was told it's being fixed. but my long tail keyword scrapes are never corrupted.

well, it depends man.. for example from my PC i can scrape all day long without any worries, because we are on cables here and the infrastructure of my ISP is set up that way so our whole city goes out in the internet through one real IP address and i think they have an arrangement with G so that IP is whitelisted. I don't know it for sure, but thats my explanation.. otherwise there's no way 100k+ customers going out through same IP not to get the IP banned instantly...

Sorry for double posting , but one more thing I would like to point out, that I believe that google also traces what is being searched. As for example, when I search with queries that look like footprints (I have seen this a lot when I search for vulnerable sites using footprints ), I sometimes get Captcha after 5-6 trials in-spite of the fact that I have javascript enabled and I am not a bot. So, that's something that we can add to it too. Like may be searching for some normal random terms with those bots.

yes, i think to try to make the bot as advanced as possible like making random search requests both on G,YT, to visit blogspot, googlemaps and etc.. - so G can see this IP is not using just the search directly. Also to make random stays on pages and etc... random clicks on different page elements ... will try different stuff by trial and error.. any other ideas are welcome so we all can try and see if we can come up with some cool working bot to keep our proxies alive..

i also think to share my bot with the community, so we can all exchange info, further develop it with new ideas and make it kickass to beat google.. just one issue arises is that i am not good at publishing PC based software due to all those dependancies for the different libraries, so i might need some help on there.. once i had an idea to sell PC based software, developed a small app to try myself and when i published and tried to install it on several different PCs i had and on couple of VPS i bumped my head in wall big times and gave it up

 

[innozemec]

I follow you, good thoughts. I'll have my developer look at this thread, we need to increase our private proxy longevity.

thanks man, the more people to comment the idea, the better.. i won't mind if someone that is PRO in developing PC based apps to help or even make the bot in quick, should be piece of cake for someone with experience..