C# + Google Searches

One factor that can be used in bot detection is whether or not a client is downloading images/css etc.

paincake said:

One factor that can be used in bot detection is whether or not a client is downloading images/css etc.

Click to expand...

this also can be faked...

xenon2010 said:

this also can be faked...

Click to expand...

Care to elaborate?

well its tricky task and doesn't worth the time and efforts if you want to accomplish this with httprequest or any other C# client. however, you need to bear in mind all of the stated facts are based on "imaginary" thoughts so you don't need to get into that road, coz you will end up thinking more than working which means = wasting your time. and eventually you are not gonna figure out what google is actually using to detect bots.

but if you really want to do that you can actually use a simple webpage control to download the page for you. all you need is to parse the innerHtml and that's it...

Where's a good source of google friendly proxies? I can't seem to get more than 1 or 2 if that from some of the lists I've found on the internet. I really want to get my program working better, but this has been the hardest part of it all.

Seuss said:

Where's a good source of google friendly proxies? I can't seem to get more than 1 or 2 if that from some of the lists I've found on the internet. I really want to get my program working better, but this has been the hardest part of it all.

Click to expand...

Good source = private proxies.
Free source = scrape proxies for all web sites you can, then check each proxy for (my tests)
1) hides your ip
2) can you make a post req
3) can you scrape search engines, just try to scrape a few results from the serps, if you can get more than one site, that proxy is good for that search engine

seo-madness said:

Use the Google API for searching. If you want more control you'll have to alter "user-agent" in the header of each request. Rotating proxies is a good idea to.

Click to expand...

thanx man..

It's been mentioned in here before but a standard rule you can use is

1.) Rotate a pool of user-agents (Minimal work and easy to do)
2.) Have plenty of proxies available (The Big G does alot with IPs and Cookies)
3.) Clear cookies each Search Query (easily done if you are rolling multiThreads. I like to keep all my 'Bot' functions that are performed each go in their own class and then Spawn a New Class each thread I do (this ensures you start with a clean slate each time (cookies...etc) It's basically spawning a new bot each time if you want to think of it in laymans terms.

Ok. Yes. Threads would make sense, and I was going to incorporate that into the project as well. I guess this scraper of mine will be at a stand still until I get some private proxies.

Seuss said:

I guess this scraper of mine will be at a stand still until I get some private proxies.

Click to expand...

There are those who suggest that using public proxies is better for scraping than using private proxies because if your private proxies are blocked... they're blocked, whereas if your public proxies are blocked, you scrape more.