Hey matt,
About the method of harvesting urls at 2 max connections.
Do you know any way to harvest in the same way but with special operators?
instead of harvesting again and again public proxies..
For example, if I want to scrape G, Y and B for a site:
http://domain.com %KW%
with thousands of keywords without having to stop and change proxies each time..
Thanks!
Well you could try with like 10% meaning 1 connection rather then 2. But even with 2 connections I am able to pull a lot of Site:
http://www.domain.com results with minimal blocking. AOL gives more site:
http://www.domain.com results then does Google, in my experience. Just heads up on that one.
Also bing supports site:
http://www.domain.com however yahoo does not support it. So you can add those in to get more results, but AOL and Bing both block me quicker then does Google, which is an irony, lol.
But again I get minimal blocking from google with 10-20% connections for advanced operators, but it does depend. I would say there is different levels/combos of ban rules inside advanced operators too. I mean it boils do to this:
The more processing power that your query uses on googles end, the faster they are going to block you. OR so it seems to me and that is how I would do it if I were them.
Experiment and see.
Possible bug report
OK, this has me a bit worried - I just tried to visit one of my competitors' site and found that my home IP address was blocked by StopForumSpam (stopforumspam.com). There are 5 reports for my IP address (one a day, Feb 14th-19th, no report on 17th) and they appear to be coming from scrapebox (same email format and keywords I'm blasting atm).
I checked SB and I have "Use proxies" checked. All my proxies are private and I have around 10. All I am doing is using fast poster to post to a huge list of auto approve blogs. I ran the link checker once. Does anyone know what might be causing this? Thanks in advance.
I have blasted from 1 ip for over a year and never had this problem and all my ips on servers and home IPs are clean and I have scrapebox running all over the place.
However I have gotten my home IP on stopforumspam.com before and it was just were I did a run and forgot to check the use proxies box. Also if one of your proxies went on vacation and decided to start passing along your IP, then that would do it.
But I can watch connections in a debugger, and see the connections hitting the proxies. I mean I haven't tried it on the .37 that just came out, but it didn't say that there were any poster or proxy changes so I doubt that that portion of the code was even messed with. Just guessing, but your dates are before that release anyway, so mute point.
Use the proxy checker to make sure that your proxies are passing the IP test and then always make sure that you checking use proxies.
I know once upon a day I forgot to check it and realized it later and then I got a notice from my ISP some days later about a complaint. After that I am mad crazy about triple checking that I have the use proxies box checked before I hit start posting.