Url harvesting problem
- Thread starter antsaoo
- Start date
justone
BANNED
- Oct 12, 2008
- 1,520
- 1,054
also it is important to know that the number of hits google shows is usually WAY wrong.
If google says it knows a million websites about a topic it is very possible that you can not get out more than a few % of them and google does not really know much more.
I don't know how they estimate the total number of sites but that algorithm is very inaccurate at least.
If google says it knows a million websites about a topic it is very possible that you can not get out more than a few % of them and google does not really know much more.
I don't know how they estimate the total number of sites but that algorithm is very inaccurate at least.
genie1
Junior Member
- May 30, 2008
- 151
- 52
I did some reading on this a few months ago because I ran into the same problem. It ''seems'' and I might be wrong, that its part of google's design not to go past 1000 results. They claim something like ''if you can't find it in the first 1000 results, your search isn't focused enough'' lol
justone
BANNED
- Oct 12, 2008
- 1,520
- 1,054
and don't forget: if you hammer google too much you get soft banned which requires a captcha to continue.
If you continue to hammer google they temp ban your ip.
If you continue to use ips in the same subnet the same process happens.
if you keep hammering they temp ban the whole subnet.
After some hours to days the ban is lifted, no idea what happens if you keep scraping google, I read about permanent bans in various mailing lists.
If you continue to hammer google they temp ban your ip.
If you continue to use ips in the same subnet the same process happens.
if you keep hammering they temp ban the whole subnet.
After some hours to days the ban is lifted, no idea what happens if you keep scraping google, I read about permanent bans in various mailing lists.