how do you guys scrape thousands of blogs?

in scrapebox, when I try to scrape, I get about 800 blogs (after removing duplicate URLs)
when I try to scrape again, I get more, but then after removing duplicate, I end up with the same 800 blogs.

I do this 2 ways:
search for something very generic, like John
search for spammy comments, like "I bookmarked your site" or another comment that a spammer always uses.
both ways, I only get about 800. I know there are a lot more blogs that contain those keywords, how come I am not getting them?

make sure you have the max number of urls to scrape to 1000 per keyword

Using elance to outsource the work has been great for me. Some good guys in Romania or India.

Bartman said:

I do this 2 ways:
search for something very generic, like John
search for spammy comments, like "I bookmarked your site" or another comment that a spammer always uses.
both ways, I only get about 800. I know there are a lot more blogs that contain those keywords, how come I am not getting them?

Click to expand...

Because you are just getting the first 1000 results from those keywords in which one site may dominate a large portion of those results (so you wind up with a lot of duplicates). You can improve the variety of domains that your search returns by adding additional keywords and footprints to your search. The more creative you are, the more results you can squeeze out of your keywords.

Removing duplicate domains and searching for a generic term like "nice blogs", I only got about 220 unique domains. Mixing it with a keyword list (so you are harvesting blogs about X that have comment Y on them somewhere) gives me several thousand unique domains.

Loopline also has a cool video that explains some extra google footprints you can use beyond the norm to get more results per keyword:

http://www.youtube.com/watch?v=hKC69k_cRsc

great forum

LOTS AND LOTS AND LOTS of keywords. Then you just need the right footprint. Then ScrapeBox crashes because its at 14 million on the multi-threaded harvester and you try to abort but it errors out and you think crap I lost all that work............except you didn't. Thank god for the Harvester crash saver thing.


irate:

800 blogs are a lot of blogs, be thankful. I have my partner look at this forum and he still says that manual submission is the best. On a weekly basis we are getting only 50 blogs. Now compare that with your 800, looks like a lot now doesn't it?

Get a better list of keywords and footprints. Like some one above said, LOTS AND LOTS of them! 800 from a single search isn't really that bad. Then again, it depends on what you do with them.