Scrapebox how to find more links from a list?!

M

meinewelt

Regular Member

Jan 11, 2012

225

90

• Jan 18, 2012

• #1

Hi all,

I have a question thats bothering me since months but I just cannot figure out how to do this:

Lets say you have a list of 1000 auto approve blogs on unique domains.

These Blogs each have sometimes 5, sometimes 10, sometimes 50 posts or more.

How would you scrape other blog post URLs from this list of 1000 blogs?!

It would require 1000 queries to google, right?!

Thanks to anyone who is smarter than me..

 

J

Joegromak

Newbie

Oct 13, 2011

19

4

• Jan 18, 2012

• #2

Scrape the aa list with
site:domain1
site:domain2
site:domain3... and so on, remove duplicate urls and that should do it.

 

M

meinewelt

Regular Member

Jan 11, 2012

225

90

• Jan 19, 2012
• Thread Starter Thread Starter

• #3

Thanks thats a nice idea cos you could also specifically search for certain keywords in those blogs.

but it would still require 1000 queries to google right? even with proxys thats probably a problem

 

woot123

Junior Member

Jan 10, 2012

113

27

• Jan 19, 2012

• #4

 

alaltaierii

Supreme Member

Jun 11, 2010

1,494

388

• Jan 20, 2012

• #5

meinewelt said:

Thanks thats a nice idea cos you could also specifically search for certain keywords in those blogs.

but it would still require 1000 queries to google right? even with proxys thats probably a problem

Click to expand...

With proxies will not be a problem.
You can use public proxies for harvesting inner pages from 1000 sites with about 20-30 connections or even use private proxies with 2-3 connections.

 

takeachance

Power Member

Jul 31, 2009

572

419

• Jan 22, 2012

• #6

Joegromak said:

Scrape the aa list with
site:domain1
site:domain2
site:domain3... and so on, remove duplicate urls and that should do it.

Click to expand...

This is a good starting point. But to further refine your scraping you can also include more defined elements to the site: scrape...i.e.

site:spammedblog.com "leave a comment" "email (not required)" etc,etc

This will then cut down your scrapes to only the pages which are suitable for commenting