1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scrapebox how to find more links from a list?!

Discussion in 'Black Hat SEO' started by meinewelt, Jan 18, 2012.

  1. meinewelt

    meinewelt Junior Member

    Joined:
    Jan 11, 2012
    Messages:
    157
    Likes Received:
    60
    Hi all,

    I have a question thats bothering me since months but I just cannot figure out how to do this:

    Lets say you have a list of 1000 auto approve blogs on unique domains.

    These Blogs each have sometimes 5, sometimes 10, sometimes 50 posts or more.

    How would you scrape other blog post URLs from this list of 1000 blogs?!

    It would require 1000 queries to google, right?!

    Thanks to anyone who is smarter than me..
     
  2. Joegromak

    Joegromak Newbie

    Joined:
    Oct 13, 2011
    Messages:
    19
    Likes Received:
    4
    Scrape the aa list with
    site:domain1
    site:domain2
    site:domain3... and so on, remove duplicate urls and that should do it.
     
    • Thanks Thanks x 1
  3. meinewelt

    meinewelt Junior Member

    Joined:
    Jan 11, 2012
    Messages:
    157
    Likes Received:
    60
    Thanks thats a nice idea cos you could also specifically search for certain keywords in those blogs.

    but it would still require 1000 queries to google right? even with proxys thats probably a problem
     
  4. woot123

    woot123 Junior Member

    Joined:
    Jan 10, 2012
    Messages:
    113
    Likes Received:
    27
    That´s no problem ;) Just get some proxies of the proxy list section on BHW, test them, and go for it.
     
  5. alaltaierii

    alaltaierii Supreme Member

    Joined:
    Jun 11, 2010
    Messages:
    1,408
    Likes Received:
    349

    With proxies will not be a problem.
    You can use public proxies for harvesting inner pages from 1000 sites with about 20-30 connections or even use private proxies with 2-3 connections.
     
  6. takeachance

    takeachance Power Member

    Joined:
    Jul 31, 2009
    Messages:
    557
    Likes Received:
    412
    Location:
    The UK of A
    This is a good starting point. But to further refine your scraping you can also include more defined elements to the site: scrape...i.e.

    site:spammedblog.com "leave a comment" "email (not required)" etc,etc

    This will then cut down your scrapes to only the pages which are suitable for commenting :)