1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Mass scraping of inner pages from domain?

Discussion in 'Black Hat SEO Tools' started by lmaestro, Jun 16, 2011.

  1. lmaestro

    lmaestro Power Member

    Joined:
    Jan 28, 2011
    Messages:
    666
    Likes Received:
    227
    How do i do this with scrapebox? I know you can do it by putting a domain as custom footprint but how do i scrape tons of domains?
     
  2. TheMatrix

    TheMatrix BANNED BANNED

    Joined:
    Dec 20, 2008
    Messages:
    3,444
    Likes Received:
    7,279
    Scrape a few keywords. Scrape a lot of URL's using those. Remove dupe, trip to root. Export to clipboard and paste into notepad.

    Next, find "http://", and replace with "site:http://"

    Now put these as keywords, and scrape.

    You will get all indexed pages of these domains.
     
    • Thanks Thanks x 1
  3. super nuver

    super nuver Newbie

    Joined:
    Aug 30, 2007
    Messages:
    42
    Likes Received:
    3
    wow i did this with public proxies and scraped about 57,000 urls. The most ive been getting is 4,000. Thanks again.