1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to scapre an entire domain with scrapebox?

Discussion in 'Black Hat SEO Tools' started by dgfalk, Mar 4, 2011.

  1. dgfalk

    dgfalk Power Member

    Joined:
    Apr 26, 2010
    Messages:
    687
    Likes Received:
    94
    I found a domain that has alot of high PR pages for commenting on it and I want to find more pages on that domain using scrapebox.

    So far im tried using the custom footprint "site:" and that only gives me like 800 results. But when I do site:URL in google it shows over 6000 pages, how do I get all 6K pages?
     
  2. Zak_A

    Zak_A Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 16, 2008
    Messages:
    808
    Likes Received:
    873
    Gender:
    Male
    Occupation:
    WP designer & developer
    Location:
    Western Europe
    Sure you can't get 6k results with only one query, so you'll need to find some various queries which will return a different batch of pages from that site every time.

    Look at the URLs you harvested already and try to find some pattern in their structure.
    For example : Maybe the site is divided into categories, and has a URL structure like
    domain . com/category1/[etc.]
    domain . com/category2/[etc.]
    and so on.

    In this case, you can harvest way more URLs with queries like :
    site:domain . com + inurl:category1
    site:domain . com + inurl:category2
    etc...

    That's just an example, if it doesn't apply to your target site, try to find another pattern. But I guess you got the idea ;)

    Hope this helped :)
     
  3. crazyflx

    crazyflx Elite Member

    Joined:
    Nov 9, 2009
    Messages:
    1,674
    Likes Received:
    4,825
    Location:
    http://CRAZYFLX.COM
    Home Page:
    Do what zak said above, or you can also look for a sitemap on the site. Since it is a blog, it is very likely that they have one, and then you can simply use the ScrapeBox Sitemap Scraper AddOn.

    Or, you can use the ScrapeBox addon that scrapes all the internal links from a page. Give it one page, scrape all internal links from it, paste those links back into the internal link scraper, scrape again, repeat, repeat, repeat, etc. until you get no more new pages.
     
  4. intop

    intop Newbie

    Joined:
    Jul 9, 2010
    Messages:
    27
    Likes Received:
    1
    Zak_A good information for me.
    thanks
     
  5. JamesHenry

    JamesHenry Junior Member

    Joined:
    Dec 9, 2009
    Messages:
    157
    Likes Received:
    102
    didn't the link extractor add-on work well enough?
     
  6. cyberzilla

    cyberzilla Elite Member Premium Member

    Joined:
    Nov 15, 2009
    Messages:
    2,204
    Likes Received:
    3,363
    Location:
    zeta reticuli
    Last edited: Mar 5, 2011
  7. JamesHenry

    JamesHenry Junior Member

    Joined:
    Dec 9, 2009
    Messages:
    157
    Likes Received:
    102