1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Can I use scrapebox to scrape a complete site and look for expired domains?

Discussion in 'Black Hat SEO' started by jon_xx_x, Mar 21, 2016.

  1. jon_xx_x

    jon_xx_x Jr. VIP Jr. VIP

    Joined:
    Nov 15, 2008
    Messages:
    3,230
    Likes Received:
    1,493
    Started using SB again, and wondering if there is a way I can scrape a whole site and look for links to expired websites? ie scrape CNN for all their outbound links, then check if any of those links are expired? I'm assuming there's a way, hopefully someone can help.
     
  2. socialsmartm

    socialsmartm BANNED BANNED

    Joined:
    Nov 6, 2014
    Messages:
    94
    Likes Received:
    7
    Gender:
    Male
    i was looking for that also, but i didn't found any solution :(
     
  3. oliviarhymes

    oliviarhymes Newbie

    Joined:
    Mar 21, 2016
    Messages:
    8
    Likes Received:
    1
    You can either load its sitemap or use the site: function to scrape as many indexed pages. You can expand by further scraping using different keywords for relevant pages.
     
  4. Keyser_Soze

    Keyser_Soze Newbie

    Joined:
    Nov 2, 2011
    Messages:
    27
    Likes Received:
    3
    I think loopline might have some YT videos on this.
     
  5. pressrelease

    pressrelease Power Member

    Joined:
    Jan 6, 2016
    Messages:
    661
    Likes Received:
    235
    Location:
    Disneyland
    I had tried that earlier and its really a bad idea unless you run a vps.it will consume all system memory
    How to do
    Load ur
    Check for 301 and 302
    Export list
    Check again with expired checker
    Bang you are done.
     
  6. accelerator_dd

    accelerator_dd Jr. VIP Jr. VIP

    Joined:
    May 14, 2010
    Messages:
    2,448
    Likes Received:
    1,009
    Occupation:
    SEO
    Location:
    IM Wonderland
    It is possible, but it will take a lot of time for a site like that. Even more if there are some crawl-delay rules. I would segment it down into category or category/date based on URL and start that way, but I am not sure if SB can do that. Maybe Xenu is worth a shot there.
     
  7. Ambitious12

    Ambitious12 Elite Member

    Joined:
    Jun 26, 2014
    Messages:
    3,097
    Likes Received:
    608
    Occupation:
    No Occupation
    Location:
    Among the Stars
    Bro Why this thread hijacking?Better start your own no?