1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Screaming Frog for Scraping

Discussion in 'White Hat SEO' started by DeadJoe, Dec 7, 2016.

  1. DeadJoe

    DeadJoe Power Member

    Joined:
    May 18, 2014
    Messages:
    557
    Likes Received:
    272
    Location:
    The Oblivion
    Hey guys,

    has anyone used screaming frog to look for expired domains? It's a really cool tool too check for 404 errors, however I keep running out of memory even though I made lots of tweaks. I'm on a 8 gm ram computer.

    Anyway, I can look into that, just wondering if anybody here had any success with looking for expired domains? getting massive seed lists and so on.

    Thanks

    DJ
     
  2. scribbles

    scribbles Newbie

    Joined:
    Nov 10, 2016
    Messages:
    26
    Likes Received:
    6
    Gender:
    Male
    honestly, I don't know much about Screaming frog but I can help you with the tools I use:

    either

    in Netpeak Checker– select parameters (1) On-Page –Status Code and (2) Whois– Expiration Date

    or

    in Netpeak Spider – enter all domains, choose the 'list of URLs' crawling mode, and select the necessary parameters

    Links to the tools: https://netpeaksoftware.com/checker and https://netpeaksoftware.com/spider

    hope it helps, PM me if you have questions
     
    • Thanks Thanks x 2
  3. ThatSEO

    ThatSEO Jr. VIP Jr. VIP

    Joined:
    Jan 22, 2016
    Messages:
    733
    Likes Received:
    313
    Gender:
    Male
    Occupation:
    Self employed marketing stuff
    Location:
    Sometimes UK
    • Thanks Thanks x 1