1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Looking for tool that can check huge lists of urls to see if these links are still live

Discussion in 'Black Hat SEO Tools' started by belgianguy, Oct 9, 2013.

  1. belgianguy

    belgianguy Registered Member

    Joined:
    Oct 7, 2010
    Messages:
    76
    Likes Received:
    13
    I'm not talking about checking if they still contain a link to my site. Just to see if these pages still exist. I don't have scrapebox, but I do have paid version of gscraper, but this doesn't do the trick.

    Any suggestions?
     
  2. Jssom

    Jssom Regular Member

    Joined:
    Feb 25, 2013
    Messages:
    406
    Likes Received:
    190
    Location:
    Kuwait
    I can make u one if u r paying anyway
     
  3. Mr.Whitehat

    Mr.Whitehat Senior Member

    Joined:
    Apr 23, 2009
    Messages:
    855
    Likes Received:
    220
    Occupation:
    Wandering Around !
    Location:
    Dating Moolah Babe^
    get scrapebox if you have other needs too like commenting, scraping etc. i guess it has that option to check if url is valid
     
  4. deangregor

    deangregor Junior Member

    Joined:
    Jul 15, 2013
    Messages:
    173
    Likes Received:
    33
    You mean a tool that can check the list of urls to see if those urls are still alive? Try to check urlopener.com/index.php...might help you.
     
  5. W9go

    W9go Jr. VIP Jr. VIP Premium Member

    Joined:
    May 16, 2011
    Messages:
    3,379
    Likes Received:
    632
    Gender:
    Male
    Occupation:
    chasing girls
    Location:
    chasing girls
    scrapebox can...there are free webpages as far i remember but it was not conftable to use them ... i use SB for a long time now. its cheap and you should get it if you need this regular.
     
  6. belgianguy

    belgianguy Registered Member

    Joined:
    Oct 7, 2010
    Messages:
    76
    Likes Received:
    13
    That looks great but doesn't really work for a list of 100,000 urls
     
  7. remon24

    remon24 Regular Member

    Joined:
    Oct 3, 2012
    Messages:
    497
    Likes Received:
    404
    Location:
    NEW DELHI
    So you are looking for a 404 checker try xenu link sleuth
     
  8. HatIsBlacked

    HatIsBlacked Regular Member

    Joined:
    Dec 30, 2010
    Messages:
    224
    Likes Received:
    57
    look for a broken link checker app.

    since you are checking a large list, youll want the app to be multithreaded and support multiple http connections to make it run faster. youll also want it to export results to csv, say, so you can use spreadsheet to filter out on status code.
     
  9. SEOking222

    SEOking222 Jr. VIP Jr. VIP

    Joined:
    Sep 29, 2012
    Messages:
    1,219
    Likes Received:
    138
    Occupation:
    LOL ya right
    Location:
    Ghosted
    Home Page:
    Are you looking for url checker tools? what is your budget?
     
  10. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    8,862
    Likes Received:
    7,469
    Occupation:
    ZLinky2Buy SEO Services
    Location:
    ⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩
    Home Page:
    Write your own Perl script. Search for WWW::Mechanize - it's a golden tool chest for any SEO.
     
  11. botrockets

    botrockets Regular Member

    Joined:
    Mar 16, 2013
    Messages:
    272
    Likes Received:
    463
    Occupation:
    Software Developer
    Location:
    Saint T.N.