1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How can I check broken links for a list of URLs?

Discussion in 'Black Hat SEO Tools' started by Traste, Dec 23, 2013.

  1. Traste

    Traste Regular Member

    Joined:
    Dec 18, 2011
    Messages:
    304
    Likes Received:
    22
    Hi,

    I want to check broken links for a list of 500 websites (including all pages). Xenu can do this for 1 site, but I can't find a way to have it automatically check broken links for next website once it's done. I also use Scrapebox, but no luck so far... Any ideas? Thanks
     
  2. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,119
    Likes Received:
    28,558
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
  3. Traste

    Traste Regular Member

    Joined:
    Dec 18, 2011
    Messages:
    304
    Likes Received:
    22
  4. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,119
    Likes Received:
    28,558
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
  5. Traste

    Traste Regular Member

    Joined:
    Dec 18, 2011
    Messages:
    304
    Likes Received:
    22
    Thanks, but it won't let you run a list of websites to check broken list :-/ Thanks
     
  6. Traste

    Traste Regular Member

    Joined:
    Dec 18, 2011
    Messages:
    304
    Likes Received:
    22
    Bump! Still looking :-/
     
  7. jhakasseo

    jhakasseo Senior Member

    Joined:
    Mar 1, 2012
    Messages:
    827
    Likes Received:
    237
    Occupation:
    Internet Marketing
    Location:
    On d Earth
    Home Page:
  8. Traste

    Traste Regular Member

    Joined:
    Dec 18, 2011
    Messages:
    304
    Likes Received:
    22
  9. tahworld

    tahworld Regular Member

    Joined:
    Aug 16, 2013
    Messages:
    457
    Likes Received:
    393
    Location:
    ✔✔✔✔✔✔✔
    What's wrong with using scrapebox alive checker?
    I use it all the time. It works great.
     
  10. Ballan

    Ballan Junior Member

    Joined:
    Oct 29, 2013
    Messages:
    105
    Likes Received:
    17
    In Xenu you can import a list of domains "check URL list" (I think) - will work through multiple domains checking links
     
  11. Traste

    Traste Regular Member

    Joined:
    Dec 18, 2011
    Messages:
    304
    Likes Received:
    22
    Thanks, but that's not what I need. That addon just checks if URLs are live. What I need is to check every single outbound link for the whole site:

    1. The bot goes to domain.com and check all outbound links.
    2. The bot goes to domain.com/page1/ and check all outbound links.
    3. The bot goes to domain.com/page2/ and check all outbound links.
    ...
    N. The bot goes to domain.com/pageN/ and check all outbound links.

    After that, I need to export the list of broken outbound links. Xenu will do that, but just for one site. I need the bot to repeat this process for a list of URLs.


    This option just checks every single URL in the file, not every page for every site in the list.

    I guess I didn't explain myself clear, sorry about that. Let's do it with an example:

    I have list.txt, which contains:

    site1.com
    site2.com
    site3.com

    What I need it the software to do this:

    1. The bot goes to site1.com and check all outbound links.
    2. The bot goes to site1.com/page1/ and check all outbound links.
    3. The bot goes to site1.com/page2/ and check all outbound links.
    ...
    N. The bot goes to site1.com/pageN/ and check all outbound links.

    When the bot is done with site1, it needs to do the same with site2:

    1. The bot goes to site2.com and check all outbound links.
    2. The bot goes to site2.com/page1/ and check all outbound links.
    3. The bot goes to site2.com/page2/ and check all outbound links.
    ...
    N. The bot goes to site2.com/pageN/ and check all outbound links.

    Xenu does it for one site only.
     
  12. Ballan

    Ballan Junior Member

    Joined:
    Oct 29, 2013
    Messages:
    105
    Likes Received:
    17
    I'm sure the xenu list option checks that - do you have the depth setting sufficiently high and "check external links" selected?

    Posted via Topify using iPhone/iPad
     
  13. Sephirot_90

    Sephirot_90 Registered Member

    Joined:
    May 20, 2011
    Messages:
    97
    Likes Received:
    36
    Location:
    Belgrade
    I have a trick that is entirely online.

    You need, check my links chrome extension and a site http://www.linkrr.com/

    Place all the links in http://www.linkrr.com/ and click on Make clickable. Then just activate check my links extension.

    Thats it :)
     
  14. shikind11

    shikind11 Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 28, 2012
    Messages:
    314
    Likes Received:
    43
    Occupation:
    CTO (Chief Thinking Officer)
    Location:
    Heaven on Earth!
    Scrapebox might do..I know a friend of mine used HREFER of senuke software to do this..

    Posted via Topify on Android
     
  15. pxoxrxn

    pxoxrxn Supreme Member

    Joined:
    Dec 21, 2011
    Messages:
    1,397
    Likes Received:
    2,066
  16. Ballan

    Ballan Junior Member

    Joined:
    Oct 29, 2013
    Messages:
    105
    Likes Received:
    17
    Hi OP,
    I've validated Xenu does do this - I uploaded 2 domains to Xenu as a file and set a high depth - it's now up to 115k urls to check (arbitrary, as the sites were arbitrary but considerably more than 2)

    For the exercise you are attempting, I'd be inclined to take all those urls that come up as "skipped external" (I set this as off) and rerun them with depth 0.

    The only thing this would prevent you from doing is finding which domain had which missing link. You could probably do something with the tab export from the first pass, but I'm ont certain.
     
  17. Traste

    Traste Regular Member

    Joined:
    Dec 18, 2011
    Messages:
    304
    Likes Received:
    22
    Sorry it took me so long to reply ("new reply" notification won't get to my inbox). I have retested this in Xenu and I found why it did not work before:

    [​IMG]

    It does work as you say, but if the domain uses a 301 redirect from the www version to the non-www version (or viceversa) Xenu won't go through it. Thanks!
     
  18. websemgem

    websemgem Newbie

    Joined:
    Feb 5, 2011
    Messages:
    20
    Likes Received:
    3
    Late to the pary, but you can automate A1 Website Analyzer for this to some degree:

    1) Create a project file using the settings you want. Set root domain to e.g: example com
    2) Now call A1 Website Analyzer command line from e.g. a bat file. In the parameters, specify the project file and override the domain. (Command line also suppors exporting e.g. all external URLs found.)
    3) Repeat #2 for each domain, i.e. line per domain in the .bat file
     
  19. AnthonyCorleone

    AnthonyCorleone Jr. VIP Jr. VIP

    Joined:
    May 9, 2013
    Messages:
    487
    Likes Received:
    273
    I code a such tool for my self back in a day works really simple in settings you type your site and in .txt file named bl.txt you load all your links and it will checked all of them pretty quick also and gave you in few seconds 2 .txt files of bad (no-follow) and good (do-follow) based on your list
    hit my PM if you need this
     
  20. King SEO

    King SEO Jr. VIP Jr. VIP

    Joined:
    Mar 19, 2014
    Messages:
    1,782
    Likes Received:
    139
    Occupation:
    Full Time Marketer