1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Bulk 404 Checker?

Discussion in 'Black Hat SEO' started by whiteblackseo, Jun 11, 2015.

Thread Status:
Not open for further replies.
  1. whiteblackseo

    whiteblackseo Jr. VIP Jr. VIP

    Joined:
    Apr 11, 2015
    Messages:
    2,494
    Likes Received:
    930
    Home Page:
    I'm scraping and I need a 404 checker. I know screaming frog seo spider tool, but it doesn't do the job right. I need a bulk 20,000-50,000 url checker. I can't find one online... :( Do you guys have one? Other than scrapebox dead or alive or Gscraper HTTP checker.
     
  2. loopline

    loopline Jr. VIP Jr. VIP

    Joined:
    Jan 25, 2009
    Messages:
    3,806
    Likes Received:
    2,027
    Gender:
    Male
    Home Page:
    So you don't want the Scrapebox one, and its accurate so I can only assume then that you don't own it and are looking for a free one?

    Or what is your qualification, other then it being a 404 checker, that you need it to meet?
     
  3. whiteblackseo

    whiteblackseo Jr. VIP Jr. VIP

    Joined:
    Apr 11, 2015
    Messages:
    2,494
    Likes Received:
    930
    Home Page:
    No, I don't own scrapebox, I own Gscraper free version. I need a software that will run bulk 404 checks, thats it.
     
  4. BassTrackerBoats

    BassTrackerBoats Super Moderator Staff Member Moderator Jr. VIP

    Joined:
    Mar 10, 2010
    Messages:
    16,748
    Likes Received:
    30,762
    Occupation:
    Selling CPA Sites
    Location:
    Not England
    Home Page:
    Scrapebox is only 57.00 using the BHW discount. You would pay 57.00 for a bulk 404 checker, wouldn't you?
     
  5. whiteblackseo

    whiteblackseo Jr. VIP Jr. VIP

    Joined:
    Apr 11, 2015
    Messages:
    2,494
    Likes Received:
    930
    Home Page:
    Pretty Broke ATM :(
     
  6. whiteblackseo

    whiteblackseo Jr. VIP Jr. VIP

    Joined:
    Apr 11, 2015
    Messages:
    2,494
    Likes Received:
    930
    Home Page:
    Shameless self-bump :(
     
  7. loopline

    loopline Jr. VIP Jr. VIP

    Joined:
    Jan 25, 2009
    Messages:
    3,806
    Likes Received:
    2,027
    Gender:
    Male
    Home Page:
    When I was broke I did grunt work to earn money. In fact when I bought Scrapebox 5 years ago for, $47 at the time, it was a scrape to get that. Today its a drop in the bucket.

    So I can honestly I understand where your coming from, but when you don't have money you find ways to work for less. So the question would be if you a tool you have now that just takes a lot more time shoving info in and out of until you can get more cash, or is it a better use of your time to go cut 2 of your neighbors lawns or answer an ad on craigslist for a saturday work or find something here on the form that can be done to earn $57 so you can buy Scrapebox and use it and then also have it to do more work to earn more money.

    I can't answer that question, but I can say what Zig Ziglar said is true, "You can have everything in life you want if you just help enough other people get what they want". So you will have to figure out how that applies to you and go chase down some cash or use your time to get the data. I think Xenu may work, or may narrow your list, it might not be an ideal fit but it could at least give you an idea of if something is dead or alive.

    You might try the scrapebox free link checker as well, shove in a fake link and your list and it will tell you not found on all of them, but it may return on error on sites that are dead rather then not found, I think. Try it and see.
     
    • Thanks Thanks x 4
  8. whiteblackseo

    whiteblackseo Jr. VIP Jr. VIP

    Joined:
    Apr 11, 2015
    Messages:
    2,494
    Likes Received:
    930
    Home Page:
    The scrapebox link checker is good. It just doesn't allow you to export 404's.

    I just wanted to say, I would love to buy scrapebox it's a great software. I just don't have the money for it, I have to save up for university and so on. Maybe in the coming years ahead I'll buy it.

    Thanks loopline :)
     
  9. Ambitious12

    Ambitious12 Elite Member

    Joined:
    Jun 26, 2014
    Messages:
    3,096
    Likes Received:
    609
    Occupation:
    No Occupation
    Location:
    Among the Stars
    What I know,Scrapebox is better tool to scarape 404.Try once for the bulk :)
     
  10. lagger

    lagger Power Member

    Joined:
    May 10, 2011
    Messages:
    511
    Likes Received:
    183
    Ive tried many tools, owned a few subscriptions to this and that. Most tools suck after a short shelf life. The only 2 tools that stand the test of time in usefulness would be scrapebox and gsa.
     
  11. fatboy

    fatboy Elite Member

    Joined:
    Aug 13, 2008
    Messages:
    1,618
    Likes Received:
    3,231
    Occupation:
    Retired
    Location:
    Old Peoples Home
    If you want a free PHP script, here you go - its basic, single threaded but if you leave it running it should be okay. First off create a text file called sites.txt and put the URLs you want to check in one per line and then save this as something like grab.php and run it in the same folder as sites.txt:

    Code:
    <?php
    error_reporting(0);
    $error_404 = "Not Found";
    $list_404 = array();
    $sites = file('sites.txt', FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
    
    foreach($sites as $site)
    {
            $response = get_headers($site);
            if (strpos($response[0], $error_404))
            {
                    $list_404[]=$site."\n";
            }
    }
    
    file_put_contents('404.txt',$list_404);
    ?>
    
    At the end, all of the 404 URLs will be in the file called 404.txt

    However, I would recommend stumping up some cash for something multithreaded and fast.
     
    • Thanks Thanks x 4
  12. whiteblackseo

    whiteblackseo Jr. VIP Jr. VIP

    Joined:
    Apr 11, 2015
    Messages:
    2,494
    Likes Received:
    930
    Home Page:
    Thank you so much, it works :D
     
    • Thanks Thanks x 1
  13. fatboy

    fatboy Elite Member

    Joined:
    Aug 13, 2008
    Messages:
    1,618
    Likes Received:
    3,231
    Occupation:
    Retired
    Location:
    Old Peoples Home
    No problems - its not the neatest PHP code ever, but I knocked it up in a few minutes. If it works for you then cracking :)
    Just remember to buy me a beer when you get rich and famous!
     
  14. BlackSpidey

    BlackSpidey Junior Member

    Joined:
    Aug 9, 2013
    Messages:
    199
    Likes Received:
    95
    Occupation:
    Waving
    Location:
    Web
    Personaly i would suggest you to go with scrapebox it better .
     
  15. whiteblackseo

    whiteblackseo Jr. VIP Jr. VIP

    Joined:
    Apr 11, 2015
    Messages:
    2,494
    Likes Received:
    930
    Home Page:
    Will do sir! :)
     
  16. HoNeYBiRD

    HoNeYBiRD Jr. VIP Jr. VIP

    Joined:
    May 1, 2009
    Messages:
    7,307
    Likes Received:
    8,271
    Gender:
    Male
    Occupation:
    Geographer, Tourism Manager
    Location:
    Ghosted
    • Thanks Thanks x 3
  17. fatboy

    fatboy Elite Member

    Joined:
    Aug 13, 2008
    Messages:
    1,618
    Likes Received:
    3,231
    Occupation:
    Retired
    Location:
    Old Peoples Home
    Anything is better than my dodgy PHP :D

     
  18. whiteblackseo

    whiteblackseo Jr. VIP Jr. VIP

    Joined:
    Apr 11, 2015
    Messages:
    2,494
    Likes Received:
    930
    Home Page:
    Awesome :) This is what you call a real forum :D
     
  19. whiteblackseo

    whiteblackseo Jr. VIP Jr. VIP

    Joined:
    Apr 11, 2015
    Messages:
    2,494
    Likes Received:
    930
    Home Page:
    I have 1 question. How do you export the non Valid URL's. Since I can only export or copy Valid ones.
     
  20. pdewxyz

    pdewxyz Registered Member

    Joined:
    Dec 30, 2009
    Messages:
    64
    Likes Received:
    107
    Do you try Xenu?
     
Thread Status:
Not open for further replies.