Tool that finds all 404 pages on a domain.

fedez · Oct 20, 2018

I need a tool that I can put in a domain (or better yet a list of domains) and it goes through all of them and finds all the 404 pages on each domain. So for example. I put in xyz.com

and it finds 2 404 pages on the website like

xyz.com/page1404

xyz.com/page2404

So far my only idea has been to find the sitemaps on domains and extract urls of whole site then check those for 404 or not, but this isnt good as not all sitemaps are easy to locate and many of them update regularly and dont include 404 pages.

I basically want to find all 404 pages on a domain that other domains are linking to.

Any suggestions would help.

appProgrammer · Oct 20, 2018

All you really need is a few lines of code. A python loop with the beautiful soup module added and boom you are done. I suggest you post in the freelancer section of this site. There are a lot of good programmers that would love to make a small tool just for you.

fedez · Oct 20, 2018

hmmm, Ive posted in freelancer section then, lets see.

EmzoLorenzo · Oct 21, 2018

did you try screamingfrog?

Aty · Oct 21, 2018

Have a look at Xenu.

fedez · Oct 28, 2018

EmzoLorenzo said:
did you try screamingfrog?

How would screaming frog do that for a list of 100 domains at a time?

fedez · Oct 28, 2018

Aty said:
Have a look at Xenu.

Xenu just looks at random links branching off any other link. How would xenu find a list of all 404s on a list of domains of 100 domains, and only those domains' 404s?

MKSKS · Oct 28, 2018

https://www.deadlinkchecker.com
http://www.urlitor.com

I mostly use the second one.

patadeperro · Oct 28, 2018

Aty said:
Have a look at Xenu.

This is the only possible answer.

fedez · Oct 28, 2018

patadeperro said:
This is the only possible answer.

Xenu just branches off randomly from a list of urls. I want a list of 404s form a list of domains, and the 404s being of those domains.

patadeperro · Oct 28, 2018

fedez said:
Xenu just branches off randomly from a list of urls. I want a list of 404s form a list of domains, and the 404s being of those domains.

That is not true, it can crawl a specific domain or list of domains BUT it seems that you need GSA search engine ranking

fedez · Oct 28, 2018

patadeperro said:
That is not true, it can crawl a specific domain or list of domains BUT it seems that you need GSA search engine ranking

Can it really? All I see is the add list of urls option or the one url option. In either case when I run it, all thats limiting it is the depth of its search, otherwise it picks up all the links for every url and branches off them in turn.

Aty · Oct 28, 2018

fedez said:
Xenu just looks at random links branching off any other link. How would xenu find a list of all 404s on a list of domains of 100 domains, and only those domains' 404s?

Use Scrapebox to make a list of links, then Xenu to check the links.

fedez · Oct 28, 2018

Aty said:
Use Scrapebox to make a list of links, then Xenu to check the links.

Now this might actually work. I just got scrape box, what would I use to get the list of urls for each domain on scrapebox?

bartosimpsonio · Oct 28, 2018

fedez said:
Xenu just looks at random links branching off any other link. How would xenu find a list of all 404s on a list of domains of 100 domains, and only those domains' 404s?

As others have said you'll likely need a custom scraper for that.

Otherwise you can simply scrape 1000's of urls, dump the result header line somewhere and then filter the 404's using some text editor.

Aty · Oct 28, 2018

fedez said:
Now this might actually work. I just got scrape box, what would I use to get the list of urls for each domain on scrapebox?

Link Extractor addon

fedez · Oct 28, 2018

Aty said:
Link Extractor addon

Sweet, will try that out.

NG-neer · Oct 28, 2018

I don't understand what you wanna achieve. Do you already know what pages you want to visit. If not you need a crawler to find them. I could create a script for you and post it there, depending on what you want.

fedez · Oct 28, 2018

NG-neer said:
I don't understand what you wanna achieve. Do you already know what pages you want to visit. If not you need a crawler to find them. I could create a script for you and post it there, depending on what you want.

So for example, say I have a domain xyz.com now that domain will have a bunch of 404 pages. I want to know all of (or as many as possible) of the 404s for that domain, and the same for an entire list of domains.

NG-neer · Oct 28, 2018

fedez said:
So for example, say I have a domain xyz.com now that domain will have a bunch of 404 pages. I want to know all of (or as many as possible) of the 404s for that domain, and the same for an entire list of domains.

I'm sorry but I'm still confused. That's now how 404 pages work. The site has existing pages and every other page you request is non existent. Meaning there is an unlimited number of 404 pages. Don't you want the pages that DO EXIST?

Tool that finds all 404 pages on a domain.

Junior Member

Junior Member

Junior Member

BANNED

Elite Member

Junior Member

Junior Member

Banned for failing to resolve dispute.

Elite Member

Junior Member

Elite Member

Junior Member

Elite Member

Junior Member

Elite Member

Elite Member

Junior Member

Newbie

Junior Member

Newbie

Main Menu

Marketplace

Making Money

BlackHat World