Tool that finds all 404 pages on a domain.

fedez said:

I need a tool that I can put in a domain (or better yet a list of domains) and it goes through all of them and finds all the 404 pages on each domain. So for example. I put in xyz.com

and it finds 2 404 pages on the website like

xyz.com/page1404

xyz.com/page2404

So far my only idea has been to find the sitemaps on domains and extract urls of whole site then check those for 404 or not, but this isnt good as not all sitemaps are easy to locate and many of them update regularly and dont include 404 pages.

I basically want to find all 404 pages on a domain that other domains are linking to.

Any suggestions would help.

Click to expand...

NG-neer said:

I'm sorry but I'm still confused. That's now how 404 pages work. The site has existing pages and every other page you request is non existent. Meaning there is an unlimited number of 404 pages. Don't you want the pages that DO EXIST?

Click to expand...

Inv4lid said:

So basically.
He inputs a site
- Crawls the site for all available links
- The tool checks the available links if it's 404ed.
- Extracts the links that were 404ed

I think that's what he wanna achieve.

Click to expand...

NG-neer said:

Okay, talk is cheap so I wrote some code to demonstrate what I have in mind. I don't know if you can read Python code but I hope you can at least understand what it does by reading my explanation I commented (everything after the hashtag sign)
import requests
from bs4 import BeautifulSoup

soup = BeautifulSoup(requests.get(url_you_want).text) # Get the raw HTML source code of the page you want
urls = [a.get('href') for a in soup.find_all('a', href=True)] # Select all the hrefs of all the a elements in the HTML source code
broken_urls = [] # The list the 404ed pages will get added to
for url in urls: # Loop through the list of urls
if requests.get(url).status_code == 404: # If the HTTP response code is 404...
broken_urls.append(url) # Add the current url to the list of broken urls

Again, I'm not sure you know how to run this code so if you need some help, please let me know!

Click to expand...

NG-neer said:

I'm sorry if it's a bit hard to read. There should've been spaces and tabs but unfortunately they don't seem to show up

Click to expand...

NG-neer said:

Okay, talk is cheap so I wrote some code to demonstrate what I have in mind. I don't know if you can read Python code but I hope you can at least understand what it does by reading my explanation I commented (everything after the hashtag sign)
import requests
from bs4 import BeautifulSoup

soup = BeautifulSoup(requests.get(url_you_want).text) # Get the raw HTML source code of the page you want
urls = [a.get('href') for a in soup.find_all('a', href=True)] # Select all the hrefs of all the a elements in the HTML source code
broken_urls = [] # The list the 404ed pages will get added to
for url in urls: # Loop through the list of urls
if requests.get(url).status_code == 404: # If the HTTP response code is 404...
broken_urls.append(url) # Add the current url to the list of broken urls

Again, I'm not sure you know how to run this code so if you need some help, please let me know!

Click to expand...