How are 404 broken links detected with tools?

Hi guys,

Just wanted ask if anybody knows how a 404 broken links get detected when running tools. What I'm trying to find out is how the tools find these links. I know they crawl them but I need to have a better answer than this. For example I've used seoProfiler and Screaming Frog to do a quick seo audit on the site and came across a broken link. the url is given but I'm not sure how it's found it as it's not a link on the navigation bar. I don't know if I've explained it well but to give you an example its found a broken link like:

www.example.example1.com/en-GBR

which shows as broken link, but this is could be due to that fact it should be

www.example.example1.com/en-GBR/

There is a canonical tag on the header which is <link rel="canonical" href="/en-GBR/">

I just need to know how it's done this so that I can explain it in a report, rather than just say its a broken link but have have no answer to how the crawler has found this broken link.

​Thanks in advance

http://en.wikipedia.org/wiki/List_of_HTTP_status_codes

when you ping a http server it responses back . So when your crawler loads a website it gets a response from server in this format . http://en.wikipedia.org/wiki/List_of_HTTP_header_fields#Response_Headers .

you can then read the values. python and php examples
https://docs.python.org/2/library/httplib.html#httplib.HTTPResponse.status
http://php.net/manual/en/function.http-response-code.php

divok said:

http://en.wikipedia.org/wiki/List_of_HTTP_status_codes

when you ping a http server it responses back . So when your crawler loads a website it gets a response from server in this format . http://en.wikipedia.org/wiki/List_of_HTTP_header_fields#Response_Headers .

you can then read the values. python and php examples
https://docs.python.org/2/library/httplib.html#httplib.HTTPResponse.status
http://php.net/manual/en/function.http-response-code.php

Click to expand...

Hey thanks for your help.

So does that mean the broken link is from the server in which the GET method returned a 404 broken link?

Also would it be safe to say that with this broken link its better to 301 redirect this url to resolve the broken link?

besupreme said:

Hey thanks for your help.

So does that mean the broken link is from the server in which the GET method returned a 404 broken link?

Also would it be safe to say that with this broken link its better to 301 redirect this url to resolve the broken link?

Click to expand...

Down load Xenu link sleuth.

It will tell you the origin and destination of the link.

besupreme said:

Also would it be safe to say that with this broken link its better to 301 redirect this url to resolve the broken link?

Click to expand...

Yes .

Techxan said:

Down load Xeny link sleuth.

It will tell you the origin and destination of the link.

Click to expand...

Thanks for the recommendation but I've already used "Xenu" . I just need to know how to the tool managed to find the broken link, so that I can write it up in a report. ( I can't just list that my tool found a broken link but not explain the process on how it managed to find it).

Thanks for you help again I do appreciate it.