Hey, so I need a crawler bot which can list all of the pages on a site, even if they don't have any links to them. For example, an user gets an email with a link to some page, and only he has the link, possibly the admin too, but there are no links to the page on other pages. Does something like this exist? Thanks in advance. Also, yes, the page I need to get to is disallowed in robots.txt and is probably not indexed.