-
Rule 3.6 - Cold DMing users - has been clarified in regards to DMs being sent to potential buyers from WTB/HAF threads. A link to the thread IS required in the DM. Read the 3.6 Rules Clarification or read the full HAF rules.
Program to find links for future product releases on sites
- Thread starter urahozer
- Start date
- Status
thedorf
Elite Member
- Oct 1, 2008
- 2,558
- 1,819
I don't think you can "crawl a site" to find links that are not live. Crawling a site means you follow the links on the site that are live from one page to another, hence the term crawling.
The only way I can think of to do this is to find a site that allows for directory listings (no default index.html/php) and find the links there. You see, if nothing links to a page you either have to know the page is there or get it via a directory listing.
However, that being said I don't know how Google finds some of my webpages. I purposely don't link to some webpages but they still end up indexed. Maybe someone else knows a trick I don't. Good luck mate.
The only way I can think of to do this is to find a site that allows for directory listings (no default index.html/php) and find the links there. You see, if nothing links to a page you either have to know the page is there or get it via a directory listing.
However, that being said I don't know how Google finds some of my webpages. I purposely don't link to some webpages but they still end up indexed. Maybe someone else knows a trick I don't. Good luck mate.
Bad use of language on my part! Thanks for the clarification!
I know the page will be there and would be looking for the link to it, or piecing a link together. It sounds like directory listing finder would be what I want.
moon world
BANNED
- Nov 7, 2013
- 907
- 157
Make proper ROR sitemap.xml for instant indexing
Great thanks for your help!
- Status
This thread has been auto closed due to the forum's thread age policy. Read more.