1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Program to find links for future product releases on sites

Discussion in 'Hire a Freelancer' started by urahozer, Feb 21, 2016.

  1. urahozer

    urahozer Newbie

    Joined:
    Feb 21, 2016
    Messages:
    10
    Likes Received:
    0
    Hi,

    I am looking for someone/thing that is able to crawl a site to find links that are not live in the system yet and in no way accessible from the webpage. this is to find early links for products on e-commerce sites
     
  2. thedorf

    thedorf Supreme Member

    Joined:
    Oct 1, 2008
    Messages:
    1,223
    Likes Received:
    721
    Occupation:
    what? I gotta have a job?
    Location:
    BHW - Where else?
    I don't think you can "crawl a site" to find links that are not live. Crawling a site means you follow the links on the site that are live from one page to another, hence the term crawling.

    The only way I can think of to do this is to find a site that allows for directory listings (no default index.html/php) and find the links there. You see, if nothing links to a page you either have to know the page is there or get it via a directory listing.

    However, that being said I don't know how Google finds some of my webpages. I purposely don't link to some webpages but they still end up indexed. Maybe someone else knows a trick I don't. Good luck mate.
     
  3. urahozer

    urahozer Newbie

    Joined:
    Feb 21, 2016
    Messages:
    10
    Likes Received:
    0
    Bad use of language on my part! Thanks for the clarification!

    I know the page will be there and would be looking for the link to it, or piecing a link together. It sounds like directory listing finder would be what I want.
     
  4. thedorf

    thedorf Supreme Member

    Joined:
    Oct 1, 2008
    Messages:
    1,223
    Likes Received:
    721
    Occupation:
    what? I gotta have a job?
    Location:
    BHW - Where else?
    The problem you'll have is most websites have a default index.html/php in each directory and that will prevent the directory listing. Also some websites disallow directory listings outright.
     
  5. moon world

    moon world BANNED BANNED Jr. VIP

    Joined:
    Nov 7, 2013
    Messages:
    909
    Likes Received:
    156
    Gender:
    Male
    Make proper ROR sitemap.xml for instant indexing
     
  6. Galleta

    Galleta Regular Member

    Joined:
    Dec 7, 2015
    Messages:
    270
    Likes Received:
    88
    There are some tools that can crawl for all possibly existing sites / html/php files on a webserver, in general.

    Sent you a PM.
     
  7. urahozer

    urahozer Newbie

    Joined:
    Feb 21, 2016
    Messages:
    10
    Likes Received:
    0
    Great thanks for your help!