1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to find all pages on a Website?

Discussion in 'Black Hat SEO Tools' started by huntytigger, Apr 1, 2013.

  1. huntytigger

    huntytigger Junior Member

    Joined:
    Dec 6, 2012
    Messages:
    145
    Likes Received:
    14
    Hello,
    Is there is a good software that finds all possible pages on a website if even no links to this page?
    Sometimes for example i am searching discount for a software selling on a landing page. With long google searches Luckily finding a page on the same website which offers discount.... etc..
    Anyone can suggest a software that i can buy?
    Thanks.
     
    • Thanks Thanks x 1
  2. cody41

    cody41 Power Member

    Joined:
    Jun 18, 2009
    Messages:
    682
    Likes Received:
    274
    Location:
    Texas
    Last edited: Apr 1, 2013
  3. huntytigger

    huntytigger Junior Member

    Joined:
    Dec 6, 2012
    Messages:
    145
    Likes Received:
    14
    Thanks man. I have tested it but it is also using internal links to get site map.
    For example: www.networking-insight.c0m
    finding just 5 pages linked at the bottom of the page only. You can try with online scanner there.
    But with google site: search i can find many pages which are not linked:
    www.networking-insight.c0m/ebay/h/order1.html
    I thought there may be a good software does this.
     
  4. kaloyan14

    kaloyan14 Regular Member

    Joined:
    Dec 12, 2011
    Messages:
    284
    Likes Received:
    65
    Occupation:
    webmaster, blogger
    Location:
    Eastern Europe
    Hi, my advice is to search for any free web crawler.
    You can use a service or if can, it's possible to do in java. :)
     
  5. rossegpz

    rossegpz Junior Member Premium Member

    Joined:
    Nov 14, 2012
    Messages:
    101
    Likes Received:
    31
    Home Page:
    If the site are fully indexed by Google, you could got them by scrape tools with "site:xxx.com". Hope it helpful. Good luck!
     
    • Thanks Thanks x 1
  6. huntytigger

    huntytigger Junior Member

    Joined:
    Dec 6, 2012
    Messages:
    145
    Likes Received:
    14
    Thanks for the answars.
    im already trying with site:xxxx but there are some pages more secret.
    i have already found an old program. owasp dirbuster Lol this is brute forcing to web sites :) got some results. But brute forcing will not make site owners happy.........
     
  7. AffiliateSwitchblade

    AffiliateSwitchblade Newbie

    Joined:
    Nov 26, 2014
    Messages:
    4
    Likes Received:
    0
    Occupation:
    Owner/Operator/Programmer affiliateswitchblade.com
    Location:
    Torrington CT
    Home Page:
    The Problem with all the above suggestions is it only reveals indexed (not hidden) pages. Try the free web crawler simple software - Google "free web crawler simple"
     
  8. patadeperro

    patadeperro Registered Member

    Joined:
    Jul 5, 2011
    Messages:
    62
    Likes Received:
    17
    If there is no way to access the pages no crawler will be able to identify them all
     
  9. scottymr

    scottymr Newbie

    Joined:
    Jan 18, 2016
    Messages:
    3
    Likes Received:
    0
    try netpeak spider, xenu (they both free) or screaming frog spider
     
  10. DarkyTheWolf

    DarkyTheWolf Newbie

    Joined:
    Jan 17, 2016
    Messages:
    9
    Likes Received:
    2
    This.
    Unless I'm getting the question wrong or the crawler is intelligent enough to think on its own.
     
  11. SamyanDesigns

    SamyanDesigns Registered Member

    Joined:
    Sep 22, 2015
    Messages:
    91
    Likes Received:
    12
    I use scrapebox to scrape pages of site.
     
  12. Loyal0

    Loyal0 Registered Member

    Joined:
    Oct 20, 2015
    Messages:
    74
    Likes Received:
    198
    Use google site:"" feature for the indexed pages and look at the robots.txt file to see what pages and directories the crawlers aren't allowed to see. You can then direct brute force/iteration attacks on those hidden directories.
     
  13. abbeywil

    abbeywil Newbie

    Joined:
    Jan 7, 2016
    Messages:
    34
    Likes Received:
    3
    Hi,
    Till now there is no such ways to find all the pages without any internal links, If there is any such tools please convey to me too.......
     
  14. SECONDS

    SECONDS Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 31, 2015
    Messages:
    135
    Likes Received:
    11
    Wordpress is having good sitemap plugin you can get it in addons section.
     
  15. craigc0302

    craigc0302 Registered Member Premium Member

    Joined:
    Apr 19, 2015
    Messages:
    63
    Likes Received:
    9
    Occupation:
    SEO
    Location:
    Glasgow
    Home Page:
    Screaming Frog is an agressive site crawler, have a go on that :D
     
  16. Whiningbaby

    Whiningbaby Junior Member

    Joined:
    Nov 25, 2015
    Messages:
    110
    Likes Received:
    11
    wow this thread very informative havent heard of such things ever in my life..... for sure everyday is a learning exprerience
     
  17. Sheraf

    Sheraf Registered Member

    Joined:
    Jan 19, 2014
    Messages:
    61
    Likes Received:
    8
  18. nonin

    nonin Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 26, 2010
    Messages:
    729
    Likes Received:
    497
    Gender:
    Male
    Location:
    I connect dots..
    Home Page:
    hmm.. nice idea.. that really would be useful :)
     
  19. nonin

    nonin Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 26, 2010
    Messages:
    729
    Likes Received:
    497
    Gender:
    Male
    Location:
    I connect dots..
    Home Page:
    :) why do you think that sitemaps display all the pages on the site? .. won't be the sword to win this battle
     
  20. slugshead

    slugshead Registered Member

    Joined:
    Apr 4, 2015
    Messages:
    58
    Likes Received:
    16
    Occupation:
    Tech Support
    Location:
    Wales, GB
    Is it a good idea to list your sitemap in robots.txt? Yoast hasn't done it for me so should this be manually done?