How to find all pages on a Website?

H

huntytigger

Junior Member
Joined
Dec 6, 2012
Messages
164
Reaction score
17
Hello,
Is there is a good software that finds all possible pages on a website if even no links to this page?
Sometimes for example i am searching discount for a software selling on a landing page. With long google searches Luckily finding a page on the same website which offers discount.... etc..
Anyone can suggest a software that i can buy?
Thanks.
 
Advertise on BHW
Thanks man. I have tested it but it is also using internal links to get site map.
For example: www.networking-insight.c0m
finding just 5 pages linked at the bottom of the page only. You can try with online scanner there.
But with google site: search i can find many pages which are not linked:
www.networking-insight.c0m/ebay/h/order1.html
I thought there may be a good software does this.
 
Hi, my advice is to search for any free web crawler.
You can use a service or if can, it's possible to do in java. :)
 
If the site are fully indexed by Google, you could got them by scrape tools with "site:xxx.com". Hope it helpful. Good luck!
 
Thanks for the answars.
im already trying with site:xxxx but there are some pages more secret.
i have already found an old program. owasp dirbuster Lol this is brute forcing to web sites :) got some results. But brute forcing will not make site owners happy.........
 
The Problem with all the above suggestions is it only reveals indexed (not hidden) pages. Try the free web crawler simple software - Google "free web crawler simple"
 
If there is no way to access the pages no crawler will be able to identify them all
 
huntytigger said:
Hello,
Is there is a good software that finds all possible pages on a website if even no links to this page?
Sometimes for example i am searching discount for a software selling on a landing page. With long google searches Luckily finding a page on the same website which offers discount.... etc..
Anyone can suggest a software that i can buy?
Thanks.
Click to expand...

try netpeak spider, xenu (they both free) or screaming frog spider
 
Use google site:"" feature for the indexed pages and look at the robots.txt file to see what pages and directories the crawlers aren't allowed to see. You can then direct brute force/iteration attacks on those hidden directories.
 
Hi,
Till now there is no such ways to find all the pages without any internal links, If there is any such tools please convey to me too.......
 
Wordpress is having good sitemap plugin you can get it in addons section.
 
wow this thread very informative havent heard of such things ever in my life..... for sure everyday is a learning exprerience
 
huntytigger said:
Hello,
Is there is a good software that finds all possible pages on a website if even no links to this page?
Sometimes for example i am searching discount for a software selling on a landing page. With long google searches Luckily finding a page on the same website which offers discount.... etc..
Anyone can suggest a software that i can buy?
Thanks.
Click to expand...

hmm.. nice idea.. that really would be useful :)
 
Is it a good idea to list your sitemap in robots.txt? Yoast hasn't done it for me so should this be manually done?
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top