Find unindexed content on a website

D

dirtysecret

Regular Member
Joined
Dec 12, 2008
Messages
298
Reaction score
21
A common method for getting freebies from a website selling info products/software is to do a site:domainname.com command from Google. Sometimes this works, sometimes it doesn't.

If the owner of the product is at least somewhat smart, they will tell spiders not to index the pages with the paid content.

I am wondering if there is a way to query a webserver and say "give me all the content on websitenamehere.com"?

Anyone know how to do this? I know how to use telnet to connect to a webserver, but I have to know exactly which content I'm trying to GET (i.e. the name of the webpage). That is obviously insufficient for finding content that you do not yet know if exists or what it's name is.
 
Advertise on BHW
I was about to give you a good hacking tech clue, then I realized that you were probably trying to steal creatives. Make your own dude. If you're trying to get other stuff, tell us what it is. And Google querying isn't going to work for the good stuff. Try XSS. Have fun, and make your own shit if it's creatives etc.
 
shadyplaces said:
I was about to give you a good hacking tech clue, then I realized that you were probably trying to steal creatives.
Click to expand...

Here's how I can assure you that's not what I was trying to do - I don't have any idea WTF ur talking about. What do you mean by "creatives"?

With that out of the way, the reason I ask is so that I could try to download internet marketing info products myself, without relying on torrents or what's available in the downloads sections (GET and REQ) here at BHW. I was trying to learn to how to be more self sufficient and less of leech.

I hope that will suffice and that you'll help me now.
 
Alexa searchengine used to ignore robot.txt files. Bet it still does ;-)
 
thats something i'm trying to knw for a long time...can anybody be kind enuf to say this?
 
seo20 said:
Alexa searchengine used to ignore robot.txt files. Bet it still does ;-)
Click to expand...
This is going to sound incredibly stupid, but I'll ask it anyway...how do I do an indexing query on alexa? When I type "site:domainnamehere.com" it thinks I'm searching for that query and obviously returns no results. I don't know how to tell alexa "give me every page you have indexed from this domain."
 
Google ignores robots.txt as well as any other measures you may take.

Try messing with stuff; what type of files are you looking for (extensions)?

Are they password protected? If so what script / software does the website use to protect? What do error messages gives you?

Find the vunerabilties within the website via Acunetix. Figure out how to exploit those vunerabilities; don't do anything malicious.

This might help, not sure, but I am just trying to provide information to encourage penetration testing and the security of all sites ;)

http://www.ilovebonnie.net/astalavistahack.txt
 
shady, appreciate the thorough answer but I'm not really looking to do vulnerability testing or anything that complicated.

Here's what I'm trying to accomplish:

There is a website with a salespage offering videos for $67. I just wanted to scan his domain to see if the videos I would be getting are also hosted on the domain.
 
dirtyc said:
shady, appreciate the thorough answer but I'm not really looking to do vulnerability testing or anything that complicated.

Here's what I'm trying to accomplish:

There is a website with a salespage offering videos for $67. I just wanted to scan his domain to see if the videos I would be getting are also hosted on the domain.
Click to expand...
the software above has a inbuilt crawler as well i guess to find all hidden pages in the site :)
 
Shady, was this really published by anti-sec?
How did you happen to stumble upon this? i tried to search for material related to their work a while back. However, i couldn't find much.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top