1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Find unindexed content on a website

Discussion in 'Black Hat SEO Tools' started by dirtysecret, Jan 25, 2010.

  1. dirtysecret

    dirtysecret Regular Member

    Joined:
    Dec 12, 2008
    Messages:
    298
    Likes Received:
    20
    A common method for getting freebies from a website selling info products/software is to do a site:domainname.com command from Google. Sometimes this works, sometimes it doesn't.

    If the owner of the product is at least somewhat smart, they will tell spiders not to index the pages with the paid content.

    I am wondering if there is a way to query a webserver and say "give me all the content on websitenamehere.com"?

    Anyone know how to do this? I know how to use telnet to connect to a webserver, but I have to know exactly which content I'm trying to GET (i.e. the name of the webpage). That is obviously insufficient for finding content that you do not yet know if exists or what it's name is.
     
  2. shadyplaces

    shadyplaces Junior Member

    Joined:
    May 12, 2009
    Messages:
    153
    Likes Received:
    47
    I was about to give you a good hacking tech clue, then I realized that you were probably trying to steal creatives. Make your own dude. If you're trying to get other stuff, tell us what it is. And Google querying isn't going to work for the good stuff. Try XSS. Have fun, and make your own shit if it's creatives etc.
     
  3. dirtysecret

    dirtysecret Regular Member

    Joined:
    Dec 12, 2008
    Messages:
    298
    Likes Received:
    20
    Here's how I can assure you that's not what I was trying to do - I don't have any idea WTF ur talking about. What do you mean by "creatives"?

    With that out of the way, the reason I ask is so that I could try to download internet marketing info products myself, without relying on torrents or what's available in the downloads sections (GET and REQ) here at BHW. I was trying to learn to how to be more self sufficient and less of leech.

    I hope that will suffice and that you'll help me now.
     
  4. SEO20

    SEO20 Elite Member

    Joined:
    Mar 25, 2009
    Messages:
    2,017
    Likes Received:
    2,259
    Alexa searchengine used to ignore robot.txt files. Bet it still does ;-)
     
  5. mogambo

    mogambo Registered Member

    Joined:
    May 31, 2009
    Messages:
    96
    Likes Received:
    9
    thats something i'm trying to knw for a long time...can anybody be kind enuf to say this?
     
  6. dirtysecret

    dirtysecret Regular Member

    Joined:
    Dec 12, 2008
    Messages:
    298
    Likes Received:
    20
    This is going to sound incredibly stupid, but I'll ask it anyway...how do I do an indexing query on alexa? When I type "site:domainnamehere.com" it thinks I'm searching for that query and obviously returns no results. I don't know how to tell alexa "give me every page you have indexed from this domain."
     
  7. shadyplaces

    shadyplaces Junior Member

    Joined:
    May 12, 2009
    Messages:
    153
    Likes Received:
    47
    Google ignores robots.txt as well as any other measures you may take.

    Try messing with stuff; what type of files are you looking for (extensions)?

    Are they password protected? If so what script / software does the website use to protect? What do error messages gives you?

    Find the vunerabilties within the website via Acunetix. Figure out how to exploit those vunerabilities; don't do anything malicious.

    This might help, not sure, but I am just trying to provide information to encourage penetration testing and the security of all sites ;)

    http://www.ilovebonnie.net/astalavistahack.txt
     
  8. dirtysecret

    dirtysecret Regular Member

    Joined:
    Dec 12, 2008
    Messages:
    298
    Likes Received:
    20
    shady, appreciate the thorough answer but I'm not really looking to do vulnerability testing or anything that complicated.

    Here's what I'm trying to accomplish:

    There is a website with a salespage offering videos for $67. I just wanted to scan his domain to see if the videos I would be getting are also hosted on the domain.
     
  9. risefromdeath

    risefromdeath Power Member

    Joined:
    Jul 1, 2009
    Messages:
    650
    Likes Received:
    107
    the software above has a inbuilt crawler as well i guess to find all hidden pages in the site :)
     
  10. redfan123

    redfan123 Regular Member

  11. cerebus

    cerebus Newbie

    Joined:
    Sep 28, 2009
    Messages:
    22
    Likes Received:
    3
    Shady, was this really published by anti-sec?
    How did you happen to stumble upon this? i tried to search for material related to their work a while back. However, i couldn't find much.