Hackers and Experts called: How to extract all sub-domains - Even possible???

[euros123dotcom]

Fellow Blackhatters and especially guys with a "Hacker-Gene",

Here a question from a coding noob. I have Googled this, but haven't been able to find this information (just general, not specific on how to do it).

There are many firms that have distributors/shops under their main domain. You buy one of these cookie cutter sites and they will host them under their main domain. Like this:

uk.MOTHERCOMPANY.com/shop/xxxxx

People that bought the online shop all choose their own fantasy names (xxxxx). I need to know ALL different URLs to be able to visit every single xxxxx shop and see how the owner is doing traffic wise. This is more of a coding question, or maybe there is a bot or software that can do this and I am not aware.

Is this even possible? Hope you understand this. Please point me in the right direction. This is a profitable idea and the Blackhatter that is able to help me will be rewarded with the idea why I am trying to do this (there is enough dough for more than one person This is a market that is desperate for traffic.

Thanks in advance for your time and thoughts.

 

[Mr Stark]

Either google search ( site:http://uk.MOTHERCOMPANY.com/shop/* ), or brute force...

 

[euros123dotcom]

Nope. Tried that before. It resolves to a standard page that says that this shop does not exist. Back to square one. How would brute force work? Basically I have to grab all the sub-domains off from the mother domain. Unless you know a specific URL it seems impossible to see who has bought a shop with these guys.

 

[HelloInsomnia]

Just a friendly reminder that hacking related talk is not allowed here.

Not that you need to look at hacking to solve this problem...

Mr Stark is right that you can use site:http://uk.MOTHERCOMPANY.com/shop/ to get more URLs and shops. Also look for a sitemap.xml just Google it along with the subdomain of the site and see if you can find it or tack it on to the end like http://uk.MOTHERCOMPANY.com/sitemap.xml

 

[Wilson Grant Fisk]

Try this:
1) Go to Wolfram Alpha
2) Enter the domain without any subdomain (ie. wordpress.com)
3) In the 3rd result box, top right, push "subdomains."
4) When it opens, push "more" (top right of the box)
Source: Various sites.http://shirishpadalkar.blogspot.co.uk/2013/05/find-subdomains-of-domain-name.html

http://security.stackexchange.com/questions/35078/how-can-i-find-subdomains-of-a-site
http://stackoverflow.com/questions/131989/how-do-i-get-a-list-of-all-subdomains-of-a-domain
http://andywolf.com/list-all-dns-records-from-a-public-domain-with-nslookup/

 

[aoe316]

couldn't a web-crawler do what he is asking?

 

[Wilson Grant Fisk]

couldn't a web-crawler do what he is asking?

I am pretty certain that I have used a tool that crawled a site, found all sub domains and directories.
Can't remember which tool though.

 

[innozemec]

most of those shops should be pretty much indexed by G, so why don't you just use Scrapebox to harvest the list for you?

 

[JustUs]

I think your question is answered.

 

[discount81]

Just a technical correction but you are looking for sub folders, not sub domains.

sub domain is the uk.mothercompany.com

 

[euros123dotcom]

Thank you all for your replies. I am sorry for the "Hacker" thing. I wasn't thinking illegal hacking, more like whitehat, smart-approach hacking (the term has been used and abused to death - nowadays my mother says hacking if the toaster is not working).

I am checking all the presented possibilities and discount81 is correct. What I really need are the subfolders. They are not indexed or not even in the companies sitemap, I can't find them through regular web search expressions. I think the database creates them dynamically "on the go" since all company sites affiliate shops are exactly the same, just the sellers or shop owners name is different - me thinks a general "shop" webpage could be sitting there and the database populates the empty fields with whatever name the shop owner gave at inscription.

This is a very dumb way to setup shop as your site is not indexed (Google would need to index 1000's of the same crap) and you will not get organic traffic. But still, there are people out there that shell money out for a generic shop like this. They are all hungry for traffic because their traffic is as dry as a old ladies.....well....as the desert.