John Mu's Robots.txt File

[prebillionaire]

Hello all,

Have you taken a look into www.johnmu.com/robots.txt file?

Can anybody explain the point of these? I've never heard of them and they involve weird words and keywords like r@pe, freemason, etc...

 

[alonelywriter]

My brother.

You find the words that you mentioned because there is no space - One word included - scrape

I think others are similar.

 

[prebillionaire]

My brother.

You find the words that you mentioned because there is no space - One word included - scrape

I think others are similar.

I know scrape and grape but there's tons that just have the keyword in it...look some more.

 

[NeedMyLink]

The specific terms you mentioned could be used for a variety of reasons - perhaps to block content or pages you don't want to link to the site, or to prevent indexing of pages that might attract the attention of bots. Perhaps it's part of a larger strategy or just an unconventional method of addressing specific problems. Without information from the site administrator, it's hard to say the exact rationale.