Why are there different robots.txt?

Puja

Newbie
Joined
Jan 24, 2018
Messages
15
Reaction score
0
I have multilingual website running on plugins. Having subdirectories to define different languages according to the destination. There are total 4 subdirectories with the main website and all are having seperate robots.txt file with seperate codes in them. See the screenshots attached. One is for genderselection dot com / robots . txt (screenshot 1) and other is for genderselection dot com / es / robots . txt (screenshot 2)
Does it makes any difference or I should ignore this as i have read it that robots.txt file is not get crawled for subdirectoires. Please help!
I have set the robots file for main website and hoping to see the same code for all subdirectories. Is it possible?
 

Attachments

  • Screenshot(1).png
    Screenshot(1).png
    9.3 KB · Views: 104
  • Screenshot(2).png
    Screenshot(2).png
    3.1 KB · Views: 118
so you can get control on which the bot can enter/crawl and not
 
So, you can control and instruct search bot how to crawl and access the content of individual.
 
i recommend you to delete the first picture, because we can search your site up and i have heard that people can make you website banned by google. (srry for bad english)
 
Back
Top