Questions on Robots.txt File

Vik Pathania

Regular Member

Oct 13, 2014

275

57

• Feb 2, 2017

• #1

Here is a question for you guys:

If I add following syntax:

User-agent: *
Disallow: /

This means it will disallow all the bots to crawl any of my page. Right?

But, would it also stop crawler from crawling my root page? Or crawler can still crawl my root page?


Looking forward for great views.

 

Emiya

Registered Member

Jun 8, 2013

99

10

• Feb 2, 2017

• #2

disallow: / will effect to your whole website inclusive your root page.

 

Vik Pathania

Regular Member

Oct 13, 2014

275

57

• Feb 2, 2017
• Thread Starter Thread Starter

• #3

Emiya said:

disallow: / will effect to your whole website inclusive your root page.

Click to expand...

Are you sure? I am bit confused on this

 

Emiya

Registered Member

Jun 8, 2013

99

10

• Feb 2, 2017

• #4

Yes, because "Disallow: (without slash)" means that robots have access to your entire site.

You can read here: http://www.robotstxt.org/robotstxt.html

 

gary2

Elite Member

Jr. VIP

Jan 20, 2013

3,354

550

• Feb 2, 2017

• #5

Yeah, It's right. Because "Disallow: /" means that will stop the crawler to crawling your entire site. By adding this it will surely affect your website traffic.