1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

What type of Robots.txt file allow all bot to visit my site, my blog and make index it?

Discussion in 'White Hat SEO' started by Dwivid, Oct 19, 2012.

  1. Dwivid

    Dwivid Newbie

    Joined:
    May 18, 2012
    Messages:
    40
    Likes Received:
    1
    Occupation:
    Internetmarketing and SEO
    Location:
    Indore, India
    Home Page:
    Hello everyone...
    I want to know that what extension helps to allow all bot to index our sites and crawl them.
    thanks
     
  2. seoways

    seoways Jr. VIP Jr. VIP Premium Member UnGagged Attendee

    Joined:
    Dec 19, 2009
    Messages:
    4,651
    Likes Received:
    724
    Location:
    Behind you!
    Are you looking for robots.txt code?

    If so,


    User-agent: *Allow: /
    Will allow all the bots to crawl all the webpages within your domain.


    User-agent: *Disallow: / Will block all the bots to all the URL of your domain
     
  3. Endire

    Endire Elite Member Premium Member

    Joined:
    Mar 27, 2012
    Messages:
    1,756
    Likes Received:
    1,061
    Gender:
    Male
    Dwivid,

    It is important to note that if you do not have pages that are off limits to search engines, you do not need a robots.txt file. This file is optional and should only be used if you have pages that you don't want indexed. Bots will crawl your site if there is no file in place.

    Here is some info on constructing a robots.txt file,

    http://www.robotstxt.org/robotstxt.html

    Hope that helps,

    Shawn