1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to analyze ROBOT.txt file?

Discussion in 'White Hat SEO' started by uasif1000, Apr 25, 2015.

  1. uasif1000

    uasif1000 Newbie

    Joined:
    Apr 23, 2015
    Messages:
    3
    Likes Received:
    0
    Hello mate, I have some qustions about ROBOT.txt file:

    1. What is ROBOT.txt file? how it is work?
    2. Why we use it for SEO?
    3. Why we analyze ROBOT.txt file?

    Please tell me in details. I'm want to know about this.

    Thank you in advance.
     
  2. 3kmilliondollar

    3kmilliondollar Newbie

    Joined:
    Apr 25, 2015
    Messages:
    29
    Likes Received:
    1
    Gender:
    Male
    Occupation:
    social media marketing
    can you tell how to analyze the autopilot txt?
     
  3. webinaz

    webinaz Jr. VIP Jr. VIP

    Joined:
    Mar 16, 2012
    Messages:
    762
    Likes Received:
    80
    Occupation:
    CTO, SEO Engineer
    Location:
    Heaven
    Robot.txt is a file which is used to give instructions to the search engines.
    It is useful for the SEO to make the search engines what to crawl and what not to.
     
    Last edited: Apr 26, 2015
  4. Zwielicht

    Zwielicht Moderator Staff Member Moderator Jr. VIP

    Joined:
    Aug 31, 2013
    Messages:
    6,569
    Likes Received:
    11,714
    Gender:
    Male
    Occupation:
    Private Investigator
    Location:
    Riverside, California
    Home Page:
    I hope you're not thinking about spamming threads.
    Anyway, here are the answers to your questions:
    1. http://www.robotstxt.org/robotstxt.html
    2. http://seoroi.com/seo-faq/robotstxt-what-it-is-why-its-used-and-how-to-write-it/
    3. https://support.google.com/webmasters/answer/6062608?hl=en
     
    • Thanks Thanks x 1