1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Is it necessary to use Robots.txt file?

Discussion in 'White Hat SEO' started by Headlines News, Feb 9, 2018.

  1. Headlines News

    Headlines News Newbie

    Joined:
    Jan 11, 2017
    Messages:
    19
    Likes Received:
    0
    Gender:
    Male
    Occupation:
    Technical Support
    Location:
    United States
    Home Page:
    Hello friends- Is it necessary to use Robots.txt file? It can be affect on website ranking in search engine.
     
  2. WPninja

    WPninja Junior Member

    Joined:
    Apr 25, 2017
    Messages:
    167
    Likes Received:
    31
    Gender:
    Male
    Occupation:
    Creative WP Front-end Expert
    Location:
    Coding-Jungle
    Robots.txt is the door of a website. Now think what type of house you wanna build? With door or without? :p:p:cool::cool:
     
  3. MuayThai

    MuayThai Jr. VIP Jr. VIP

    Joined:
    Aug 25, 2015
    Messages:
    721
    Likes Received:
    222
    It's not necessary. It's up to you.
     
  4. markmueller

    markmueller Newbie

    Joined:
    Feb 5, 2018
    Messages:
    42
    Likes Received:
    16
    Gender:
    Male
    Location:
    Switzerland
    Home Page:
    It is important, because there you can define how search robots like google can "work" on your website. For example you can define there, that google only will crawl some specific pages of your website. Or no page, or only pictures, or whatever.

    So you have some power there over search engines :)
     
  5. MuayThai

    MuayThai Jr. VIP Jr. VIP

    Joined:
    Aug 25, 2015
    Messages:
    721
    Likes Received:
    222
    It's absolutely NOT a door.
     
    • Thanks Thanks x 1
  6. WPninja

    WPninja Junior Member

    Joined:
    Apr 25, 2017
    Messages:
    167
    Likes Received:
    31
    Gender:
    Male
    Occupation:
    Creative WP Front-end Expert
    Location:
    Coding-Jungle
    A house can be access without a door mate, but when there's a door, obviously this is the first place that bot come at first :cool:
     
  7. We Bring Rank

    We Bring Rank Jr. VIP Jr. VIP

    Joined:
    Aug 21, 2016
    Messages:
    787
    Likes Received:
    88
    Gender:
    Male
    Occupation:
    Digital Marketing Analyst
    Home Page:
    Not certainly, That's for your own preference, Everything have certain limit,
    If you use too much, Definitely it's affect your search engine ranking.
     
  8. We Bring Rank

    We Bring Rank Jr. VIP Jr. VIP

    Joined:
    Aug 21, 2016
    Messages:
    787
    Likes Received:
    88
    Gender:
    Male
    Occupation:
    Digital Marketing Analyst
    Home Page:
    Without using "Robots.txt" can't able to create the website, This your theory right?,
    Definitely wrong, its help to hide your sensitive data in google eyes. But it's not necessary.

     
    • Thanks Thanks x 1
  9. WPninja

    WPninja Junior Member

    Joined:
    Apr 25, 2017
    Messages:
    167
    Likes Received:
    31
    Gender:
    Male
    Occupation:
    Creative WP Front-end Expert
    Location:
    Coding-Jungle
    You can't get my point then. I said you can build a house without a door, it already means that you can build a site without robots file, but when there's a robot file, it's the first place where google or even any other bot landed.
     
  10. BMW™

    BMW™ Newbie

    Joined:
    Feb 6, 2018
    Messages:
    20
    Likes Received:
    2
    Gender:
    Male
    Occupation:
    Student
    Location:
    Canada
    just say in simple word guys o_O
    It's up to you, you wana list/show your website on google search engine or not? o_O If not then they will not list your site on there search data and it's not necessary unless you have a secret website :p
     
  11. BMW™

    BMW™ Newbie

    Joined:
    Feb 6, 2018
    Messages:
    20
    Likes Received:
    2
    Gender:
    Male
    Occupation:
    Student
    Location:
    Canada
    And also it helps to crawl new data from your website and an important part for website SEO :)
     
  12. WPninja

    WPninja Junior Member

    Joined:
    Apr 25, 2017
    Messages:
    167
    Likes Received:
    31
    Gender:
    Male
    Occupation:
    Creative WP Front-end Expert
    Location:
    Coding-Jungle
    Mate, I made a dramatic intro at first, but both of those people maybe don't have enough entertainment on their life and couldn't get my point of view at all and start arguing with me :-D \

    Nowhere I said that robots file is a mandatory one :D:D
     
  13. RankBoss

    RankBoss Jr. VIP Jr. VIP

    Joined:
    Nov 21, 2016
    Messages:
    679
    Likes Received:
    179
    Gender:
    Male
    Occupation:
    Online business service consultant
    Location:
    Within your range
    it's not necessary but adding it just take 10 minutes.
    Robot file can be used if you need to block bots from a certain content
    just add one.
     
  14. BMW™

    BMW™ Newbie

    Joined:
    Feb 6, 2018
    Messages:
    20
    Likes Received:
    2
    Gender:
    Male
    Occupation:
    Student
    Location:
    Canada
    Yes you are right well you said indirectly :D
    let me clear for them if there's door means they can see your contents on your website through that and if you don't wana show to the public like in previous case there was a door to see but in this case you got a invisible magic door that only certain people allowed to view :p
     
  15. digimarket

    digimarket Regular Member

    Joined:
    Sep 20, 2016
    Messages:
    262
    Likes Received:
    28
    Gender:
    Male
    Occupation:
    Digital Marketer
    Yes, because Robosts.txt generator tells the search engines which pages to access and index on your website and which pages not to.
     
  16. Leon jamesh

    Leon jamesh Jr. VIP Jr. VIP

    Joined:
    Jun 10, 2016
    Messages:
    387
    Likes Received:
    164
    Gender:
    Male
    Not so important, only need of robots.txt file is to disallow search engine to not index some pages.
     
    • Thanks Thanks x 1