1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

What is the meaning of this Robots.txt file

Discussion in 'White Hat SEO' started by ambalaldarji, Dec 31, 2012.

Tags:
  1. ambalaldarji

    ambalaldarji Newbie

    Joined:
    Jun 8, 2011
    Messages:
    29
    Likes Received:
    4
    Home Page:
    User-agent: *
    Disallow: /*?
    Disallow: *.swf$
    Disallow: *.js$
    Disallow: /admin/
    Disallow: /xml/

    In one of the site i have seen this kind of robots.txt file. can some one tell me what does this mean. As i have less knowledge in Robots.txt so to get more knowledge about this i want to know.
     
  2. Checkmate

    Checkmate Elite Member

    Joined:
    Aug 9, 2010
    Messages:
    1,534
    Likes Received:
    637
    It means the person wants to not show those extensions or file names in any results pages.
     
    • Thanks Thanks x 1
  3. ambalaldarji

    ambalaldarji Newbie

    Joined:
    Jun 8, 2011
    Messages:
    29
    Likes Received:
    4
    Home Page:
    and what about this

    Disallow: /admin/
    Disallow: /xml/
     
  4. Checkmate

    Checkmate Elite Member

    Joined:
    Aug 9, 2010
    Messages:
    1,534
    Likes Received:
    637
    Means he doesn't want any search engines to show /admin or /xml directory in results.
     
    • Thanks Thanks x 1
  5. ambalaldarji

    ambalaldarji Newbie

    Joined:
    Jun 8, 2011
    Messages:
    29
    Likes Received:
    4
    Home Page:
    So is it good idea to put this in robots.txt file or not
     
  6. Zapdos

    Zapdos Power Member

    Joined:
    Oct 22, 2011
    Messages:
    597
    Likes Received:
    708
    Location:
    Eastern North Carolina
    If you're trying to hide those directories from people, probably not. /admin/ shouldn't be linked to from anywhere. Also, if both are supposed to be hidden they should also be password protected.
     
    • Thanks Thanks x 1
  7. AboveAll

    AboveAll Junior Member

    Joined:
    Apr 27, 2010
    Messages:
    150
    Likes Received:
    58
    Robots.txt tell search engines what to index and what not. However, you don't need to have robots.txt, we are 2012 and after 1 day - 2013. Search engines are smarter.

    Spend time making good long, unique and usable content that provides people a value. Don't deal with robots.txt or .htaccess, it is useless today, unless you are making some really complicated BH stuff.
     
  8. ambalaldarji

    ambalaldarji Newbie

    Joined:
    Jun 8, 2011
    Messages:
    29
    Likes Received:
    4
    Home Page:
    Thanks for sharing information about robots.txt but know all this i just want to know that the term which i have mentioned here for what purpose people are use it.
     
  9. guptaabhijit318

    guptaabhijit318 Newbie

    Joined:
    Dec 21, 2012
    Messages:
    17
    Likes Received:
    1
    Location:
    India
    Home Page:
    The robots.txt is a simple text file in your web site that notifies search engine bots how to crawl and index website or web pages.
     
  10. wottoinc

    wottoinc Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 5, 2012
    Messages:
    1,506
    Likes Received:
    62
    you have already got answers they control what you wish to share on the search engines and what do you wish to hide.
     
  11. ugjunk

    ugjunk Jr. VIP Jr. VIP Premium Member

    Joined:
    Jan 1, 2011
    Messages:
    2,345
    Likes Received:
    721
    Location:
    Los Angeles
    Home Page: