1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

robot.txt needed?

Discussion in 'Black Hat SEO' started by reisuke, Oct 3, 2010.

  1. reisuke

    reisuke Registered Member

    Joined:
    Sep 25, 2010
    Messages:
    53
    Likes Received:
    0
    Hi SEO gurus,

    May i know if there is a need to include a robot.txt?
    From what i read , it's like disable other websites for google to crawl in...

    If need , may can anyone teach me how to write a basic robot.txt in my website?

    Thanks,
    SEO newbie
     
  2. edgematch

    edgematch Elite Member

    Joined:
    May 24, 2010
    Messages:
    2,539
    Likes Received:
    1,949
    Occupation:
    You can never guess!
    Location:
    :noitacoL
    Last edited: Oct 3, 2010
  3. Rwh909r

    Rwh909r Newbie

    Joined:
    Jul 13, 2009
    Messages:
    46
    Likes Received:
    37
    Occupation:
    SEO / SEM / Social Media Marketing
    Location:
    Inland Empire
    yes you need a robots.txt
    bots will crawl your site by default but you can use the robots to disallow bots crawling certain parts of your site like cgi/bin files or login pages that you dont want public or photo galleries that can cause infinite loops with bots.
    You also use the robots.txt to list the sitemap.xml location on your site.

    So yes its very much needed and very useful.
     
  4. jlzandrad

    jlzandrad Junior Member

    Joined:
    Apr 22, 2009
    Messages:
    148
    Likes Received:
    18
    Occupation:
    Web Design / Network Admin
    Location:
    In front my computer
    Home Page:
    I did'nt know about this... Creating mine now
    Thanks

     
  5. Wings[SENSE]

    Wings[SENSE] Regular Member

    Joined:
    Jan 10, 2010
    Messages:
    373
    Likes Received:
    177
    Occupation:
    SEO co.
    Location:
    Quebec
    Is it necessary for a wordpress site? You can manually decide which page you want to index/noindex in the Cpanel.
     
  6. Alexfrance

    Alexfrance Regular Member

    Joined:
    Jul 26, 2010
    Messages:
    337
    Likes Received:
    73
    Occupation:
    Intel Pentium I 75 Mhz Processor
    Location:
    Somewhere in France, 20 km from Belgium
    yes, i also do it for the sites that have anything BH on it, for some other sites (all 100% WH), I dont do it cuz I'm too lazy
     
  7. rohuhottie

    rohuhottie Newbie

    Joined:
    Aug 11, 2010
    Messages:
    32
    Likes Received:
    4
    There is no need or compulsion of a robots.txt file.. If you don't have a robots.txt file then the Google bot will have its default behavior on your site.. Robots.txt is helpful if you want to block or make the Google bot work in a specified way..