1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

What Is Robots.txt?

Discussion in 'Black Hat SEO' started by Samlko, Mar 6, 2012.

  1. Samlko

    Samlko Newbie

    Joined:
    May 28, 2009
    Messages:
    11
    Likes Received:
    0
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.
     
  2. Gradimir Stankovic

    Gradimir Stankovic Power Member

    Joined:
    Jan 10, 2010
    Messages:
    737
    Likes Received:
    845
    Location:
    404 not found
    Yes it is? Are you asking question or explaining what robots.txt is?
     
  3. calebmann

    calebmann Power Member Premium Member

    Joined:
    Sep 12, 2010
    Messages:
    541
    Likes Received:
    299
    Occupation:
    FULL TIME INTERNET
    Location:
    Digitopia
    that is exactly what it is! nice job answering your own question buddy!!
     
  4. proxygo

    proxygo Jr. VIP Jr. VIP

    Joined:
    Nov 2, 2008
    Messages:
    19,623
    Likes Received:
    10,242
    Gender:
    Male
    Occupation:
    PROVIDING PROXIES FOR URL SCRAPING
    Location:
    UK - ALWAYS ON BHW
    Home Page:
    techly reported again for replying to threads from 1-5 years ago
     
  5. Sonia Sharma

    Sonia Sharma Junior Member

    Joined:
    Oct 16, 2017
    Messages:
    186
    Likes Received:
    42
    Gender:
    Female
    Location:
    India
    Stupid thread
     
    • Thanks Thanks x 1