1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

robot.txt

Discussion in 'Black Hat SEO' started by sadave cxx, Jun 26, 2017.

  1. sadave cxx

    sadave cxx Newbie

    Joined:
    May 27, 2017
    Messages:
    25
    Likes Received:
    1
    Gender:
    Male
    I don't understand what is the sole purpose of the robot.txt on SEO strategy, can someone explain to me?
     
  2. darknetz

    darknetz BANNED BANNED

    Joined:
    Jun 22, 2017
    Messages:
    40
    Likes Received:
    3
    Web site owners use the /robots.txt file to give instructions about their site to web robots
     
  3. Neo-Hat

    Neo-Hat Newbie

    Joined:
    Jun 27, 2017
    Messages:
    2
    Likes Received:
    0
    Gender:
    Male
    Occupation:
    Security Information Analyst
    Location:
    Brazil
    You also can use Robots.txt to find hidden pages, generally as a web-crawler does, ex: spider technique.
     
  4. loedown

    loedown Jr. VIP Jr. VIP

    Joined:
    Jun 29, 2009
    Messages:
    1,610
    Likes Received:
    505
    Home Page:
    Use google.
     
  5. drimpalarez

    drimpalarez Newbie

    Joined:
    Jun 27, 2017
    Messages:
    5
    Likes Received:
    1
    Sometimes you dont want certain pages f your website to be indexed on search engines, so you use robots to allow and disallow pages/directories to crawl and index.
     
  6. Initial Effort

    Initial Effort Jr. VIP Jr. VIP Premium Member

    Joined:
    Oct 29, 2010
    Messages:
    1,865
    Likes Received:
    271
    Location:
    Minnesota
  7. Burglar SEO

    Burglar SEO Junior Member

    Joined:
    May 7, 2017
    Messages:
    154
    Likes Received:
    27
    Gender:
    Female
    Robot.txt are used to tell search engines to not crawl this particular pages that is the only purpose
     
  8. We Bring Rank

    We Bring Rank Jr. VIP Jr. VIP

    Joined:
    Aug 21, 2016
    Messages:
    600
    Likes Received:
    65
    Gender:
    Male
    Occupation:
    Digital Marketing Analyst
    Home Page:
    Yes I would accept @Burglar SEO views, Hide particular links in google & maintained your site trusted status :)
     
  9. seoways

    seoways Jr. VIP Jr. VIP

    Joined:
    Dec 19, 2009
    Messages:
    5,313
    Likes Received:
    803
    Gender:
    Male
    Location:
    MILKYWAY ;)
    It controls the bots/crawlers/spiders or whatever it is, what should be visited and what should not be.
     
  10. Aiden_1

    Aiden_1 Jr. VIP Jr. VIP

    Joined:
    Jun 25, 2015
    Messages:
    1,162
    Likes Received:
    199
    Gender:
    Male
    Occupation:
    SEO - Freelancer
    Location:
    At where you in
    Robot.txt file helps to reindex your modified page or index your new page at first while submitting it in GWT. Also, it gives the news to the search engine robots. So it's easy to crawl the new or modified site to index them.
     
  11. cycooperman24

    cycooperman24 Newbie

    Joined:
    Jul 5, 2017
    Messages:
    10
    Likes Received:
    3
    Gender:
    Male
    If you only start your web project, robot.txt disallow spy bots to check your not ready website and don't get them ranking your website without content. More often that's mean you will be far from top 100.