1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

how to create robots.txt file?

Discussion in 'White Hat SEO' started by LiveSolution14, Apr 23, 2014.

  1. LiveSolution14

    LiveSolution14 Junior Member

    Joined:
    Mar 4, 2014
    Messages:
    145
    Likes Received:
    0
    hello guys, i need to know how to create robots.txt file for this website... if you have code give me.. thanks for help
     
  2. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,115
    Likes Received:
    28,557
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:

    Did you even try Google?
    There are hundreds of examples, I wouldn't lie to you buddy.

    https://www.google.com/#q=robots.txt
     
    • Thanks Thanks x 2
  3. V

    V Elite Member

    Joined:
    May 18, 2012
    Messages:
    2,113
    Likes Received:
    2,543
    Occupation:
    Student
    Location:
    /tmp
    Code:
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    
    Sitemap: http://xyz.com/sitemap.xml
    [COLOR=#222222][FONT=Verdana]
    [/FONT][/COLOR]The directories you don't want to be indexed and crawled should be added after "Disallow: /dir/"
    and you should give the location of your site's sitemap in "Sitemap: ". It's basic, but you can learn more by searching on google, as W130SN said above. :)
     
  4. Finez

    Finez Regular Member

    Joined:
    Dec 28, 2013
    Messages:
    334
    Likes Received:
    58
    Occupation:
    Licking Girls!
    Location:
    India
  5. Aty

    Aty Jr. VIP Jr. VIP

    Joined:
    Jan 27, 2011
    Messages:
    5,408
    Likes Received:
    3,696
    Home Page:
  6. Rank 360

    Rank 360 Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 24, 2012
    Messages:
    1,921
    Likes Received:
    447
    Gender:
    Male
    Location:
    Google
  7. blackmagicseo

    blackmagicseo Junior Member

    Joined:
    Mar 2, 2014
    Messages:
    106
    Likes Received:
    10
    Occupation:
    SEO analyst and CEO
    Location:
    India
    Home Page:
    - create new notepad
    - save the document the name should be "robots.txt"
    - paste the below code
    Code:
    [COLOR=#000000]User-agent: *
    [/COLOR]Disallow: /cgi-bin/
    Disallow: /tmp/ 
    [FONT=Verdana]Disallow: /junk/
    [/FONT][COLOR=#000000]sitemap: http://yousitenamehere.com/sitemap.xml
    [/COLOR]
    you need create sitemap before create robots file.
    -save the document once again
    - upload it in your server and save it.
    thats it.
     
    Last edited: Apr 29, 2014
  8. StrixMedia

    StrixMedia Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 25, 2013
    Messages:
    2,067
    Likes Received:
    264
    Gender:
    Male
    Location:
    In a competitive world
    create a new text document and save it as "robots.txt" and then put it in to your site folder...then you can write the formulas inside the txt file,like allow disallow...etc