1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

how to create robots.txt file?

Discussion in 'White Hat SEO' started by LiveSolution14, Apr 23, 2014.

  1. LiveSolution14

    LiveSolution14 Junior Member

    Joined:
    Mar 4, 2014
    Messages:
    145
    Likes Received:
    0
    hello guys, i need to know how to create robots.txt file for this website... if you have code give me.. thanks for help
     
  2. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Jr. VIP

    Joined:
    Nov 10, 2012
    Messages:
    12,916
    Likes Received:
    35,697
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:

    Did you even try Google?
    There are hundreds of examples, I wouldn't lie to you buddy.

    https://www.google.com/#q=robots.txt
     
    • Thanks Thanks x 2
  3. V

    V Elite Member

    Joined:
    May 18, 2012
    Messages:
    2,433
    Likes Received:
    2,646
    Occupation:
    Student
    Location:
    /tmp
    Code:
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    
    Sitemap: http://xyz.com/sitemap.xml
    [COLOR=#222222][FONT=Verdana]
    [/FONT][/COLOR]The directories you don't want to be indexed and crawled should be added after "Disallow: /dir/"
    and you should give the location of your site's sitemap in "Sitemap: ". It's basic, but you can learn more by searching on google, as W130SN said above. :)
     
  4. Finez

    Finez Regular Member

    Joined:
    Dec 28, 2013
    Messages:
    463
    Likes Received:
    90
    Gender:
    Male
    Occupation:
    Being lazzzzyyyy!
    Location:
    Wild East
  5. Aty

    Aty Jr. VIP Jr. VIP

    Joined:
    Jan 27, 2011
    Messages:
    6,011
    Likes Received:
    4,097
    Location:
    your 6 o'clock
    Home Page:
  6. Rank 360

    Rank 360 Jr. VIP Jr. VIP

    Joined:
    Dec 24, 2012
    Messages:
    2,389
    Likes Received:
    491
    Gender:
    Male
    Location:
    Google
  7. blackmagicseo

    blackmagicseo BANNED BANNED

    Joined:
    Mar 2, 2014
    Messages:
    117
    Likes Received:
    12
    - create new notepad
    - save the document the name should be "robots.txt"
    - paste the below code
    Code:
    [COLOR=#000000]User-agent: *
    [/COLOR]Disallow: /cgi-bin/
    Disallow: /tmp/ 
    [FONT=Verdana]Disallow: /junk/
    [/FONT][COLOR=#000000]sitemap: http://yousitenamehere.com/sitemap.xml
    [/COLOR]
    you need create sitemap before create robots file.
    -save the document once again
    - upload it in your server and save it.
    thats it.
     
    Last edited: Apr 29, 2014
  8. StrixMedia

    StrixMedia Jr. VIP Jr. VIP

    Joined:
    Jul 25, 2013
    Messages:
    3,003
    Likes Received:
    421
    Gender:
    Male
    Occupation:
    Internet Marketer
    Location:
    In a competitive world
    Home Page:
    create a new text document and save it as "robots.txt" and then put it in to your site folder...then you can write the formulas inside the txt file,like allow disallow...etc