1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

how to create robots.txt file?

Discussion in 'White Hat SEO' started by LiveSolution14, Apr 23, 2014.

  1. LiveSolution14

    LiveSolution14 Junior Member

    Joined:
    Mar 4, 2014
    Messages:
    145
    Likes Received:
    0
    hello guys, i need to know how to create robots.txt file for this website... if you have code give me.. thanks for help
     
  2. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Jr. VIP

    Joined:
    Nov 10, 2012
    Messages:
    12,463
    Likes Received:
    34,390
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:

    Did you even try Google?
    There are hundreds of examples, I wouldn't lie to you buddy.

    https://www.google.com/#q=robots.txt
     
    • Thanks Thanks x 2
  3. V

    V Elite Member

    Joined:
    May 18, 2012
    Messages:
    2,331
    Likes Received:
    2,603
    Occupation:
    Student
    Location:
    /tmp
    Code:
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    
    Sitemap: http://xyz.com/sitemap.xml
    [COLOR=#222222][FONT=Verdana]
    [/FONT][/COLOR]The directories you don't want to be indexed and crawled should be added after "Disallow: /dir/"
    and you should give the location of your site's sitemap in "Sitemap: ". It's basic, but you can learn more by searching on google, as W130SN said above. :)
     
  4. Finez

    Finez Regular Member

    Joined:
    Dec 28, 2013
    Messages:
    457
    Likes Received:
    90
    Gender:
    Male
    Occupation:
    Being lazzzzyyyy!
    Location:
    Wild East
  5. Aty

    Aty Jr. VIP Jr. VIP

    Joined:
    Jan 27, 2011
    Messages:
    5,994
    Likes Received:
    4,087
    Occupation:
    SEO (Senior Erection Officer)
    Location:
    your 6 o'clock
    Home Page:
  6. Rank 360

    Rank 360 Jr. VIP Jr. VIP

    Joined:
    Dec 24, 2012
    Messages:
    2,316
    Likes Received:
    480
    Gender:
    Male
    Location:
    Google
  7. blackmagicseo

    blackmagicseo Junior Member

    Joined:
    Mar 2, 2014
    Messages:
    117
    Likes Received:
    12
    Occupation:
    SEO analyst and CEO
    Location:
    India
    Home Page:
    - create new notepad
    - save the document the name should be "robots.txt"
    - paste the below code
    Code:
    [COLOR=#000000]User-agent: *
    [/COLOR]Disallow: /cgi-bin/
    Disallow: /tmp/ 
    [FONT=Verdana]Disallow: /junk/
    [/FONT][COLOR=#000000]sitemap: http://yousitenamehere.com/sitemap.xml
    [/COLOR]
    you need create sitemap before create robots file.
    -save the document once again
    - upload it in your server and save it.
    thats it.
     
    Last edited: Apr 29, 2014
  8. StrixMedia

    StrixMedia Jr. VIP Jr. VIP

    Joined:
    Jul 25, 2013
    Messages:
    2,815
    Likes Received:
    389
    Gender:
    Male
    Occupation:
    Internet Marketer
    Location:
    In a competitive world
    Home Page:
    create a new text document and save it as "robots.txt" and then put it in to your site folder...then you can write the formulas inside the txt file,like allow disallow...etc