1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

how to create robots.txt file?

Discussion in 'White Hat SEO' started by LiveSolution14, Apr 23, 2014.

  1. LiveSolution14

    LiveSolution14 Junior Member

    Joined:
    Mar 4, 2014
    Messages:
    145
    Likes Received:
    0
    hello guys, i need to know how to create robots.txt file for this website... if you have code give me.. thanks for help
     
  2. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Jr. VIP

    Joined:
    Nov 10, 2012
    Messages:
    11,150
    Likes Received:
    31,533
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:

    Did you even try Google?
    There are hundreds of examples, I wouldn't lie to you buddy.

    https://www.google.com/#q=robots.txt
     
    • Thanks Thanks x 2
  3. V

    V Elite Member

    Joined:
    May 18, 2012
    Messages:
    2,225
    Likes Received:
    2,567
    Occupation:
    Student
    Location:
    /tmp
    Code:
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    
    Sitemap: http://xyz.com/sitemap.xml
    [COLOR=#222222][FONT=Verdana]
    [/FONT][/COLOR]The directories you don't want to be indexed and crawled should be added after "Disallow: /dir/"
    and you should give the location of your site's sitemap in "Sitemap: ". It's basic, but you can learn more by searching on google, as W130SN said above. :)
     
  4. Finez

    Finez Regular Member

    Joined:
    Dec 28, 2013
    Messages:
    396
    Likes Received:
    75
    Gender:
    Male
    Occupation:
    Being lazzzzyyyy!
    Location:
    Wild East
  5. Aty

    Aty Jr. VIP Jr. VIP

    Joined:
    Jan 27, 2011
    Messages:
    5,837
    Likes Received:
    4,003
    Home Page:
  6. Rank 360

    Rank 360 Jr. VIP Jr. VIP

    Joined:
    Dec 24, 2012
    Messages:
    2,122
    Likes Received:
    460
    Gender:
    Male
    Location:
    Google
  7. blackmagicseo

    blackmagicseo Junior Member

    Joined:
    Mar 2, 2014
    Messages:
    110
    Likes Received:
    10
    Occupation:
    SEO analyst and CEO
    Location:
    India
    Home Page:
    - create new notepad
    - save the document the name should be "robots.txt"
    - paste the below code
    Code:
    [COLOR=#000000]User-agent: *
    [/COLOR]Disallow: /cgi-bin/
    Disallow: /tmp/ 
    [FONT=Verdana]Disallow: /junk/
    [/FONT][COLOR=#000000]sitemap: http://yousitenamehere.com/sitemap.xml
    [/COLOR]
    you need create sitemap before create robots file.
    -save the document once again
    - upload it in your server and save it.
    thats it.
     
    Last edited: Apr 29, 2014
  8. StrixMedia

    StrixMedia Jr. VIP Jr. VIP

    Joined:
    Jul 25, 2013
    Messages:
    2,420
    Likes Received:
    314
    Gender:
    Male
    Occupation:
    BHW Seller
    Location:
    In a competitive world
    Home Page:
    create a new text document and save it as "robots.txt" and then put it in to your site folder...then you can write the formulas inside the txt file,like allow disallow...etc