1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

how to create robots.txt file?

Discussion in 'White Hat SEO' started by LiveSolution14, Apr 23, 2014.

  1. LiveSolution14

    LiveSolution14 Junior Member

    Joined:
    Mar 4, 2014
    Messages:
    145
    Likes Received:
    0
    hello guys, i need to know how to create robots.txt file for this website... if you have code give me.. thanks for help
     
  2. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Jr. VIP

    Joined:
    Nov 10, 2012
    Messages:
    11,299
    Likes Received:
    31,975
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:

    Did you even try Google?
    There are hundreds of examples, I wouldn't lie to you buddy.

    https://www.google.com/#q=robots.txt
     
    • Thanks Thanks x 2
  3. V

    V Elite Member

    Joined:
    May 18, 2012
    Messages:
    2,228
    Likes Received:
    2,570
    Occupation:
    Student
    Location:
    /tmp
    Code:
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    
    Sitemap: http://xyz.com/sitemap.xml
    [COLOR=#222222][FONT=Verdana]
    [/FONT][/COLOR]The directories you don't want to be indexed and crawled should be added after "Disallow: /dir/"
    and you should give the location of your site's sitemap in "Sitemap: ". It's basic, but you can learn more by searching on google, as W130SN said above. :)
     
  4. Finez

    Finez Regular Member

    Joined:
    Dec 28, 2013
    Messages:
    419
    Likes Received:
    78
    Gender:
    Male
    Occupation:
    Being lazzzzyyyy!
    Location:
    Wild East
  5. Aty

    Aty Jr. VIP Jr. VIP

    Joined:
    Jan 27, 2011
    Messages:
    5,946
    Likes Received:
    4,066
    Home Page:
  6. Rank 360

    Rank 360 Jr. VIP Jr. VIP

    Joined:
    Dec 24, 2012
    Messages:
    2,186
    Likes Received:
    466
    Gender:
    Male
    Location:
    Google
  7. blackmagicseo

    blackmagicseo Junior Member

    Joined:
    Mar 2, 2014
    Messages:
    110
    Likes Received:
    10
    Occupation:
    SEO analyst and CEO
    Location:
    India
    Home Page:
    - create new notepad
    - save the document the name should be "robots.txt"
    - paste the below code
    Code:
    [COLOR=#000000]User-agent: *
    [/COLOR]Disallow: /cgi-bin/
    Disallow: /tmp/ 
    [FONT=Verdana]Disallow: /junk/
    [/FONT][COLOR=#000000]sitemap: http://yousitenamehere.com/sitemap.xml
    [/COLOR]
    you need create sitemap before create robots file.
    -save the document once again
    - upload it in your server and save it.
    thats it.
     
    Last edited: Apr 29, 2014
  8. StrixMedia

    StrixMedia Jr. VIP Jr. VIP

    Joined:
    Jul 25, 2013
    Messages:
    2,562
    Likes Received:
    348
    Gender:
    Male
    Occupation:
    BHW Seller
    Location:
    In a competitive world