1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

how to restrict crawler to crawl some of my website pages?

Discussion in 'White Hat SEO' started by mishellrogen, Apr 26, 2013.

  1. mishellrogen

    mishellrogen Newbie

    Joined:
    Apr 22, 2013
    Messages:
    3
    Likes Received:
    0
    Occupation:
    php developer
    Location:
    india
    Home Page:
    hi guys, some of my site pages are incomplete,so i want that crawler should not crawled them .please tell me the answer as quick as possible.
     
  2. proweb

    proweb Regular Member

    Joined:
    Feb 24, 2011
    Messages:
    291
    Likes Received:
    144
    <meta name="robots" content="noindex, nofollow">

    OR

    Robots.txt

    User-agent: *
    Disallow: /private_file.html