1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How Can i stop Google Spieder to Not to crawl my website/

Discussion in 'Black Hat SEO' started by balu360, Dec 15, 2009.

  1. balu360

    balu360 Newbie

    Joined:
    Aug 24, 2009
    Messages:
    14
    Likes Received:
    0
    I have a .info web page, that i have bought it for cheap from godaddy.com and i have hosted basic designs in several .info pages.

    all the designs are same and only Title tag changes..

    Would i have problem with google spieder becoz of similar page.


    Pls help me..

    i need your guideline.

    PS: Sorry for my bad english..
     
  2. origin

    origin Regular Member

    Joined:
    Nov 11, 2008
    Messages:
    334
    Likes Received:
    90
    Home Page:
    Similar web design won't affect the way Google "sees" your site, it's only crawling the contents and not the design. So, as long as your title/tag and contents are different, your sites are just fine.
     
  3. balu360

    balu360 Newbie

    Joined:
    Aug 24, 2009
    Messages:
    14
    Likes Received:
    0
    all the web page are same the contents and jpgs only Title tag is different.
     
  4. origin

    origin Regular Member

    Joined:
    Nov 11, 2008
    Messages:
    334
    Likes Received:
    90
    Home Page:
    You need a rewriter program to make your site contents a bit different. Check download forum.
     
  5. MuonTrail

    MuonTrail Registered Member

    Joined:
    Mar 5, 2009
    Messages:
    96
    Likes Received:
    93
  6. white92

    white92 Regular Member

    Joined:
    Nov 22, 2009
    Messages:
    310
    Likes Received:
    89
    Occupation:
    Student
    Location:
    Portugal
    I also have bought some .info domains but i can't get those indexed in google :/

    Regards.
     
  7. crazyflx

    crazyflx Elite Member

    Joined:
    Nov 9, 2009
    Messages:
    1,674
    Likes Received:
    4,825
    Location:
    http://CRAZYFLX.COM
    Home Page:
    Here is how you stop google, and any other search engine from crawling either:

    A: A specific page of your site
    B: Your whole site

    You need to open notepad on your computer. Then type the following (or just copy and paste it):

    Code:
    User-agent: *
    Disallow: /
    The * that appears after "user-agent:" refers to what search engine spider you want to follow the rules the come after "user-agent:"

    An * is a wild card, which means that ALL search engine spiders will follow the rules to come.

    Now, anything that comes after "Disallow:" will not be crawled by whatever search engine spiders you have "directed" with the "user agent" line.

    So, to place a " / " after "Disallow:" means that your ENTIRE site won't be crawled. You can also enter specific pages, like this:

    " /sample.htm "

    You can also enter entire folders of your site, and anything in those folders won't be crawled. You'd do that like this:

    " /samplefolder/ "

    After you're done typing all of your "rules" you save the .txt file as "robots". Then you upload the "robots.txt" file to the root folder on your server.