1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Advice please - Robots.txt trick?

Discussion in 'Black Hat SEO' started by YellowHats, Apr 10, 2013.

  1. YellowHats

    YellowHats Junior Member

    Joined:
    Mar 10, 2013
    Messages:
    195
    Likes Received:
    135
    Occupation:
    Director of SEO
    Location:
    Middle East
    I'm working for a client who just launched two big websites. Both websites are platforms that were purchased from another company so, at the moment almost every one of the pages on the site is an exact content-duplicate of the other company's pages.

    All the pages are blocked in robots.txt so Google isn't indexing the duplicate content. The duplicate content is being rewritten and as each page is rewritten, it's removed from the robotst.txt.

    Will Google see this as new content being added to the website? Will this help keep the website 'fresh'? Or will it just see this as previously disallowed pages now being allowed?

    Any ideas?
     
  2. Stufferizer

    Stufferizer Regular Member

    Joined:
    Nov 6, 2012
    Messages:
    237
    Likes Received:
    73
    If it's not duplicate content, it is fresh in g's eyes. However, you should be careful as the algorithm might be able to spot if the difference between articles is not "sufficient" (did not prog the algo myself, so I cannot give a more strict sense of sufficient).

    Anyways, it might take some more time until your "new" pages get indexed since the crawler has to do several steps: download robots.txt, put results into an internal database (say the "allow database") that is used by the crawler - then, only if the crawler uses the updated database, the articles will get indexed.
     
    • Thanks Thanks x 1
  3. the_demon

    the_demon Jr. Executive VIP

    Joined:
    Nov 23, 2008
    Messages:
    3,177
    Likes Received:
    1,563
    Occupation:
    Search Engine Marketing
    Location:
    The Internet
    Why didn't you just 301 the old domain to the new one if they are about the same exact thing???
     
  4. YellowHats

    YellowHats Junior Member

    Joined:
    Mar 10, 2013
    Messages:
    195
    Likes Received:
    135
    Occupation:
    Director of SEO
    Location:
    Middle East
    It's not a change in domain. Our client (company x) bought a platform from company y. Part of the platform included exactly the same website as company y.

    Stufferizer - thanks. I guess in the end this could end up working decently. Keep the site fresh in google's eyes for a while...