Post Penguin Observation - WordPress As Addon To A Website

I have a website that got hit very hard by Penguin. Previously, it ranked on the first page for a variety of keyword phrases that are highly searched in its niche. Now, the site bounces around between page 49 and page 56 of Google's SERP's for those main keywords.

The site was started ten years ago, and has good quality content (I know that because humans that visit the site email me feedback congratulating me on the site regularly. When it first was started, it was built on PHP and HTML. From almost day one, it has held excellent rankings in Google until Penguin.

There are many articles on the site that cover a wide variety of sub-categories of it's topic.

About 5 years ago (I think), I added a blog to the website with WordPress. In other words, the WP platform is separate from the rest of the website's platform, but is part of the domain name. In quite a few of the blog posts, I refer to and link to articles within the "main" part of the website that have been published, and I have tagged those posts.

What is interesting to me is that for longer keyword phrases, my site does have some rankings, but in the SERP's, it is most often the blog's tag subdirectory that is ranking. In other words, the articles themselves seem to have lost rank but the WP part of the site still has some rankings.

As far as what caused the Penguin slap on this particular site, I can only surmise as there were a number of things that were done to the site back in February which caused some issues:

1. We installed a script that in retrospect, wish we had not. It scraped youtube videos within the genre of this site - but did this in a "never ending" way. I received a "thin content" warning from Google about a month after the script was installed. We immediately removed the script.

2. Somehow Google discovered both the IP address and servername the website resides on. One day when I checked GWMT's, it was reporting a massive increase in links to the site - from about 25,000 to over one million! Note that I did NOT receive a "unnatural link building" notification (although I expected one). Upon further investigation, these additional so called backlinks were from the IP address and servername as if they were two separate websites linking to the site - and most of these links were from the video script pages that were created).

I removed the video script entirely, and now have GWMT's reporting about 280,000 404 errors on the website. I've also ensured rel canonical has been set correctly throughout the site to hopefully let Google know that when they are indexing ipaddress/some directory, it's actually the website. Hopefully the 404 errors Google is reporting will also disappear over time.

I am not sure if the above is what caused the Penguin slap, but at the same time I thought it was interesting that WP tag pages are ranking for long tail keyword phrases whereas the actual articles that contain the content being linked to, are not ranking.

Just some further data to digest here!

I would think you could clean a lot of that up in the robots.txt file. You can also do a 404 redirect in your .htaccess file by simply adding this line

ErrorDocument 404 /

Click to expand...

Or you can redirect the WP 404's to the main wordpress blog home page by creating a 404.php page and put it in the WP directory. Make a page that has only this code:

<?php
header("HTTP/1.1 301 Moved Permanently");
header("Location: ".get_bloginfo('url'));
exit();
?>

Click to expand...

Both of those methods will clean up the 404's for sure.

Thanks for the reply - yes, the 404's are being cleaned up - slowly. The video script was placed in its own subdirectory and robots.txt instructs search engines not to index that subdirectory.

However, I still find it interesting that Google's SERP's, for long tail keyword phrases are providing results for the "tag" pages in the blog, but not results for the actual article that is on the site, but outside of the WP blog.

That is because you did not have your robots.txt file setup properly. You needed to have added this to it:

Disallow: /tag/

Click to expand...

That solution I gave you in my first reply will clean up the 404 instantly, why on earth would you be happy with slowly? That's crazy when you have 280,000 404's, I'd be for cleaning that up in a hurry.

It is no secret Google loves wordpress. If your main site is old and has a lot of depreciated HTML in the markup then Google will most likely index what it likes best, in this case it was the wordpress stuff.

Its also customary in BHW when someone gives you a fix you need to either click the little thanks button under their reply or the add reputation if it was a good fix doing so will get a lot more people helping you out in the future

caitlin said:

That is because you did not have your robots.txt file setup properly. You needed to have added this to it:


That solution I gave you in my first reply will clean up the 404 instantly, why on earth would you be happy with slowly? That's crazy when you have 280,000 404's, I'd be for cleaning that up in a hurry.

It is no secret Google loves wordpress. If your main site is old and has a lot of depreciated HTML in the markup then Google will most likely index what it likes best, in this case it was the wordpress stuff.

Click to expand...

A misunderstanding - Google is "slowly" reducing the number of 404's it is reporting in GWMT. I have instituted an "instant" solution, but Google is taking its time in lowering the number.

And we are beginning to export the entire site to WP - however that is a really big job to do as it's a fairly massive site (hundreds upon hundreds of articles) so it is going to take a bit of time.

But thank you! And.. thank your advising me of the "custom" here

Please can you PM me with the script name?

psytrance said:

Please can you PM me with the script name?

Click to expand...

My partner installed it so I am not sure what it was. I have sent an email asking for it; when I get a reply, will send PM.