- Apr 23, 2012
- 2,919
- 5,946
Hey all,
Haven't been on much lately. Have been busy overhauling things at my service end.
Time to get back in the game though.
--
So - almost everyone of us has, at some point in our lives, come face to face with a hacked wordpress site. We could have all the Wordfence in the world - but sometimes, somewhere, some 0-day exploit makes it way and ends up infecting our site. The site then turns into a doorway page, a parasite for some unscrupulous link builder, a host for some mailer, a phishing page host or one of many.
As someone who owns several websites - I have either found or am continually finding ways to automate a lot of work.
One of the most common issues with a hacked wordpress website is the creation of thousands of doorway pages (often in Chinese). These pages are then run through indexers or blasted with Xrumer or Scrapebox resulting in them getting indexed. This is a problem!
A bigger problem is these pages getting indexed and cannibalizing your actual rankings.
And an even bigger problem is, Google let's you submit one link at a time for removal. You read that right, ONE!
Don't they get removed on their own?
A lot of people will tell you that once you remove these pages and they all start hitting 404, they'll be gone from Google, as well.
But that is simply not the case. I have seen first hand several sites (authority sites, 10+ year old business sites with a solid link profile etc.) get hacked, get restored and still have their SERPs showing up those pages from 6 months ago.
--
Earlier today, a client/friend pinged me on Skype. He has a Private network of 17 money sites - which were hacked in September, 2018 owing to the company managing them not keeping them updated.
The sites stayed hacked for 6 days, before they were noticed, restored to their older selves and updated.
His sites have at the most, 100 posts/pages, on an average. However, a few of those sites were showing as many as 15000 pages indexed in Google. Pages, that when clicked on led to a 404.
He wanted me to write a bot/macro/automation program to remove all of these from the SERPs.
I didn't have time - so I sent him the how-to. Below is the same for you guys. I thought it'd benefit the community. (Edited for brevity and first/third person purposes)
//
Things you'll need -
- A Google Search Console account linked to your site.
- Google Chrome
Step 1 - Scraping Google SERPs
Goto This page and drag the Bookmarket Button to your Bookmark bar.
https://nichemojo.com/bookmark.php
(BHW won't let me add the code here, so added it to the above page. No ads/tracking or signup)
Shout out CognitiveSEO for developing the Bookmarklet.
To test if the above is working, just goto Google and search for anything. Then click the Bookmarklet in your Bookmark Toolbar and it should scrape and show you the list of all URLs.
Step 2 - Getting Google to show 100 results per page.
If you're going to do this, you might as well, do it large.
Google, by default, displays 10 results per page. So in order to get more, click on "Settings" just below the search bar and goto "Search Settings".
On that page look for the option, "Results per Page" drag it to 100.
Restart Chrome.
Step 3 - Even more results per page
While scrapping, you need as many results as you can. Scrapping 100 results at one go can get time consuming, if you have 15000 pages to work through.
In comes, ginfinity. A chrome extension that adds infinity scroll to Google Search pages.
https://chrome.google.com/webstore/detail/ginfinity/dgomfdmdnjbnfhodggijhpbmkgfabcmn?hl=en
Install this. Try searching and scrolling and you'll see what I mean.
Step 4 - Automating the URL Submission
To do this, we need to do some changes to Chrome. Worry not, they're easy.
1. Go here - https://github.com/noitcudni/google-webmaster-tools-bulk-url-removal and click on the green button at the top that says "Clone or Download". In the box that opens up, click "Download Zip".
2. Once the zip has downloaded, extract it. Remember the location.
3. In the Chrome URL bar enter chrome://extensions/ and hit enter.
4. Towards the top right, you'll see a toggle switch with the words 'Developer Mode" - toggle it to ON.
5. Once you do that, a button should appear on the top left of the screen that says "Load Unpacked" click on that.
6. In the window that opens, select the directory where you extracted the above zip and hit Open.
1. Search for "site:yoursite.com" in Google. Keep scrolling down as much as you can, so ginfinity keeps on loading links.
2. Click on the Bookmarklet and you'll get all the URLs in a new Window.
3. Scroll down in this new Window and you'll find text versions of your URLs. Copy them and save them to a text file.
4. Give the file a lookover once and remove all valid URLs (Legit pages on your site). You can also compare this against your sitemap and remove URLs from there.
5. Log in to your Google Search Console and visit https://www.google.com/webmasters/tools/removals
5. Right besides the "Request Removal" button - you should see a "Upload your File" button - thanks to the extension we installed.
6. Click on this button, select the text file with your links and let 'er rip.
7. Sit back and enjoy as it submits all hacked URLs for removals automatically, one by one.
--
P.S - Removal is done in a few hours. Have never seen it take more than 24 hours. When you submit, they will appear as "Pending" in a column below - if you visit the same page again after a while, they'll show up as "Removed".
P.P.S - After the first pages have been marked REMOVED, run a site: query again and repeat the steps if some pages still exist. Continue till the SERPs are completely cleaned.
P.P.P.S - You can move away from the tab and continue with your work.
P.P.P.P.S - If you accidentally add a link to a page that exists and isn't a 404 - the process will stop, so keep checking periodically.
P.P.P.P.P.S - The Amazon guide will come. Just let me get my services back on track and recalibrated with the market's nerve (which am almost done with), then I can concentrate on that and build your guide. Thank you for the patience.
Haven't been on much lately. Have been busy overhauling things at my service end.
Time to get back in the game though.
--
So - almost everyone of us has, at some point in our lives, come face to face with a hacked wordpress site. We could have all the Wordfence in the world - but sometimes, somewhere, some 0-day exploit makes it way and ends up infecting our site. The site then turns into a doorway page, a parasite for some unscrupulous link builder, a host for some mailer, a phishing page host or one of many.
As someone who owns several websites - I have either found or am continually finding ways to automate a lot of work.
One of the most common issues with a hacked wordpress website is the creation of thousands of doorway pages (often in Chinese). These pages are then run through indexers or blasted with Xrumer or Scrapebox resulting in them getting indexed. This is a problem!
A bigger problem is these pages getting indexed and cannibalizing your actual rankings.
And an even bigger problem is, Google let's you submit one link at a time for removal. You read that right, ONE!
Don't they get removed on their own?
A lot of people will tell you that once you remove these pages and they all start hitting 404, they'll be gone from Google, as well.
But that is simply not the case. I have seen first hand several sites (authority sites, 10+ year old business sites with a solid link profile etc.) get hacked, get restored and still have their SERPs showing up those pages from 6 months ago.
--
Earlier today, a client/friend pinged me on Skype. He has a Private network of 17 money sites - which were hacked in September, 2018 owing to the company managing them not keeping them updated.
The sites stayed hacked for 6 days, before they were noticed, restored to their older selves and updated.
His sites have at the most, 100 posts/pages, on an average. However, a few of those sites were showing as many as 15000 pages indexed in Google. Pages, that when clicked on led to a 404.
He wanted me to write a bot/macro/automation program to remove all of these from the SERPs.
I didn't have time - so I sent him the how-to. Below is the same for you guys. I thought it'd benefit the community. (Edited for brevity and first/third person purposes)
//
Things you'll need -
- A Google Search Console account linked to your site.
- Google Chrome
Step 1 - Scraping Google SERPs
Goto This page and drag the Bookmarket Button to your Bookmark bar.
https://nichemojo.com/bookmark.php
(BHW won't let me add the code here, so added it to the above page. No ads/tracking or signup)
Shout out CognitiveSEO for developing the Bookmarklet.
To test if the above is working, just goto Google and search for anything. Then click the Bookmarklet in your Bookmark Toolbar and it should scrape and show you the list of all URLs.
Step 2 - Getting Google to show 100 results per page.
If you're going to do this, you might as well, do it large.
Google, by default, displays 10 results per page. So in order to get more, click on "Settings" just below the search bar and goto "Search Settings".
On that page look for the option, "Results per Page" drag it to 100.
Restart Chrome.
Step 3 - Even more results per page
While scrapping, you need as many results as you can. Scrapping 100 results at one go can get time consuming, if you have 15000 pages to work through.
In comes, ginfinity. A chrome extension that adds infinity scroll to Google Search pages.
https://chrome.google.com/webstore/detail/ginfinity/dgomfdmdnjbnfhodggijhpbmkgfabcmn?hl=en
Install this. Try searching and scrolling and you'll see what I mean.
Step 4 - Automating the URL Submission
To do this, we need to do some changes to Chrome. Worry not, they're easy.
1. Go here - https://github.com/noitcudni/google-webmaster-tools-bulk-url-removal and click on the green button at the top that says "Clone or Download". In the box that opens up, click "Download Zip".
2. Once the zip has downloaded, extract it. Remember the location.
3. In the Chrome URL bar enter chrome://extensions/ and hit enter.
4. Towards the top right, you'll see a toggle switch with the words 'Developer Mode" - toggle it to ON.
5. Once you do that, a button should appear on the top left of the screen that says "Load Unpacked" click on that.
6. In the window that opens, select the directory where you extracted the above zip and hit Open.
You now have everything you need.
Final Steps
1. Search for "site:yoursite.com" in Google. Keep scrolling down as much as you can, so ginfinity keeps on loading links.
2. Click on the Bookmarklet and you'll get all the URLs in a new Window.
3. Scroll down in this new Window and you'll find text versions of your URLs. Copy them and save them to a text file.
4. Give the file a lookover once and remove all valid URLs (Legit pages on your site). You can also compare this against your sitemap and remove URLs from there.
5. Log in to your Google Search Console and visit https://www.google.com/webmasters/tools/removals
5. Right besides the "Request Removal" button - you should see a "Upload your File" button - thanks to the extension we installed.
6. Click on this button, select the text file with your links and let 'er rip.
7. Sit back and enjoy as it submits all hacked URLs for removals automatically, one by one.
--
P.S - Removal is done in a few hours. Have never seen it take more than 24 hours. When you submit, they will appear as "Pending" in a column below - if you visit the same page again after a while, they'll show up as "Removed".
P.P.S - After the first pages have been marked REMOVED, run a site: query again and repeat the steps if some pages still exist. Continue till the SERPs are completely cleaned.
P.P.P.S - You can move away from the tab and continue with your work.
P.P.P.P.S - If you accidentally add a link to a page that exists and isn't a 404 - the process will stop, so keep checking periodically.
P.P.P.P.P.S - The Amazon guide will come. Just let me get my services back on track and recalibrated with the market's nerve (which am almost done with), then I can concentrate on that and build your guide. Thank you for the patience.
