stop google indexing my links page
- Thread starter nickjett
- Start date
- Dec 6, 2011
- 1,014
- 673
well, in order to prevent Google from indexing your outbound links but than again to make them visible to the users, you need a server side script (so it wont be visible to the users who viewsource your page code) to handle the Google VS. Visitor detection.
you need to pay attention to the Google Crawlers patterns available to you which are:
1. Useragent string.
2. Bot Ip net ranges.
The UA string of Google bot will normally include the string 'google' and/or 'googlebot' - we need to grab this
The IP ranges vary, and you'll need to conduct a research for grabbing those ip ranges - I'll attach here an IP ranges that you can start with.
The process is pretty simple, you set a PHP script that checks the above parameters against the visitor to determine whether it's a user or a Google bot. If a visitor UA contains the 'google' string, and/or is within the IP ranges defined by you as of Google bot, it's Google bot so print different HTML code for it; otherwise, it's a real user so print the HTML code that presents the outbound links.
Check out the following PHP code - it should be inserted into your site at the relevant page.
you need to pay attention to the Google Crawlers patterns available to you which are:
1. Useragent string.
2. Bot Ip net ranges.
The UA string of Google bot will normally include the string 'google' and/or 'googlebot' - we need to grab this
The IP ranges vary, and you'll need to conduct a research for grabbing those ip ranges - I'll attach here an IP ranges that you can start with.
The process is pretty simple, you set a PHP script that checks the above parameters against the visitor to determine whether it's a user or a Google bot. If a visitor UA contains the 'google' string, and/or is within the IP ranges defined by you as of Google bot, it's Google bot so print different HTML code for it; otherwise, it's a real user so print the HTML code that presents the outbound links.
Check out the following PHP code - it should be inserted into your site at the relevant page.
PHP:
<?php $useragent = $_SERVER['HTTP_USER_AGENT']; $ip = $_SERVER['REMOTE_ADDR']; function checkUserAgent(){ global $useragent; $useragents = array('Googlebot', 'google','AdsBot-Google','adsbot'); foreach($useragents as $string){ if (stristr($useragent, $string)){ return true; } } } function checkGoogleIP (){ global $ip; $parts = explode('.',$ip); if( ($parts[0] == 66 AND $parts[1] == 249 AND (64 <= $parts[2]) AND ($parts[2] <= 95)) ) return true; } function checkGoogle(){ if ( checkGoogleIP() OR checkUserAgent() ){ // it's Google so output html code without links }else{ // it's a Visitor so output html code with links } } // fire the function checkGoogle();?>
- Dec 6, 2011
- 1,014
- 673
not good. since he wants to prevent users from checking on his outbound links. a quick robots.txt check will reveal his method - not good!
GiorgioB
Supreme Member
- Feb 28, 2012
- 1,288
- 1,332
users? I thought it was just the spiders he wanted to prevent from crawling and indexing (btw i meant noindex, not no follow). I did that for my admin URL for my websites, thought it was fine
really? pages i told them not to index were not indexed...
- Dec 6, 2011
- 1,014
- 673
Check out the bold & red. He wants one-way backlins thus needs to prevent from links exchangers to find out that their links will never get indexed - therefore, he needs to prevent them from finding out his trick, and prevent Google from indexing those links
I have checked a few ips Google uses at seems to be anywhere between 66.249.73.... 66.249.75...., in your script the range seems to be a bit wider, but shouldnt be a problem. Im not very good with scripts, is the script you gave me complete?
All i have to do is add that to a php page? and my outbound links wont show up to googlebot? How can i test it works?
phpbuilt
Account compromised December 2025
- May 16, 2011
- 1,865
- 5,369
First create a page called cloak.php. Save the following to it.
Next, on your page that you want to restrict google/yahoo/bing from seeing, put this at the very top.
This will make Google/Yahoo/Bing see your link page as a 301 to your homepage (so you recover any lost PR, and Google doesn't see it).
Code:
<?php
/*
NiX Cloak Copyright (C) 2012 MyProxyLists.com
License Agreement: You may not use this code to sell any kind of cloaking services. Feel free to modify for your needs.
DO NOT SHARE THIS CODE OUTSIDE BHW (blackhatworld.com). THE CODE IS BHW EXCLUSIVE!
*/
function search_engine($ipaddr = '') {
if (!valid_ip($ipaddr)) {
return false;
}
/* Ranges for Google/Yahoo/Bing are 99.9% accurate as of 16th of November 2012 */
$ranges = array(
"89.207.224.0/21",
"8.8.8.0/24",
"8.8.4.0/24",
"8.35.200.0/21",
"8.35.192.0/21",
"8.34.216.0/21",
"8.34.208.0/21",
"74.125.0.0/16",
"72.14.192.0/18",
"70.32.128.0/19",
"66.249.64.0/19",
"66.102.0.0/20",
"64.233.182.0/23",
"64.233.173.0/24",
"64.233.172.0/24",
"64.233.168.0/23",
"64.233.160.0/19",
"64.15.112.0/20",
"216.58.192.0/19",
"216.239.32.0/19",
"209.85.128.0/17",
"208.65.152.0/22",
"208.117.224.0/19",
"207.223.160.0/20",
"199.223.232.0/21",
"199.192.112.0/22",
"193.142.125.0/24",
"192.178.0.0/15",
"173.255.112.0/20",
"173.194.0.0/16",
"172.217.0.0/16",
"142.250.0.0/15",
"113.197.105.0/24",
"108.59.80.0/20",
"108.177.0.0/17",
"108.170.192.0/18",
"192.197.157.0/24",
"168.62.0.0/15",
"168.61.0.0/16",
"167.220.40.0/21",
"157.60.31.0/24",
"157.60.23.0/24",
"157.58.90.0/24",
"157.55.0.0/16",
"157.56.0.0/16",
"138.91.0.0/16",
"137.135.0.0/16",
"137.116.0.0/15",
"134.170.0.0/16",
"132.245.0.0/16",
"131.253.8.0/24",
"131.253.64.0/18",
"131.253.62.0/23",
"131.253.61.0/24",
"131.253.6.0/24",
"131.253.5.0/24",
"131.253.32.0/20",
"131.253.3.0/24",
"131.253.24.0/21",
"131.253.21.0/24",
"131.253.18.0/24",
"131.253.12.0/22",
"131.253.1.0/24",
"111.221.69.0/24",
"111.221.66.0/24",
"111.221.23.0/24",
"111.221.112.0/21",
"94.245.64.0/18",
"70.37.135.0/24",
"70.37.128.0/23",
"70.37.128.0/18",
"70.37.0.0/17",
"65.55.86.0/23",
"65.55.117.0/24",
"65.54.92.0/23",
"65.54.88.0/23",
"65.52.0.0/14",
"64.4.0.0/18",
"216.32.240.0/22",
"216.32.180.0/22",
"213.199.128.0/18",
"209.240.192.0/19",
"209.185.128.0/22",
"208.68.136.0/21",
"207.82.250.0/23",
"207.68.128.0/18",
"207.46.98.0/24",
"207.46.96.0/19",
"207.46.77.0/24",
"207.46.62.0/24",
"207.46.58.0/24",
"207.46.32.0/20",
"207.46.128.0/17",
"207.46.0.0/19",
"206.191.224.0/19",
"206.138.168.0/21",
"205.248.96.0/19",
"204.95.96.0/20",
"204.79.197.0/24",
"204.79.196.0/24",
"204.79.195.0/24",
"204.152.140.0/23",
"199.74.210.0/24",
"199.60.28.0/24",
"199.30.16.0/20",
"199.242.48.0/21",
"199.103.90.0/23",
"199.103.122.0/24",
"198.49.8.0/24",
"198.206.164.0/24",
"198.200.130.0/24",
"194.69.96.0/19",
"193.149.64.0/19",
"192.84.160.0/23",
"192.84.159.0/24",
"192.48.225.0/24",
"98.139.96.0/22",
"98.139.48.0/20",
"98.139.104.0/22",
"98.139.0.0/20",
"98.137.64.0/20",
"98.137.216.0/21",
"98.137.208.0/21",
"98.137.192.0/20",
"98.136.64.0/18",
"98.136.128.0/19",
"8.12.146.0/24",
"76.13.5.0/24",
"76.13.238.0/23",
"76.13.236.0/23",
"76.13.232.0/23",
"76.13.234.0/23",
"76.13.232.0/22",
"76.13.192.0/20",
"76.13.215.0/24",
"76.13.216.0/21",
"76.13.224.0/21",
"74.6.232.0/22",
"74.6.236.0/22",
"74.6.240.0/21",
"74.6.64.0/19",
"76.13.0.0/17",
"76.13.128.0/18",
"74.6.228.0/22",
"74.6.156.0/23",
"74.6.152.0/23",
"74.6.148.0/23",
"74.6.150.0/23",
"74.6.0.0/19",
"72.30.96.0/20",
"72.30.64.0/19",
"72.30.240.0/20",
"72.30.224.0/21",
"72.30.220.0/22",
"72.30.216.0/22",
"72.30.192.0/20",
"72.30.128.0/18",
"68.180.152.0/21",
"67.28.115.0/24",
"67.195.224.0/19",
"67.195.192.0/19",
"67.195.160.0/19",
"66.228.188.0/22",
"66.228.184.0/22",
"66.218.64.0/19",
"66.163.184.0/21",
"216.109.112.0/20",
"209.73.176.0/21",
"209.73.160.0/20"
);
shuffle($ranges);
foreach ($ranges as $cidr) {
if (ip_in_block($ipaddr, $cidr)) {
return true;
}
}
return false;
}
function valid_ip($ipaddr = '') {
if (preg_match("/^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/", $ipaddr)) {
$parts = explode(".", $ipaddr);
foreach ($parts as $part) {
if (intval($part) > 255 || intval($part) < 0) {
return false;
}
}
return true;
} else {
return false;
}
}
function ip_in_block($ipaddr = '', $cidr = '') {
if (valid_ip($ipaddr) && preg_match('/^([0-9]{1,3}\.)([0-9]{1,3}\.)([0-9]{1,3}\.)([0-9]{1,3})+(\/[0-9]{1,2})$/', $cidr, $m)) {
$dq_host = $m[1] . $m[2] . $m[3] . $m[4];
$cdr_nmask = strtok("$m[5]", "/");
$bin_host = dqtobin($dq_host);
$bin_net = (str_pad(substr($bin_host, 0, $cdr_nmask), 32, 0));
$network = bintodq($bin_net);
if (ip_in_network($ipaddr, $network, $cdr_nmask)) {
return true;
} else {
return false;
}
} else {
return false;
}
}
function dqtobin($dqin) {
$bin = null;
$dq = explode(".", $dqin);
for ($i = 0; $i < 4; $i++) {
$bin[$i] = str_pad(decbin($dq[$i]), 8, "0", STR_PAD_LEFT);
}
return implode("", $bin);
}
function bintodq($binin = '') {
$dq = null;
$binin = explode(".", chunk_split($binin, 8, "."));
for ($i = 0; $i < 4; $i++) {
$dq[$i] = bindec($binin[$i]);
}
return implode(".", $dq);
}
function ip_in_network($ip, $net_addr, $net_mask) {
$ip_binary_string = sprintf("%032b", ip2long($ip));
$net_binary_string = sprintf("%032b", ip2long($net_addr));
return (substr_compare($ip_binary_string, $net_binary_string, 0, $net_mask) === 0);
}
?>Next, on your page that you want to restrict google/yahoo/bing from seeing, put this at the very top.
Code:
require_once("cloak.php");
$remote_addr = $_SERVER['REMOTE_ADDR'];
if (search_engine($remote_addr)) {
$newurl = "http://".$_SERVER["HTTP_HOST"];
Header( "HTTP/1.1 301 Moved Permanently" );
Header( "Location: $newurl" );
exit;
}This will make Google/Yahoo/Bing see your link page as a 301 to your homepage (so you recover any lost PR, and Google doesn't see it).