Redirect browser traffic but allow SE spiders

christopher · Aug 28, 2009

like the title says, how can i redirect traffic coming from a certain site but allow the SE spiders through?

I know I can set up an htaccess to redirect traffic but will it also redirect SE's?

I'm fairly positive that code can be written in the htaccess to look at the source of a hit and know if it's a browser or a spider and then redirect accordingly, but, if I knew what I was doing I wouldn't be asking.

thanks

matapples01 · Aug 28, 2009

It's called cloaking. You can find solutions from $50 to $5,000. If you're using wordpress check out WPCloaker. There's a thread on here with a BHW discount too.

jonnyh431 · Aug 29, 2009

Try putting this somewhere in your index.php file, I haven't tested it but it should work.

Code:

<?php
$browser = $_SERVER['HTTP_USER_AGENT'];
if($browser == "googlebot" || "msnbot" || "yahoo")
{
header('Location: http://searchenginesgohere.com/');
}
else
{
header('Location: http://normalusersgohere.com/');
}
?>

BadArticle · Aug 29, 2009

You also need reverse DNS check to filter out fake robots

Code:

        $result = false;
        $ip = $_SERVER['REMOTE_ADDR'];
        $ip12 = substr($ip, 0, 12);
        $ip9 = substr($ip, 0, 9);
        $ip8 = substr($ip, 0, 8);
        $ip7 = substr($ip, 0, 7);
        $ip6 = substr($ip, 0, 6);
        $ip5 = substr($ip, 0, 5);
        if (($ip7=='66.249.')||($ip6=='65.55.')||($ip6=='72.30.')||($ip5=='74.6.')||($ip12=='216.129.119.')) {
            $result = true;
        }
        if ( !$result && preg_match("/.*(googlebot|slurp|msnbot|teoma|twiceler).*/i", $agent, $matches) ) {
            $host = gethostbyaddr( $ip );
            if ( $matches[1] == 'googlebot' ) {
                $result = ( substr($host, -14) == '.googlebot.com' );
            } elseif ( $matches[1] == 'slurp' ) {
                $result = ( substr($host, -16) == '.crawl.yahoo.net' ) || ( substr($host, -15) == '.corp.yahoo.com' );
            } elseif ( $matches[1] == 'msnbot' ) {
                $result = preg_match("/.*\.search\.(msn|live|bing)\.com$/i", $host);
            } elseif ( $matches[1] == 'twiceler' ) {
                $result = preg_match("/.*\.(cuill|cuil)\.com$/i", $host);
            } elseif ( $matches[1] == 'teoma' ) {
                $result = preg_match("/^crawler\d+\.ask\.com$/i", $host);
            } else {
                $result = true;
            }
            if ( $result ) {
                $result = false;
                foreach ( gethostbynamel($host) as $realip ) {
                    if ( $realip == $ip ){
                        $result = true;
                        break;
                    }
                }
            }

You also can try

http://referer.us/hide-http-referer.html
http://referer.us/spoof-http-referer.html

iglow · Dec 23, 2009

BadArticle said:

You also need reverse DNS check to filter out fake robots

Code:

        $result = false;
        $ip = $_SERVER['REMOTE_ADDR'];
        $ip12 = substr($ip, 0, 12);
        $ip9 = substr($ip, 0, 9);
        $ip8 = substr($ip, 0, 8);
        $ip7 = substr($ip, 0, 7);
        $ip6 = substr($ip, 0, 6);
        $ip5 = substr($ip, 0, 5);
        if (($ip7=='66.249.')||($ip6=='65.55.')||($ip6=='72.30.')||($ip5=='74.6.')||($ip12=='216.129.119.')) {
            $result = true;
        }
        if ( !$result && preg_match("/.*(googlebot|slurp|msnbot|teoma|twiceler).*/i", $agent, $matches) ) {
            $host = gethostbyaddr( $ip );
            if ( $matches[1] == 'googlebot' ) {
                $result = ( substr($host, -14) == '.googlebot.com' );
            } elseif ( $matches[1] == 'slurp' ) {
                $result = ( substr($host, -16) == '.crawl.yahoo.net' ) || ( substr($host, -15) == '.corp.yahoo.com' );
            } elseif ( $matches[1] == 'msnbot' ) {
                $result = preg_match("/.*\.search\.(msn|live|bing)\.com$/i", $host);
            } elseif ( $matches[1] == 'twiceler' ) {
                $result = preg_match("/.*\.(cuill|cuil)\.com$/i", $host);
            } elseif ( $matches[1] == 'teoma' ) {
                $result = preg_match("/^crawler\d+\.ask\.com$/i", $host);
            } else {
                $result = true;
            }
            if ( $result ) {
                $result = false;
                foreach ( gethostbynamel($host) as $realip ) {
                    if ( $realip == $ip ){
                        $result = true;
                        break;
                    }
                }
            }

You also can try

Hide HTTP Referer
Spoof HTTP Referer

is it possible to add to it some kind of ip verification from .txt file? i mean when somebody ivist it checks not only useragent but also list of ips from txt file [without txt the code will be too long and page will eb too slow].

AUTOATTACKER · Dec 23, 2009

Im looking for someone to setup dmr

please pm

Redirect browser traffic but allow SE spiders

christopher

Newbie

matapples01

Regular Member

jonnyh431

Junior Member

BadArticle

Junior Member

iglow

Elite Member

AUTOATTACKER

Registered Member

Main Menu

Marketplace

Making Money

BlackHat World