mainhoondon
Regular Member
- Jun 15, 2009
- 403
- 187
Now I get it . Smart
but the links to PBN are crawleable (if that matters)
Thanks
[HowTo] Hiding your PBN Like The Pros And Why Doing It Wrong = Deindexing
- Thread starter accelerator_dd
- Start date
Now I get it . Smart
but the links to PBN are crawleable (if that matters)
Thanks
accelerator_dd
Elite Member
- May 14, 2010
- 2,447
- 1,022
Now I get it . Smart
but the links to PBN are crawleable (if that matters)
Thanks
They are crawlable but that doesn't really matter. When a competitor enters your moneysite in ahrefs or any other checker, your PBN domains won't come up - that keeps you safe.
That's not true. Using your code there WILL be domains showing up in Ahrefs & Majestic (and any backlink checker for that matter). I'm not saying that all will, but there is a rather specific glitch that will make some of your domains showing up in those backlink checkers.
Funny how your OP title is so grotesque, while it's pretty obvious you haven't even tried your own code on a substantial amount of PBN domains.
eh....I dont really see the value in hiding your PBN.
as a matter of fact, I like to show it to frustrate my competitors. when competitors check your backlinks to try to copy you, they will say "dammit, it's a private network. I can't copy this" and they will fuck off.
as a matter of fact, I like to show it to frustrate my competitors. when competitors check your backlinks to try to copy you, they will say "dammit, it's a private network. I can't copy this" and they will fuck off.
clockhouse
Regular Member
- Feb 2, 2015
- 245
- 77
What if the person comes in and is like "lol ranking with lots of PBN links? okay no problem, easy to recreate." I'd rather hide what it takes to rank then let everyone see it.
accelerator_dd
Elite Member
- May 14, 2010
- 2,447
- 1,022
I have been using this method on my PBN and I rechecked all my domains today so I can say with certainty that none are showing up as backlinks for my money sites.
I also wanted to make sure I haven't overseen anything, so I tried crawling one of my PBN domains with Screaming Frog SEO a number of times to see how it would react against GoogleBot, Bing's bot, ahrefs, majestic, firefox, opera, safari, chrome and see if there are any anomalies I should be focusing on. The crawls with the search engine user-agents got the whole domain crawled, the link checker bots got the redirect, and the requests with an empty user-agent (or no user-agent at all) got a 403 error.
I set up a local proxy so I can intercept and change the requests with/without referrer, user agent/no user agent, differnt types of file requests, everything available to tweak a different response - everything worked like it was supposed to.
If you can suggest any scenario where this doesn't hold true, please, enlighten me.
They could neg seo your PBN, report those to Google, write emails to the linking sites to say the website they are linking to hsa changed - all of which will end up costing you money.
Thanks man!
Thanks Aty!
Exactly, I now do this as I install all my sites. If ahrefs/majestic has already indexed your links, you will first need to put a robots.txt file so the link checkers know to remove it from the index and then after it's removed, take down the robots file and do the edits to the htaccess file.
I'm definitely willing to help people on an individual basis, but I really don't feel comfortable sharing this publically.
I will say this since you mentioned Screaming Frog: crawl your domain with a non-blocked user agent. Save those URL's and recrawl those with a blocked user agent. Then it'll be obvious. I wouldn't share the fix though since there is nothing to gain from it for you personally.
accelerator_dd
Elite Member
- May 14, 2010
- 2,447
- 1,022
I'm definitely willing to help people on an individual basis, but I really don't feel comfortable sharing this publically.
I will say this since you mentioned Screaming Frog: crawl your domain with a non-blocked user agent. Save those URL's and recrawl those with a blocked user agent. Then it'll be obvious. I wouldn't share the fix though since there is nothing to gain from it for you personally.
Giving it a try now:
Here's the crawl with GoogleBot user-agent:
And here's one with ahrefsbot:
(all the way down all are 301ed, no exceptions)
Looks good to me
Giving it a try now:
Here's the crawl with GoogleBot user-agent:
(all pages on a default fresh installation including sample page and sample post with a sample comment give a 200 OK)
And here's one with ahrefsbot:
(all the way down all are 301ed, no exceptions)
Looks good to me
Hmm, try it on a site that's completely done, webhosted, has a normal URL structure and has content and pages on it. I just double checked and the code isn't working then. In fact I can PM you one of my testing domains so you can check it out.
Edit: http://i.imgur.com/nIl8U0C.png
Last edited:
- Aug 21, 2009
- 2,556
- 2,664
He's probably having the same issue I described above.
Try this code and then check again
SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
SetEnvIfNoCase User-Agent .*exabot.* bad_bot
SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
SetEnvIfNoCase User-Agent .*BlekkoBot.* bad_bot
SetEnvIfNoCase User-Agent .*Semrush.* bad_bot
SetEnvIfNoCase User-Agent .*BLEXBot.* bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>
Ahhh shhhh
Pretty sure Semrush's UA are SemrushBot & SemrushBot-SA though. Those were the ones I picked up when I last updated my code 3 months ago.
OP atleast give CCcarter some credits. For people who are interested in the longer version of this method search CCcarter hiding pbn on Google, you will find the longer version of it.
I tried this and till today the redirect is not showing in ahrefs, majestic, open site explore or other well known backlink analyzer tool.
I tried this and till today the redirect is not showing in ahrefs, majestic, open site explore or other well known backlink analyzer tool.
Professor88
Registered Member
- Jan 24, 2014
- 94
- 31
Nice one accelerator_dd. Great share!
KevinK
Junior Member
- Jul 24, 2008
- 156
- 87
I don't quite understand...
IF a spider or ahref, etc. crawls your site and
is malicious to ignore your robots file, et al.,
then why wouldn't they fake the referer? and
crawl your pbn's?
by the way, didn't like your comma asterisks,
,* in your code but you took them out so I'm
good on that.
I would add IP ranges to this and only allow
certain ranges, but there are still footprints
that can be looked at. All-in-all -- good share.
edit: while I am thinking about this... might add,
might also want to honey-pot your robots file.
IF a spider or ahref, etc. crawls your site and
is malicious to ignore your robots file, et al.,
then why wouldn't they fake the referer? and
crawl your pbn's?
by the way, didn't like your comma asterisks,
,* in your code but you took them out so I'm
good on that.
I would add IP ranges to this and only allow
certain ranges, but there are still footprints
that can be looked at. All-in-all -- good share.
edit: while I am thinking about this... might add,
might also want to honey-pot your robots file.
Last edited:
neverquitting
Regular Member
- Nov 10, 2014
- 410
- 327
So...for a dumdum like me, can someone use the findings from the whole thread conversation (specifically Dj_Co2's post) to make something I can copy/paste into .htaccess? I'd be eternally grateful.
globalahsan
Newbie
- Mar 20, 2015
- 7
- 0
A good thread for reference, bookmarked.