Anyone still blocking bots for PBNs in 2019?

tentimes · Oct 23, 2019

I've been using Spyder spanker for years but I think they stopped updating it so I'm looking for other options.

I've done the .htaccess route but sometimes it'll screw up the site and cause it to break or 404 posts.

I've tried botnemesis but that was failing to work at all.

Tried the EBN free plugin but that's creating a fatal error when I install it.

I know most of you will say, "but just create great looking sites and you're fine", yeah well thats fine and we do but it's about hiding these sites from competitors not Google or trying to pass a manual review.

Curious if anyone has any recent solutions to this issue.

Rj268 · Oct 23, 2019

.htaccess is your best bet.

tentimes · Oct 23, 2019

Rj268 said:
.htaccess is your best bet.

Do you know where I can get an updated code?

Rj268 · Oct 23, 2019

This may not be the most up to date list but here are 1,500

Code:

12soso
192.comagent
1noonbot
1on1searchbot
3de_search2
3d_search
3g bot
3gse
50.nu
a1 sitemap generator
a1 website download
a6-indexer
aasp
abachobot
abonti
abotemailsearch
aboundex
aboutusbot
accmonitor compliance server
accoon
achulkov.net page walker
acme.spider
acoonbot
acquia-crawler
activetouristbot
ad muncher
adamm bot
adbeat_bot
adminshop.com
advanced email extractor
aesop_com_spiderman
aespider
af knowledge now verity spider
aggregator:vocus
ah-ha.com crawler
ahrefs
ahrefsbot
aibot
aidu
aihitbot
aipbot
aisiid
aitcsrobot/1.1
ajsitemap
akamai-sitesnapshot
alexawebsearchplatform
alexfdownload
alexibot
alkalinebot
all acronyms bot
alpha search agent
amerla search bot
amfibibot
ampmppc.com
amznkassocbot
anemone
anonymous
anotherbot
answerbot
answerbus
answerchase prove
antbot
antibot
antisantyworm
antro.net
aonde-spider
aport
appengine-google
appid: s~stremor-crawler-
aqua_products
arabot
arachmo
arachnophilia
archive.org_bot
aria equalizer
arianna.libero.it
arikus_spider
art-online.com
artavisbot
artera
asaha search engine turkey
ask
aspider
aspseek
asterias
astrofind
athenusbot
atlocalbot
atomic_email_hunter
attach
attrakt
attributor
augurfind
auresys
autobaron crawler
autoemailspider
autowebdir
avsearch-
axfeedsbot
axonize-bot
ayna
b2w
backdoorbot
backrub
backstreet browser
backweb
baidu
bandit
batchftp
baypup
bdfetch
becomebot
becomejpbot
beetlebot
bender
besserscheitern-crawl
betabot
big brother
big data
bigado.com
bigcliquebot
bigfoot
biglotron
bilbo
bilgibetabot
bilgibot
bintellibot
bitlybot
bitvouseragent
bizbot003
bizbot04
bizworks retriever
black hole
black.hole
blackbird
blackmask.net search engine
blackwidow
bladder fusion
blaiz-bee
blexbot
blinkx
blitzbot
blog conversation project
blogmyway
blogpulselive
blogrefsbot
blogscope
blogslive
bloobybot
blowfish
blt
bnf.fr_bot
boaconstrictor
boardreader
boia-scan-agent
boia.org
boitho
boi_crawl_00
bookmark buddy bookmark checker
bookmark search tool
bosug
bot apoena
botalot
botrighthere
botswana
bottybot
bpbot
braintime_search
brokenlinkcheck.com
browseremulator
browsermob
bruinbot
bsearchr&d
bspider
btbot
btsearch
bubing
buddy
buibui
buildcms crawler
builtbottough
bullseye
bumblebee
bunnyslippers
buscadorclarin
buscaplus robi
butterfly
buyhawaiibot
buzzbot
byindia
byspider
byteserver
bzbot
c r a w l 3 r
cacheblaster
caddbot
cafi
camcrawler
camelstampede
canon-webrecord
careerbot
cataguru
catchbot
cazoodle
ccbot
ccgcrawl
ccubee
cd-preload
ce-preload
cegbfeieh
cerberian drtrs
cert figleafbot
cfetch
cfnetwork
chameleon
charlotte
check&get
checkbot
checklinks
cheesebot
chemiede-nodebot
cherrypicker
chilkat
chinaclaw
cipinetbot
cis455crawler
citeseerxbot
cizilla
clariabot
climate ark
climateark spider
clshttp
clushbot
coast scan engine
coast webmaster pro
coccoc
collapsarweb
collector
colocrossing
combine
connectsearch
conpilot
contentsmartz
contextad bot
contype
cookienet
coolbot
coolcheck
copernic
copier
copyrightcheck
core-project
cosmos
covario-ids
cowbot-
cowdog bot
crabbybot
[email protected]
crawler.kpricorn.org
crawler43.ejupiter.com
crawler4j
crawler@
crawler_for_infomine
crawly
crawl_application
creativecommons
crescent
cs-crawler
cse html validator
cshttpclient
cuasarbot
culsearch
curl
custo
cvaulev
cyberdog
cybernavi_webget
cyberpatrol sitecat webbot
cyberspyder
cydralspider
d1garabicengine
datacha0s
datafountains
dataparksearch
dataprovider.com
datascape robot
dataspearspiderbot
dataspider
dattatec.com
daumoa
dblbot
dcpbot
declumbot
deepindex
deepnet crawler
deeptrawl
dejan
del.icio.us-thumbnails
deltascan
delvubot
der gro§e bildersauger
der große bildersauger
deusu
dfs-fetch
diagem
diamond
dibot
didaxusbot
digext
digger
digi-rssbot
digitalarchivesbot
digout4u
diibot
dillo
dir_snatch.exe
disco
distilled-reputation-monitor
djangotraineebot
dkimrepbot
dmoz downloader
docomo
dof-verify
domaincrawler
domainscan
domainwatcher bot
dotbot
dotspotsbot
dow jones searchbot
download
doy
dragonfly
drip
drone
dtaagent
dtsearchspider
dumbot
dwaar
dxseeker
e-societyrobot
eah
earth platform indexer
earth science educator  robot
easydl
ebingbong
ec2linkfinder
ecairn-grabber
ecatch
echoosebot
edisterbot
edugovsearch
egothor
eidetica.com
eirgrabber
elblindo the blind bot
elisabot
ellerdalebot
email exractor
emailcollector
emailleach
emailsiphon
emailwolf
emeraldshield
empas_robot
enabot
endeca
enigmabot
enswer neuro bot
enter user-agent
entitycubebot
erocrawler
estylesearch
esyndicat bot
eurosoft-bot
evaal
eventware
everest-vulcan inc.
exabot
exactsearch
exactseek
exooba
exploder
express webpictures
extractor
eyenetie
ez-robot
ezooms
f-bot test pilot
factbot
fairad client
falcon
fast data search document retriever
fast esp
fast-search-engine
fastbot crawler
fastbot.de crawler
fatbot
favcollector
faviconizer
favorites sweeper
fdm
fdse robot
fedcontractorbot
fembot
fetch api request
fetch_ici
fgcrawler
filangy
filehound
findanisp.com_isp_finder
findlinks
findweb
firebat
firstgov.gov search
flaming attackbot
flamingo_searchengine
flashcapture
flashget
flickysearchbot
fluffy the spider
flunky
focused_crawler
followsite
foobot
fooooo_web_video_crawl
fopper
formulafinderbot
forschungsportal
francis
freewebmonitoring sitechecker
freshcrawler
freshdownload
freshlinks.exe
friendfeedbot
frodo.at
froggle
frontpage
froola bot
fr_crawler
fu-nbi
full_breadth_crawler
funnelback
furlbot
g10-bot
gaisbot
galaxybot
gazz
gbplugin
generate_infomine_category_classifiers
genevabot
geniebot
genieo
geomaxenginebot
geometabot
geonabot
geovisu
germcrawler
gethtmlcontents
getleft
getright
getsmart
geturl.rexx
getweb!
giant
gigablastopensource
gigabot
girafabot
gleamebot
gnome-vfs
go!zilla
go-ahead-got-it
go-http-client
goforit.com
goforitbot
gold crawler
goldfire server
golem
goodjelly
gordon-college-google-mini
goroam
goseebot
gotit
govbot
gpu p2p crawler
grabber
grabnet
grafula
grapefx
grapeshot
grbot
greenyogi
gromit
grub
gsa
gslfbot
gulliver
gulperbot
gurujibot
gvc business crawler
gvc crawler
gvc search bot
gvc web crawler
gvc weblink crawler
gvc world links
gvcbot.com
happyfunbot
harvest
hatena antenna
hawler
hcat
hclsreport-crawler
hd nutch agent
header_test_client
healia
[NC,OR]
#500 new rule
RewriteCond %{HTTP_USER_AGENT}
helix
here will be link to crawler site
heritrix
hiscan
hisoftware accmonitor server
hisoftware accverify
hitcrawler
hivabot
hloader
hmsebot
hmview
hoge
holmes
homepagesearch
hooblybot-image
hoowwwer
hostcrawler
hsft - link scanner
hsft - lvu scanner
hslide
ht://check
htdig
html link validator
htmlparser
httplib
httrack
huaweisymantecspider
hul-wax
humanlinks
hyperestraier
hyperix
iaarchiver-
ia_archiver
ibuena
icab
icds-ingestion
ichiro
icopyright conductor
ieautodiscovery
iecheck
ihwebchecker
iiitbot
iim_405
ilsebot
iltrovatore
image stripper
image sucker
image-fetcher
imagebot
imagefortress
imageshereimagesthereimageseverywhere
imagevisu
imds_monitor
imo-google-robot-intelink
inagist.com url crawler
indexer
industry cortex webcrawler
indy library
indylabs_marius
inelabot
inet32 ctrl
inetbot
info seeker
infolink
infomine
infonavirobot
informant
infoseek sidewinder
infotekies
infousabot
ingrid
inktomi
insightscollector
insightsworksbot
inspirebot
insumascout
intelix
intelliseek
interget
internet ninja
internet radio crawler
internetlinkagent
interseek
ioi
ip-web-crawler.com
ipadd bot
ipselonbot
ips-agent
iria
irlbot
iron33
isara
isearch
isilox
istellabot
its-learning crawler
iu_csci_b659_class_crawler
ivia
jadynave
java
jbot
jemmathetourist
jennybot
jetbot
jetbrains omea pro
jetcar
jim
jobo
jobspider_ba
joc
joedog
joyscapebot
jspyda
junut bot
justview
jyxobot
k.s.bot
kakclebot
kalooga
katatudo-spider
kbeta1
keepni web site monitor
kenjin.spider
keybot translation-search-machine
keywenbot
keyword density
keyword.density
kinjabot
kitenga-crawler-bot
kiwistatus
kmbot-
kmccrew bot search
knight
knowitall
knowledge engine
knowledge.com
koepabot
koninklijke
korniki
krowler
ksbot
kuloko-bot
kulturarw3
kummhttp
kurzor
kyluka crawl
l.webis
labhoo
labourunions411
lachesis
lament
lamerexterminator
lapozzbot
larbin
lbot
leaptag
leechftp
leechget
letscrawl.com
lexibot
lexxebot
lftp
libcrawl
libiviacore
libw
likse
linguee bot
link checker
link validator
linkalarm
linkbot
linkcheck by siteimprove.com
linkcheck scanner
linkchecker
linkdex.com
linkextractorpro
linklint
linklooker
linkman
links sql
linkscan
linksmanager.com_bot
linksweeper
linkwalker
link_checker
litefinder
litlrbot
little grabber at skanktale.com
livelapbot
lm harvester
lmqueuebot
lnspiderguy
loadtimebot
localcombot
locust
lolongbot
lookbot
lsearch
lssbot
lt scotland checklink
ltx71.com
lwp
lycos_spider
lydia entity spider
lynnbot
lytranslate
mag-net
magnet
magpie-crawler
magus bot
mail.ru
mainseek_bot
mammoth
map robot
markwatch
masagool
masidani_bot_
mass downloader
mata hari
mata.hari
matentzn at cs dot man dot ac dot uk
maxamine.com--robot
maxamine.com-robot
maxomobot
mcbot
medrabbit
megite
memacbot
memo
mendeleybot
mercator-
mercuryboard_user_agent_sql_injection.nasl
metacarta
metaeuro web search
metager2
metagloss
metal crawler
metaquerier
metaspider
metaspinner
metauri
mfcrawler
mfhttpscan
midown tool
miixpc
mini-robot
minibot
minirank
mirror
missigua locator
mister pix
mister.pix
miva
mj12bot
mnogosearch
moduna.com
mod_accessibility
moget
mojeekbot
monkeycrawl
moses
mowserbot
mqbot
mse360
msindianwebcrawl
msmobot
msnptc
msrbot
mt-soft
multitext
my-heritrix-crawler
myapp
mycompanybot
mycrawler
myengines-us-bot
myfamilybot
myra
my_little_searchengine_project
nabot
najdi.si
nambu
nameprotect
nasa search
natchcvs
natweb-bad-link-mailer
naver
navroad
nearsite
nec-meshexplorer
neosciocrawler
nerdbynature.bot
nerdybot
nerima-crawl-
nessus
nestreader
net vampire
net::trackback
netants
netcarta cyberpilot pro
netcraft
netexperts
netid.com bot
netmechanic
netprospector
netresearchserver
netseer
netshift=
netsongbot
netsparker
netspider
netsrcherp
netzip
newmedhunt
news bot
newsgatherer
newsgroupreporter
newstrovebot
news_search_app
nextgensearchbot
nextthing.org
nicebot
nicerspro
niki-bot
nimblecrawler
nimbus-1
ninetowns
ninja
njuicebot
nlese
nogate
norbert the spider
noteworthybot
npbot
nrcan intranet crawler
nsdl_search_bot
nuggetize.com bot
nusearch spider
nutch
nu_tch
nwspider
nymesis
nys-crawler
objectssearch
obot
obvius external linkcheck
ocelli
octopus
odp entries t_st
oegp
offline navigator
offline.explorer
ogspider
omiexplorer_bot
omniexplorer
omnifind
omniweb
onetszukaj
online link validator
oozbot
openbot
openfind
openintelligencedata
openisearch
openlink virtuoso rdf crawler
opensearchserver_bot
opidig
optidiscover
oracle secure enterprise search
oracle ultra search
orangebot
orisbot
ornl_crawler
ornl_mercury
osis-project.jp
oso
outfoxbot
outfoxmelonbot
owler-bot
owsbot
ozelot
p3p client
pagebiteshyperbot
pagebull
pagedown
pagefetcher
pagegrabber
pagepeeker
pagerank monitor
page_verifier
pamsnbot.htm
panopy bot
panscient.com
pansophica
papa foto
paperlibot
parasite
parsijoo
pathtraq
pattern
patwebbot
pavuk
paxleframework
pbbot
pcbrowser
pcore-http
pd-crawler
penthesila
perform_crawl
perman
personal ultimate crawler
php version tracker
phpcrawl
phpdig
picosearch
pieno robot
pipbot
pipeliner
pita
pixfinder
piyushbot
planetwork bot search
plucker
plukkie
plumtree
pockey
pocohttp
pogodak.ba
pogodak.co.yu
poirot
polybot
pompos
poodle predictor
popscreenbot
postpost
privacyfinder
projectwf-java-test-crawler
propowerbot
prowebwalker
proxem websearch
proximic
proxy crawler
psbot
pss-bot
psycheclone
pub-crawler
pucl
pulsebot
pump
pwebot
python
qeavis agent
qfkbot
qualidade
qualidator.com bot
quepasacreep
queryn metasearch
queryn.metasearch
quest.durato
quintura-crw
qunarbot
qwantify
qweerybot
qweery_robot.txt_checkbot
r2ibot
r6_commentreader
r6_feedfetcher
r6_votereader
rabot
radian6
radiation retriever
rampybot
rankivabot
rankur
rational sitecheck
rcstartbot
realdownload
reaper
rebi-shoveler
recorder
redbot
redcarpet
reget
repomonkey
research robot
riddler
riight
risenetbot
riverglassscanner
[NC,OR]
#1000 new rule
RewriteCond %{HTTP_USER_AGENT}
robopal
robosourcer
robotek
robozilla
roger
rome client
rondello
rotondo
roverbot
rpt-httpclient
rtgibot
rufusbot
runnk online rss reader
runnk rss aggregator
s2bot
safaribookmarkchecker
safednsbot
safetynet robot
saladspoon
sapienti
sapphireweb
sbider
sbl-bot
scfcrawler
scich
scientificcommons.org
scollspider
scooperbot
scooter
scoutjet
scrapebox
scrapy
scrawltest
screaming frog
scrubby
scspider
scumbot
search publisher
search x-bot
search-channel
search-engine-studio
search.kumkie.com
search.updated.com
search.usgs.gov
searcharoo.net
searchblox
searchbot
searchengine
searchhippo.com
searchit-bot
searchmarking
searchmarks
searchmee!
searchmee_v
searchmining
searchnowbot
searchpreview
searchspider.com
searqubot
seb spider
seekbot
seeker.lookseek.com
seeqbot
seeqpod-vertical-crawler
selflinkchecker
semager
semanticdiscovery
semantifire
semisearch
semrushbot
seoengworldbot
seokicks
seznambot
shablastbot
shadowwebanalyzer
shareaza
shelob
sherlock
shim-crawler
shopsalad
shopwiki
showlinks
showyoubot
siclab
silk
simplepie
siphon
sitebot
sitecheck
sitefinder
siteguardbot
siteorbiter
sitesnagger
sitesucker
sitesweeper
sitexpert
skimbot
skimwordsbot
skreemrbot
skywalker
sleipnir
slow-crawler
slysearch
smart-crawler
smartdownload
smarte bot
smartwit.com
snake
snap.com beta crawler
snapbot
snappreviewbot
snappy
snookit
snooper
snoopy
societyrobot
socscibot
soft411 directory
sogou
sohu agent
sohu-search
sokitomi crawl
solbot
sondeur
sootle
sosospider
space bison
space fung
spacebison
spankbot
spanner
spatineo monitor controller
spatineo serval controller
spatineo serval getmapbot
special_archiver
speedy
sphere scout
sphider
spider.terranautic.net
spiderengine
spiderku
spiderman
spinn3r
spinne
sportcrew-bot
sproose
spyder3.microsys.com
sq webscanner
sqlmap
squid-prefetch
squidclamav_redirector
sqworm
srevbot
sslbot
ssm agent
stackrambler
stardownloader
statbot
statcrawler
statedept-crawler
steeler
stegmann-bot
stero
stripper
stumbler
suchclip
sucker
sumeetbot
sumitbot
summizebot
summizefeedreader
sunrise xp
superbot
superhttp
superlumin downloader
superpagesbot
supremesearch.net
supybot
surdotlybot
surf
surveybot
suzuran
swebot
swish-e
sygolbot
synapticwalker
syntryx ant scout chassis pheromone
systemsearch-robot
szukacz
s~stremor-crawler
t-h-u-n-d-e-r-s-t-o-n-e
tailrank
takeout
talkro web-shot
tamu_crawler
tapuzbot
tarantula
targetblaster.com
targetyournews.com bot
tausdatabot
taxinomiabot
teamsoft wininet component
tecomi bot
teezirbot
teleport
telesoft
teradex mapper
teragram_crawler
terrawizbot
testbot
testing of bot
textbot
thatrobotsite.com
the dyslexalizer
the intraformant
the.intraformant
thenomad
theophrastus
theusefulbot
thumbbot
thumbnail.cz robot
thumbshots-de-bot
tigerbot
tighttwatbot
tineye
titan
to-dress_ru_bot_
to-night-bot
tocrawl
topicalizer
topicblogs
toplistbot
topserver php
topyx-crawler
touche
tourlentascanner
tpsystem
traazi
transgenikbot
travel-search
travelbot
travellazerbot
treezy
trendiction
trex
tridentspider
trovator
true_robot
tscholarsbot
tsm translation-search-machine
tswebbot
tulipchain
turingos
turnitinbot
tutorgigbot
tweetedtimes bot
tweetmemebot
twengabot
twice
twikle
twinuffbot
twisted pagegetter
twitturls
twitturly
tygobot
tygoprowler
typhoeus
u.s. government printing office
uberbot
ucb-nutch
udmsearch
ufam-crawler-
ultraseek
unchaos
unisterbot
unidentified
unitek uniengine
universalsearch
unwindfetchor
uoftdb_experiment
updated
url control
url-checker
urlappendbot
urlblaze
urlchecker
urlck
urldispatcher
urlspiderpro
urly warning
urly.warning
url_gather
usaf afkn k2spider
usasearch
uss-cosmix
usyd-nlp-spider
vacobot
vacuum
vadixbot
vagabondo
validator
valkyrie
vbseo
vci webviewer vci webviewer win32
verbstarbot
vericitecrawler
verifactrola
verity-url-gateway
vermut
versus crawler
versus.integis.ch
viasarchivinginformation.html
vipr
virus-detector
virus_detector
visbot
vishal for clia
visweb
vital search'n urchin
vlad
vlsearch
voilabot
vmbot
vocusbot
voideye
voil
vortex
voyager
vspider
w3c-webcon
w3c_unicorn
w3search
wacbot
wanadoo
wastrix
water conserve portal
water conserve spider
watzbot
wauuu
wavefire
waypath
wazzup
wbdbot
web ceo online robot
web crawler
web downloader
web image collector
web link validator
web magnet
web site downloader
web sucker
web-agent
web-sniffer
web.image.collector
webaltbot
webauto
webbot
webbul-bot
webcapture
webcheck
webclipping.com
webcollage
webcopier
webcopy
webcorp
webcrawl.net
webcrawler
webdatacentrebot
webdownloader for x
webdup
webemailextrac
webenhancer
webfetch
webgather
webgo is
webgobbler
webimages
webinator-search2
webinator-wbi
webindex
weblayers
webleacher
weblexbot
weblinker
weblyzard
webmastercoffee
webmasterworld extractor
webmasterworldforumbot
webminer
webmoose
webot
webpix
webreaper
webripper
websauger
webscan
websearchbench
website
webspear
websphinx
webspider
webster
webstripper
webtrafficexpress
webtrends link analyzer
webvac
webwalk
webwasher
webwatch
webwhacker
webxm
webzip
weddings.info
wenbin
wep search
wepa
werelatebot
wget
whacker
whirlpool web engine
whowhere robot
widow
wikiabot
wikio
wikiwix-bot-
winhttp
wire
wisebot
wisenutbot
wish-la
wish-project
wisponbot
wmcai-robot
wminer
wmsbot
woriobot
worldshop
worqmada
wotbox
wume_crawler
www collector
www-collector-e
www-mechanize
wwwoffle
wwwrobot
wwwster
wwwwanderer
wwwxref
wysigot
x-clawler
x-crawler
xaldon
xenu
xerka metabot
xerka webbot
xget
xirq
xmarksfetch
xqrobot
y!j
yacy.net
yacybot
yanga worldsearch bot
yarienavoir.net
yasaklibot
yats crawler
ybot
yebolbot
yellowjacket
yeti
yolinkbot
yooglifetchagent
yoono
yottacars_bot
yourls
z-add link checker
zagrebin
zao
zedzo.validate
zermelo
zeus
zibber-v
zimeno
zing-bottabot
zipppbot
zongbot
zoomspider
zotag search
zsebot
zuibot
zyborg
zyte

Or Just use this code

Code:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(12soso|192.comagent|1noonbot|1on1searchbot|3de_search2|3d_search|3g bot|3gse|50.nu|a1 sitemap generator|a1 website download|a6-indexer|aasp|abachobot|abonti|abotemailsearch|aboundex|aboutusbot|accmonitor compliance server|accoon|achulkov.net page walker|acme.spider|acoonbot|acquia-crawler|activetouristbot|ad muncher|adamm bot|adbeat_bot|adminshop.com|advanced email extractor|aesop_com_spiderman|aespider|af knowledge now verity spider|aggregator:vocus|ah-ha.com crawler|ahrefs|ahrefsbot|aibot|aidu|aihitbot|aipbot|aisiid|aitcsrobot/1.1|ajsitemap|akamai-sitesnapshot|alexawebsearchplatform|alexfdownload|alexibot|alkalinebot|all acronyms bot|alpha search agent|amerla search bot|amfibibot|ampmppc.com|amznkassocbot|anemone|anonymous|anotherbot|answerbot|answerbus|answerchase prove|antbot|antibot|antisantyworm|antro.net|aonde-spider|aport|appengine-google|appid: s~stremor-crawler-|aqua_products|arabot|arachmo|arachnophilia|archive.org_bot|aria equalizer|arianna.libero.it|arikus_spider|art-online.com|artavisbot|artera|asaha search engine turkey|ask|aspider|aspseek|asterias|astrofind|athenusbot|atlocalbot|atomic_email_hunter|attach|attrakt|attributor|augurfind|auresys|autobaron crawler|autoemailspider|autowebdir|avsearch-|axfeedsbot|axonize-bot|ayna|b2w|backdoorbot|backrub|backstreet browser|backweb|baidu|bandit|batchftp|baypup|bdfetch|becomebot|becomejpbot|beetlebot|bender|besserscheitern-crawl|betabot|big brother|big data|bigado.com|bigcliquebot|bigfoot|biglotron|bilbo|bilgibetabot|bilgibot|bintellibot|bitlybot|bitvouseragent|bizbot003|bizbot04|bizworks retriever|black hole|black.hole|blackbird|blackmask.net search engine|blackwidow|bladder fusion|blaiz-bee|blexbot|blinkx|blitzbot|blog conversation project|blogmyway|blogpulselive|blogrefsbot|blogscope|blogslive|bloobybot|blowfish|blt|bnf.fr_bot|boaconstrictor|boardreader|boia-scan-agent|boia.org|boitho|boi_crawl_00|bookmark buddy bookmark checker|bookmark search tool|bosug|bot apoena|botalot|botrighthere|botswana|bottybot|bpbot|braintime_search|brokenlinkcheck.com|browseremulator|browsermob|bruinbot|bsearchr&d|bspider|btbot|btsearch|bubing|buddy|buibui|buildcms crawler|builtbottough|bullseye|bumblebee|bunnyslippers|buscadorclarin|buscaplus robi|butterfly|buyhawaiibot|buzzbot|byindia|byspider|byteserver|bzbot|c r a w l 3 r|cacheblaster|caddbot|cafi|camcrawler|camelstampede|canon-webrecord|careerbot|cataguru|catchbot|cazoodle|ccbot|ccgcrawl|ccubee|cd-preload|ce-preload|cegbfeieh|cerberian drtrs|cert figleafbot|cfetch|cfnetwork|chameleon|charlotte|check&get|checkbot|checklinks|cheesebot|chemiede-nodebot|cherrypicker|chilkat|chinaclaw|cipinetbot|cis455crawler|citeseerxbot|cizilla|clariabot|climate ark|climateark spider|clshttp|clushbot|coast scan engine|coast webmaster pro|coccoc|collapsarweb|collector|colocrossing|combine|connectsearch|conpilot|contentsmartz|contextad bot|contype|cookienet|coolbot|coolcheck|copernic|copier|copyrightcheck|core-project|cosmos|covario-ids|cowbot-|cowdog bot|crabbybot|[email protected]|crawler.kpricorn.org|crawler43.ejupiter.com|crawler4j|crawler@|crawler_for_infomine|crawly|crawl_application|creativecommons|crescent|cs-crawler|cse html validator|cshttpclient|cuasarbot|culsearch|curl|custo|cvaulev|cyberdog|cybernavi_webget|cyberpatrol sitecat webbot|cyberspyder|cydralspider|d1garabicengine|datacha0s|datafountains|dataparksearch|dataprovider.com|datascape robot|dataspearspiderbot|dataspider|dattatec.com|daumoa|dblbot|dcpbot|declumbot|deepindex|deepnet crawler|deeptrawl|dejan|del.icio.us-thumbnails|deltascan|delvubot|der gro§e bildersauger|der große bildersauger|deusu|dfs-fetch|diagem|diamond|dibot|didaxusbot|digext|digger|digi-rssbot|digitalarchivesbot|digout4u|diibot|dillo|dir_snatch.exe|disco|distilled-reputation-monitor|djangotraineebot|dkimrepbot|dmoz downloader|docomo|dof-verify|domaincrawler|domainscan|domainwatcher bot|dotbot|dotspotsbot|dow jones searchbot|download|doy|dragonfly|drip|drone|dtaagent|dtsearchspider|dumbot|dwaar|dxseeker|e-societyrobot|eah|earth platform indexer|earth science educator  robot|easydl|ebingbong|ec2linkfinder|ecairn-grabber|ecatch|echoosebot|edisterbot|edugovsearch|egothor|eidetica.com|eirgrabber|elblindo the blind bot|elisabot|ellerdalebot|email exractor|emailcollector|emailleach|emailsiphon|emailwolf|emeraldshield|empas_robot|enabot|endeca|enigmabot|enswer neuro bot|enter user-agent|entitycubebot|erocrawler|estylesearch|esyndicat bot|eurosoft-bot|evaal|eventware|everest-vulcan inc.|exabot|exactsearch|exactseek|exooba|exploder|express webpictures|extractor|eyenetie|ez-robot|ezooms|f-bot test pilot|factbot|fairad client|falcon|fast data search document retriever|fast esp|fast-search-engine|fastbot crawler|fastbot.de crawler|fatbot|favcollector|faviconizer|favorites sweeper|fdm|fdse robot|fedcontractorbot|fembot|fetch api request|fetch_ici|fgcrawler|filangy|filehound|findanisp.com_isp_finder|findlinks|findweb|firebat|firstgov.gov search|flaming attackbot|flamingo_searchengine|flashcapture|flashget|flickysearchbot|fluffy the spider|flunky|focused_crawler|followsite|foobot|fooooo_web_video_crawl|fopper|formulafinderbot|forschungsportal|francis|freewebmonitoring sitechecker|freshcrawler|freshdownload|freshlinks.exe|friendfeedbot|frodo.at|froggle|frontpage|froola bot|fr_crawler|fu-nbi|full_breadth_crawler|funnelback|furlbot|g10-bot|gaisbot|galaxybot|gazz|gbplugin|generate_infomine_category_classifiers|genevabot|geniebot|genieo|geomaxenginebot|geometabot|geonabot|geovisu|germcrawler|gethtmlcontents|getleft|getright|getsmart|geturl.rexx|getweb!|giant|gigablastopensource|gigabot|girafabot|gleamebot|gnome-vfs|go!zilla|go-ahead-got-it|go-http-client|goforit.com|goforitbot|gold crawler|goldfire server|golem|goodjelly|gordon-college-google-mini|goroam|goseebot|gotit|govbot|gpu p2p crawler|grabber|grabnet|grafula|grapefx|grapeshot|grbot|greenyogi|gromit|grub|gsa|gslfbot|gulliver|gulperbot|gurujibot|gvc business crawler|gvc crawler|gvc search bot|gvc web crawler|gvc weblink crawler|gvc world links|gvcbot.com|happyfunbot|harvest|hatena antenna|hawler|hcat|hclsreport-crawler|hd nutch agent|header_test_client|healia|[NC,OR]|#500 new rule|RewriteCond %{HTTP_USER_AGENT}|helix|here will be link to crawler site|heritrix|hiscan|hisoftware accmonitor server|hisoftware accverify|hitcrawler|hivabot|hloader|hmsebot|hmview|hoge|holmes|homepagesearch|hooblybot-image|hoowwwer|hostcrawler|hsft - link scanner|hsft - lvu scanner|hslide|ht://check|htdig|html link validator|htmlparser|httplib|httrack|huaweisymantecspider|hul-wax|humanlinks|hyperestraier|hyperix|iaarchiver-|ia_archiver|ibuena|icab|icds-ingestion|ichiro|icopyright conductor|ieautodiscovery|iecheck|ihwebchecker|iiitbot|iim_405|ilsebot|iltrovatore|image stripper|image sucker|image-fetcher|imagebot|imagefortress|imageshereimagesthereimageseverywhere|imagevisu|imds_monitor|imo-google-robot-intelink|inagist.com url crawler|indexer|industry cortex webcrawler|indy library|indylabs_marius|inelabot|inet32 ctrl|inetbot|info seeker|infolink|infomine|infonavirobot|informant|infoseek sidewinder|infotekies|infousabot|ingrid|inktomi|insightscollector|insightsworksbot|inspirebot|insumascout|intelix|intelliseek|interget|internet ninja|internet radio crawler|internetlinkagent|interseek|ioi|ip-web-crawler.com|ipadd bot|ipselonbot|ips-agent|iria|irlbot|iron33|isara|isearch|isilox|istellabot|its-learning crawler|iu_csci_b659_class_crawler|ivia|jadynave|java|jbot|jemmathetourist|jennybot|jetbot|jetbrains omea pro|jetcar|jim|jobo|jobspider_ba|joc|joedog|joyscapebot|jspyda|junut bot|justview|jyxobot|k.s.bot|kakclebot|kalooga|katatudo-spider|kbeta1|keepni web site monitor|kenjin.spider|keybot translation-search-machine|keywenbot|keyword density|keyword.density|kinjabot|kitenga-crawler-bot|kiwistatus|kmbot-|kmccrew bot search|knight|knowitall|knowledge engine|knowledge.com|koepabot|koninklijke|korniki|krowler|ksbot|kuloko-bot|kulturarw3|kummhttp|kurzor|kyluka crawl|l.webis|labhoo|labourunions411|lachesis|lament|lamerexterminator|lapozzbot|larbin|lbot|leaptag|leechftp|leechget|letscrawl.com|lexibot|lexxebot|lftp|libcrawl|libiviacore|libw|likse|linguee bot|link checker|link validator|linkalarm|linkbot|linkcheck by siteimprove.com|linkcheck scanner|linkchecker|linkdex.com|linkextractorpro|linklint|linklooker|linkman|links sql|linkscan|linksmanager.com_bot|linksweeper|linkwalker|link_checker|litefinder|litlrbot|little grabber at skanktale.com|livelapbot|lm harvester|lmqueuebot|lnspiderguy|loadtimebot|localcombot|locust|lolongbot|lookbot|lsearch|lssbot|lt scotland checklink|ltx71.com|lwp|lycos_spider|lydia entity spider|lynnbot|lytranslate|mag-net|magnet|magpie-crawler|magus bot|mail.ru|mainseek_bot|mammoth|map robot|markwatch|masagool|masidani_bot_|mass downloader|mata hari|mata.hari|matentzn at cs dot man dot ac dot uk|maxamine.com--robot|maxamine.com-robot|maxomobot|mcbot|medrabbit|megite|memacbot|memo|mendeleybot|mercator-|mercuryboard_user_agent_sql_injection.nasl|metacarta|metaeuro web search|metager2|metagloss|metal crawler|metaquerier|metaspider|metaspinner|metauri|mfcrawler|mfhttpscan|midown tool|miixpc|mini-robot|minibot|minirank|mirror|missigua locator|mister pix|mister.pix|miva|mj12bot|mnogosearch|moduna.com|mod_accessibility|moget|mojeekbot|monkeycrawl|moses|mowserbot|mqbot|mse360|msindianwebcrawl|msmobot|msnptc|msrbot|mt-soft|multitext|my-heritrix-crawler|myapp|mycompanybot|mycrawler|myengines-us-bot|myfamilybot|myra|my_little_searchengine_project|nabot|najdi.si|nambu|nameprotect|nasa search|natchcvs|natweb-bad-link-mailer|naver|navroad|nearsite|nec-meshexplorer|neosciocrawler|nerdbynature.bot|nerdybot|nerima-crawl-|nessus|nestreader|net vampire|net::trackback|netants|netcarta cyberpilot pro|netcraft|netexperts|netid.com bot|netmechanic|netprospector|netresearchserver|netseer|netshift=|netsongbot|netsparker|netspider|netsrcherp|netzip|newmedhunt|news bot|newsgatherer|newsgroupreporter|newstrovebot|news_search_app|nextgensearchbot|nextthing.org|nicebot|nicerspro|niki-bot|nimblecrawler|nimbus-1|ninetowns|ninja|njuicebot|nlese|nogate|norbert the spider|noteworthybot|npbot|nrcan intranet crawler|nsdl_search_bot|nuggetize.com bot|nusearch spider|nutch|nu_tch|nwspider|nymesis|nys-crawler|objectssearch|obot|obvius external linkcheck|ocelli|octopus|odp entries t_st|oegp|offline navigator|offline.explorer|ogspider|omiexplorer_bot|omniexplorer|omnifind|omniweb|onetszukaj|online link validator|oozbot|openbot|openfind|openintelligencedata|openisearch|openlink virtuoso rdf crawler|opensearchserver_bot|opidig|optidiscover|oracle secure enterprise search|oracle ultra search|orangebot|orisbot|ornl_crawler|ornl_mercury|osis-project.jp|oso|outfoxbot|outfoxmelonbot|owler-bot|owsbot|ozelot|p3p client|pagebiteshyperbot|pagebull|pagedown|pagefetcher|pagegrabber|pagepeeker|pagerank monitor|page_verifier|pamsnbot.htm|panopy bot|panscient.com|pansophica|papa foto|paperlibot|parasite|parsijoo|pathtraq|pattern|patwebbot|pavuk|paxleframework|pbbot|pcbrowser|pcore-http|pd-crawler|penthesila|perform_crawl|perman|personal ultimate crawler|php version tracker|phpcrawl|phpdig|picosearch|pieno robot|pipbot|pipeliner|pita|pixfinder|piyushbot|planetwork bot search|plucker|plukkie|plumtree|pockey|pocohttp|pogodak.ba|pogodak.co.yu|poirot|polybot|pompos|poodle predictor|popscreenbot|postpost|privacyfinder|projectwf-java-test-crawler|propowerbot|prowebwalker|proxem websearch|proximic|proxy crawler|psbot|pss-bot|psycheclone|pub-crawler|pucl|pulsebot|pump|pwebot|python|qeavis agent|qfkbot|qualidade|qualidator.com bot|quepasacreep|queryn metasearch|queryn.metasearch|quest.durato|quintura-crw|qunarbot|qwantify|qweerybot|qweery_robot.txt_checkbot|r2ibot|r6_commentreader|r6_feedfetcher|r6_votereader|rabot|radian6|radiation retriever|rampybot|rankivabot|rankur|rational sitecheck|rcstartbot|realdownload|reaper|rebi-shoveler|recorder|redbot|redcarpet|reget|repomonkey|research robot|riddler|riight|risenetbot|riverglassscanner|[NC,OR]|#1000 new rule|RewriteCond %{HTTP_USER_AGENT}|robopal|robosourcer|robotek|robozilla|roger|rome client|rondello|rotondo|roverbot|rpt-httpclient|rtgibot|rufusbot|runnk online rss reader|runnk rss aggregator|s2bot|safaribookmarkchecker|safednsbot|safetynet robot|saladspoon|sapienti|sapphireweb|sbider|sbl-bot|scfcrawler|scich|scientificcommons.org|scollspider|scooperbot|scooter|scoutjet|scrapebox|scrapy|scrawltest|screaming frog|scrubby|scspider|scumbot|search publisher|search x-bot|search-channel|search-engine-studio|search.kumkie.com|search.updated.com|search.usgs.gov|searcharoo.net|searchblox|searchbot|searchengine|searchhippo.com|searchit-bot|searchmarking|searchmarks|searchmee!|searchmee_v|searchmining|searchnowbot|searchpreview|searchspider.com|searqubot|seb spider|seekbot|seeker.lookseek.com|seeqbot|seeqpod-vertical-crawler|selflinkchecker|semager|semanticdiscovery|semantifire|semisearch|semrushbot|seoengworldbot|seokicks|seznambot|shablastbot|shadowwebanalyzer|shareaza|shelob|sherlock|shim-crawler|shopsalad|shopwiki|showlinks|showyoubot|siclab|silk|simplepie|siphon|sitebot|sitecheck|sitefinder|siteguardbot|siteorbiter|sitesnagger|sitesucker|sitesweeper|sitexpert|skimbot|skimwordsbot|skreemrbot|skywalker|sleipnir|slow-crawler|slysearch|smart-crawler|smartdownload|smarte bot|smartwit.com|snake|snap.com beta crawler|snapbot|snappreviewbot|snappy|snookit|snooper|snoopy|societyrobot|socscibot|soft411 directory|sogou|sohu agent|sohu-search|sokitomi crawl|solbot|sondeur|sootle|sosospider|space bison|space fung|spacebison|spankbot|spanner|spatineo monitor controller|spatineo serval controller|spatineo serval getmapbot|special_archiver|speedy|sphere scout|sphider|spider.terranautic.net|spiderengine|spiderku|spiderman|spinn3r|spinne|sportcrew-bot|sproose|spyder3.microsys.com|sq webscanner|sqlmap|squid-prefetch|squidclamav_redirector|sqworm|srevbot|sslbot|ssm agent|stackrambler|stardownloader|statbot|statcrawler|statedept-crawler|steeler|stegmann-bot|stero|stripper|stumbler|suchclip|sucker|sumeetbot|sumitbot|summizebot|summizefeedreader|sunrise xp|superbot|superhttp|superlumin downloader|superpagesbot|supremesearch.net|supybot|surdotlybot|surf|surveybot|suzuran|swebot|swish-e|sygolbot|synapticwalker|syntryx ant scout chassis pheromone|systemsearch-robot|szukacz|s~stremor-crawler|t-h-u-n-d-e-r-s-t-o-n-e|tailrank|takeout|talkro web-shot|tamu_crawler|tapuzbot|tarantula|targetblaster.com|targetyournews.com bot|tausdatabot|taxinomiabot|teamsoft wininet component|tecomi bot|teezirbot|teleport|telesoft|teradex mapper|teragram_crawler|terrawizbot|testbot|testing of bot|textbot|thatrobotsite.com|the dyslexalizer|the intraformant|the.intraformant|thenomad|theophrastus|theusefulbot|thumbbot|thumbnail.cz robot|thumbshots-de-bot|tigerbot|tighttwatbot|tineye|titan|to-dress_ru_bot_|to-night-bot|tocrawl|topicalizer|topicblogs|toplistbot|topserver php|topyx-crawler|touche|tourlentascanner|tpsystem|traazi|transgenikbot|travel-search|travelbot|travellazerbot|treezy|trendiction|trex|tridentspider|trovator|true_robot|tscholarsbot|tsm translation-search-machine|tswebbot|tulipchain|turingos|turnitinbot|tutorgigbot|tweetedtimes bot|tweetmemebot|twengabot|twice|twikle|twinuffbot|twisted pagegetter|twitturls|twitturly|tygobot|tygoprowler|typhoeus|u.s. government printing office|uberbot|ucb-nutch|udmsearch|ufam-crawler-|ultraseek|unchaos|unisterbot|unidentified|unitek uniengine|universalsearch|unwindfetchor|uoftdb_experiment|updated|url control|url-checker|urlappendbot|urlblaze|urlchecker|urlck|urldispatcher|urlspiderpro|urly warning|urly.warning|url_gather|usaf afkn k2spider|usasearch|uss-cosmix|usyd-nlp-spider|vacobot|vacuum|vadixbot|vagabondo|validator|valkyrie|vbseo|vci webviewer vci webviewer win32|verbstarbot|vericitecrawler|verifactrola|verity-url-gateway|vermut|versus crawler|versus.integis.ch|viasarchivinginformation.html|vipr|virus-detector|virus_detector|visbot|vishal for clia|visweb|vital search'n urchin|vlad|vlsearch|voilabot|vmbot|vocusbot|voideye|voil|vortex|voyager|vspider|w3c-webcon|w3c_unicorn|w3search|wacbot|wanadoo|wastrix|water conserve portal|water conserve spider|watzbot|wauuu|wavefire|waypath|wazzup|wbdbot|web ceo online robot|web crawler|web downloader|web image collector|web link validator|web magnet|web site downloader|web sucker|web-agent|web-sniffer|web.image.collector|webaltbot|webauto|webbot|webbul-bot|webcapture|webcheck|webclipping.com|webcollage|webcopier|webcopy|webcorp|webcrawl.net|webcrawler|webdatacentrebot|webdownloader for x|webdup|webemailextrac|webenhancer|webfetch|webgather|webgo is|webgobbler|webimages|webinator-search2|webinator-wbi|webindex|weblayers|webleacher|weblexbot|weblinker|weblyzard|webmastercoffee|webmasterworld extractor|webmasterworldforumbot|webminer|webmoose|webot|webpix|webreaper|webripper|websauger|webscan|websearchbench|website|webspear|websphinx|webspider|webster|webstripper|webtrafficexpress|webtrends link analyzer|webvac|webwalk|webwasher|webwatch|webwhacker|webxm|webzip|weddings.info|wenbin|wep search|wepa|werelatebot|wget|whacker|whirlpool web engine|whowhere robot|widow|wikiabot|wikio|wikiwix-bot-|winhttp|wire|wisebot|wisenutbot|wish-la|wish-project|wisponbot|wmcai-robot|wminer|wmsbot|woriobot|worldshop|worqmada|wotbox|wume_crawler|www collector|www-collector-e|www-mechanize|wwwoffle|wwwrobot|wwwster|wwwwanderer|wwwxref|wysigot|x-clawler|x-crawler|xaldon|xenu|xerka metabot|xerka webbot|xget|xirq|xmarksfetch|xqrobot|y!j|yacy.net|yacybot|yanga worldsearch bot|yarienavoir.net|yasaklibot|yats crawler|ybot|yebolbot|yellowjacket|yeti|yolinkbot|yooglifetchagent|yoono|yottacars_bot|yourls|z-add link checker|zagrebin|zao|zedzo.validate|zermelo|zeus|zibber-v|zimeno|zing-bottabot|zipppbot|zongbot|zoomspider|zotag search|zsebot|zuibot|zyborg|zyte).*$ [NC]
RewriteRule .* - [F,L]

thunderhd · Oct 23, 2019

I dont block bots. you are getting links from all sites whose bots blocked . lol its a huge footprint itself

Rj268 · Oct 23, 2019

thunderhd said:
I dont block bots. you are getting links from all sites whose bots blocked . lol its a huge footprint itself

It's is far more likely that a competitor will find your PBN and report it than Google finding out. No footprint if bots are blocked in .htaccess. If you block them in .robotstxt then it is a footprint.

thunderhd · Oct 23, 2019

Rj268 said:
It's is far more likely that a competitor will find your PBN and report it than Google finding out. No footprint if bots are blocked in .htaccess. If you block them in .robotstxt then it is a footprint.

Ok, do you think genuine websites block these bots? Do you ever block these bots on your money sites?
If google will see that all links you got from site that block specific bots. isn't it a sign that you are on a network? So how you think google gonna treat your website?

Now what happen if you don't block.

You will build genuine website on exp. domains. Let say you have 10 money sites. You link out to your money site + other sites as well. Don't you think its a genuine approach?

Offcourse if your PBN site looks ugly af, anyone can guess it as a network site.

When I create, I treat every network domains as money site. Even I rank them on easy keywords. Even if your comp report, it's a lesser chance that Google gonna deindex.

P.S: Do you think by blocking these bot can safeguard your network? Not. a little hack can expose your network. Yes, these methods exists.

Even if you block in htaccess, it's easy to find some bots that you have blocked.

Howver, my $0.02, It your wish.

Not to forget, yes, you can mixup with some blocked or non blocked.

I Build my network as my money sites. Atleast someone will think twice before report.

KHR · Oct 23, 2019

I only allow Google & Bing BOT using .htaccess for all of my money sites but not for PBN without any issue.

itz_styx · Oct 23, 2019

i don't bother blocking them, but if you really want to .htaccess is ok, but a custom php script with rDNS checks is the best option as if you only do UA checks its not very reliable.

Rj268 · Oct 23, 2019

thunderhd said:
Ok, do you think genuine websites block these bots? Do you ever block these bots on your money sites?
If google will see that all links you got from site that block specific bots. isn't it a sign that you are on a network? So how you think google gonna treat your website?

Now what happen if you don't block.

You will build genuine website on exp. domains. Let say you have 10 money sites. You link out to your money site + other sites as well. Don't you think its a genuine approach?

Offcourse if your PBN site looks ugly af, anyone can guess it as a network site.

When I create, I treat every network domains as money site. Even I rank them on easy keywords. Even if your comp report, it's a lesser chance that Google gonna deindex.

P.S: Do you think by blocking these bot can safeguard your network? Not. a little hack can expose your network. Yes, these methods exists.

Even if you block in htaccess, it's easy to find some bots that you have blocked.

Howver, my $0.02, It your wish.

Not to forget, yes, you can mixup with some blocked or non blocked.

I Build my network as my money sites. Atleast someone will think twice before report.

Yep... I have seen many big websites block crawlers using .robotstxt. But the list is usually pretty short.

I don't have many PBN's with blocked crawlers anymore. I build them as money sites too. If you build crappy PBN's and spun content then by all means block the crawlers, it may help you add a few months or years to the life of the website. It sure isn't going to make it get deindexed any faster.

PbnGeek · Oct 23, 2019

tentimes said:
it's about hiding these sites from competitors not Google

Make it look like a legit site and sell your competitors the links for sky-high prices then get even better outreach links from that $$

xue8510 · Oct 26, 2019

if you site is wordpress there is a plugin is "stop the bad bots"

laur.laurix · Oct 26, 2019

True, but G will have to crawl the web using diff user agents, ips etc. to find this kind of footprint because the G bot si not blocked by these scripts. If you ask me it's to much hussle at this point for them because the web is expanding at a huge rate.

thunderhd said:
I dont block bots. you are getting links from all sites whose bots blocked . lol its a huge footprint itself

FostexKiller · Oct 26, 2019

thunderhd said:
Ok, do you think genuine websites block these bots? Do you ever block these bots on your money sites?
If google will see that all links you got from site that block specific bots. isn't it a sign that you are on a network? So how you think google gonna treat your website?

If you're blocking at a http level, via your .htaccess... how can Google even tell?

It's not going to recrawl the whole web spoofing Majestic's or Ahref's bot lol

Mr Positive · Oct 26, 2019

Open pbn sellers pages and you’ll notice crawlers blocked, hence yes people are yet blocking crawler

thunderhd · Oct 26, 2019

FostexKiller said:
If you're blocking at a http level, via your .htaccess... how can Google even tell?

It's not going to recrawl the whole web spoofing Majestic's or Ahref's bot lol

They Do, That how they detect cloaking.

Google reached at quantum supremacy, and you think things like this are impossible LOL.

mhadzer · Oct 26, 2019

I am not blocking them. My PBN is still alive for 4 years.

tiiberius · Oct 26, 2019

What I found be be more effective strategy is to only allow "good guys" in. Instead of blocking "bad guys" like everyone else is.

thunderhd said:
I dont block bots. you are getting links from all sites whose bots blocked . lol its a huge footprint itself

It's not a footprint per se, but it can trigger a flag for sure.

All depends on the niche.

Competitors are bigger issue than Google.

In some niches you might get away with it but in others, aggressive ones, you can kiss your network goodbye.

They will report it for sure but that's not really the problem. They can do negative SEO attack destroying the link profile, DDOS or SQL injections which will devalue the links.

FostexKiller · Oct 27, 2019

thunderhd said:
They Do, That how they detect cloaking. Google reached at quantum supremacy, and you think things like this are impossible LOL.

1# Every http server log in the world would record them doing it. There isn't one case of anyone ever identifying them doing that.
2# The PR/legal ramifications of them spoofing other commercial bots would be massive.
3# Quantum processing can't speed up web crawling or magically request web pages without leaving a trace.

TomTheCat · Oct 27, 2019

Bot blocking is good and effective, IF done right, from my experience. For this, you have to build up a list with user-agents, IP classes and keep it up to date. A custom solution is perfect, if you want to scale things.

Anyone still blocking bots for PBNs in 2019?

Newbie

Supreme Member

Newbie

Supreme Member

Regular Member

Supreme Member

Regular Member

Elite Member

Elite Member

Supreme Member

Regular Member

Registered Member

Elite Member

Junior Member

Elite Member

Regular Member

Regular Member

Elite Member

Junior Member

Elite Member

Main Menu

Marketplace

Making Money

BlackHat World