How to hide links from SEO crawler
- Thread starter Marabunta
- Start date
ContentWriter
Banned - Selling via PM
- May 8, 2013
- 5,735
- 975
@Marabunta
When it comes to hiding links from SEO crawlers, here are some solutions you can consider. Robots.txt is one of the best ways to prevent spiders from accessing your website. Another way is via the Meta Robots Tag. It allows the blocking of crawlers to access your website by page level through applying "noindex." Using Iframes can embed content from other URLs into any selected web page. Lastly, you can consider using Java Applets, which hides text in images.
- Jun 20, 2013
- 11,201
- 8,546
OP you can block them using robot txt file, but nowadays they are ignoring it.
amitjat
Registered Member
- Jul 28, 2018
- 73
- 21
Most SEOs accomplish through abhorrent .htaccess lists of bots to block.
The current solutions are…
The current solutions are…
- .htaccess files
- Not always applicable if you don’t have full control over the site
- Often times blocked from being edited
- Require some technical know-how, updating, and blind trust
- Footprint if set up incorrectly
- Blind trust/copying pasting code you find online is bound to lead to problems
- robots.txt
- (lol)
- WordPress plugins
- Not so great if you’re not using WordPress
- Require blind-trust in the author of the plugin
- Footprint if (when) set up incorrectly (more on that in another post)
Well, I haven't tried this but just thought of it and logically seems like it should work..
1) Create a new folder
2) Block bot permissions to said folder
3) Any shady link you want to redirect you create a 301 redirect within that folder
4) Link from your website to whichever one of those 301s you want to link to
You can use a combination of the other comments above too..
So you can use robots + htaccess to block bot access to that folder of redirects as well as include the noindex, nofollow meta tag within your 301 redirects. Also on your href link out you can add a rel=nofollow.
1) Create a new folder
2) Block bot permissions to said folder
3) Any shady link you want to redirect you create a 301 redirect within that folder
4) Link from your website to whichever one of those 301s you want to link to
You can use a combination of the other comments above too..
So you can use robots + htaccess to block bot access to that folder of redirects as well as include the noindex, nofollow meta tag within your 301 redirects. Also on your href link out you can add a rel=nofollow.
rafark
Elite Member
- Jan 15, 2013
- 3,249
- 1,976
Robots.txt is the WORST way to block them. It's basically just asking crawlers not to crawl your site, but it's up to them to do it or not, and bot companies need the data to survive... And you're leaving a footprint.
If you don't have your own script, at least do the checks in an htaccess file.
The best way to do it is to trick them into thinking they are viewing a legit page, if you block them by user agent and send them an error or redirection code as a response, some might try to re-crawl your site disguised as a regar user or even as google bot.
If you don't have your own script, at least do the checks in an htaccess file.
The best way to do it is to trick them into thinking they are viewing a legit page, if you block them by user agent and send them an error or redirection code as a response, some might try to re-crawl your site disguised as a regar user or even as google bot.
