htaccess file blocking crawler

vmaart

vmaart

Regular Member
Joined
May 19, 2017
Messages
200
Reaction score
9
Most people like me have the google index issue.

Since we use share hosting, is there a possibility to restrict the .htaccess file? It can block the crawler Or may give difficulty.
If this is true how to handle the .htaccess file correctly,

Plz advice me
 
Advertise on BHW
Paste your present .htaccess content here. And what do you mean by handle ? htaccess is respected by apache whatever rules you enter it should be followed by hhtpd on your server.
 
xcodes said:
Paste your present .htaccess content here. And what do you mean by handle ? htaccess is respected by apache whatever rules you enter it should be followed by hhtpd on your server.
Click to expand...
<IfModule mod_deflate.c>
SetOutputFilter DEFLATE
<IfModule mod_setenvif.c>
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
# BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html

# Don't compress images
SetEnvIfNoCase Request_URI .(?:gif|jpe?g|png)$ no-gzip dont-vary
</IfModule>

<IfModule mod_headers.c>
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</IfModule>
</IfModule>
 
Take .htaccess backup first and replace the code with below

<IfModule mod_deflate.c>
SetOutputFilter DEFLATE

AddOutputFilterByType DEFLATE text/html text/css text/plain text/xml application/x-javascript application/x-httpd-php image/svg+xml application/xhtml+xml application/xml application/rss+xml application/atom_xml

BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip

<IfModule mod_headers.c>
Header append Vary User-Agent env=!dont-vary
</IfModule>

</IfModule>

<IfModule mod_expires.c>
ExpiresActive On
ExpiresDefault "access plus 1 month"
ExpiresByType audio/ogg "access plus 1 month"
ExpiresByType image/bmp "access plus 1 month"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType video/mp4 "access plus 1 month"
ExpiresByType video/ogg "access plus 1 month"
ExpiresByType video/webm "access plus 1 month"
AddType image/x-icon .ico
ExpiresByType image/ico "access plus 1 month"
ExpiresByType image/icon "access plus 1 month"
ExpiresByType image/x-icon "access plus 1 month"
ExpiresByType text/css "access plus 1 month"
ExpiresByType text/javascript "access plus 1 month"
ExpiresByType text/html "access plus 7200 seconds"
ExpiresByType application/xhtml+xml "access plus 7200 seconds"
ExpiresByType application/javascript "access plus 1 month"
ExpiresByType application/x-javascript "access plus 1 month"
ExpiresByType application/x-shockwave-flash "access plus 1 month"
AddType application/font-woff2 .woff2
ExpiresByType application/font-woff "access plus 1 month"
ExpiresByType application/font-woff2 "access plus 1 month"
ExpiresByType application/vnd.ms-fontobject "access plus 1 month"
ExpiresByType application/x-font-ttf "access plus 1 month"
ExpiresByType font/opentype "access plus 1 month"
ExpiresByType image/svg+xml "access plus 1 month"
ExpiresByType application/x-font-opentype "access plus 1 month"
ExpiresByType application/x-font-woff "access plus 1 month"

</IfModule>

<IfModule mod_headers.c>

<FilesMatch "\.(ico|jpe?g|png|gif|swf|css|gz)$">
Header set Cache-Control "max-age=2592000, public"
</FilesMatch>

<FilesMatch "\.(js)$">
Header set Cache-Control "max-age=2592000, private"
</FilesMatch>

<filesMatch "\.(html|htm)$">
Header set Cache-Control "max-age=7200, public"
</filesMatch>

<FilesMatch "\.(pl|php|cgi|spl|scgi|fcgi)$">
Header unset Cache-Control
</FilesMatch>

</IfModule>
 
xcodes said:
Take .htaccess backup first and replace the code with below

<IfModule mod_deflate.c>
SetOutputFilter DEFLATE

AddOutputFilterByType DEFLATE text/html text/css text/plain text/xml application/x-javascript application/x-httpd-php image/svg+xml application/xhtml+xml application/xml application/rss+xml application/atom_xml

BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip

<IfModule mod_headers.c>
Header append Vary User-Agent env=!dont-vary
</IfModule>

</IfModule>

<IfModule mod_expires.c>
ExpiresActive On
ExpiresDefault "access plus 1 month"
ExpiresByType audio/ogg "access plus 1 month"
ExpiresByType image/bmp "access plus 1 month"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType video/mp4 "access plus 1 month"
ExpiresByType video/ogg "access plus 1 month"
ExpiresByType video/webm "access plus 1 month"
AddType image/x-icon .ico
ExpiresByType image/ico "access plus 1 month"
ExpiresByType image/icon "access plus 1 month"
ExpiresByType image/x-icon "access plus 1 month"
ExpiresByType text/css "access plus 1 month"
ExpiresByType text/javascript "access plus 1 month"
ExpiresByType text/html "access plus 7200 seconds"
ExpiresByType application/xhtml+xml "access plus 7200 seconds"
ExpiresByType application/javascript "access plus 1 month"
ExpiresByType application/x-javascript "access plus 1 month"
ExpiresByType application/x-shockwave-flash "access plus 1 month"
AddType application/font-woff2 .woff2
ExpiresByType application/font-woff "access plus 1 month"
ExpiresByType application/font-woff2 "access plus 1 month"
ExpiresByType application/vnd.ms-fontobject "access plus 1 month"
ExpiresByType application/x-font-ttf "access plus 1 month"
ExpiresByType font/opentype "access plus 1 month"
ExpiresByType image/svg+xml "access plus 1 month"
ExpiresByType application/x-font-opentype "access plus 1 month"
ExpiresByType application/x-font-woff "access plus 1 month"

</IfModule>

<IfModule mod_headers.c>

<FilesMatch "\.(ico|jpe?g|png|gif|swf|css|gz)$">
Header set Cache-Control "max-age=2592000, public"
</FilesMatch>

<FilesMatch "\.(js)$">
Header set Cache-Control "max-age=2592000, private"
</FilesMatch>

<filesMatch "\.(html|htm)$">
Header set Cache-Control "max-age=7200, public"
</filesMatch>

<FilesMatch "\.(pl|php|cgi|spl|scgi|fcgi)$">
Header unset Cache-Control
</FilesMatch>

</IfModule>
Click to expand...
Thank you

Yes I did backup my htaccess file and replace your code.

Let me know what the function of this code.
 
vmaart said:
<IfModule mod_deflate.c>
SetOutputFilter DEFLATE
<IfModule mod_setenvif.c>
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
# BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html

# Don't compress images
SetEnvIfNoCase Request_URI .(?:gif|jpe?g|png)$ no-gzip dont-vary
</IfModule>

<IfModule mod_headers.c>
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</IfModule>
</IfModule>
Click to expand...


Wait. Does your htaccess say Netscape 4? When was this created, year 2000 or something?
 
bartosimpsonio said:
Wait. Does your htaccess say Netscape 4? When was this created, year 2000 or something?
Click to expand...
Thank you bro,
I'm waiting.

Can you just elaborate me about this coding.....
 
vmaart said:
Thank you bro,
I'm waiting.

Can you just elaborate me about this coding.....
Click to expand...

What is your indexing problem? htaccess rarely has anything to do with what Google will index.
 
Wow netscape i didn't hear that name for a long time :)
htaccess rarely has anything to do with Google indexing. Did you try calling your hosting company?
Ask them if they see any problem. check your robots.txt file. see that you don't have any disallow there.
 
vmaart said:
Most people like me have the google index issue.

Since we use share hosting, is there a possibility to restrict the .htaccess file? It can block the crawler Or may give difficulty.
If this is true how to handle the .htaccess file correctly,

Plz advice me
Click to expand...
Are you saying that .htaccess of someone else's website is blocking your website? If this is what you are saying then.... NO the .htaccess of other websites cannot block pages of other websites on even shared hosting.
 
vmaart said:
Thank you

Yes I did backup my htaccess file and replace your code.

Let me know what the function of this code.
Click to expand...

Better security + optimization of your own version :)
any by the way your not blocking any crawlers or bot with htaccess presently so its not htaccess check your robot.txt
 
Gogol said:
It could play a role, e.g. if the htaccess blocks bot browser agents.
;)
Click to expand...

Yeah, htaccess can almost always overwrite options. But OP hasn't told us what the indexing issue is, thus my previous post.
 
bartosimpsonio said:
Yeah, htaccess can almost always overwrite options. But OP hasn't told us what the indexing issue is, thus my previous post.
Click to expand...
Most people that block bots and crawlers without relying on the unreliable robots.txt do so by using an htaccess file.

I assume the op wants to block seo crawlers (ahrefs & co). Blocking by user agent is not 100% bullet proof (as anyone can spoof it because it's just a plain HTTP header) but it can get the job done in some cases.
 
Hi bro
I already asked server log.
My hosting providers ignore my request. They didn't able to provide log files for shared hosting.
Htacceas file may block some bits.
 
vmaart said:
Hi bro
I already asked server log.
My hosting providers ignore my request. They didn't able to provide log files for shared hosting.
Htacceas file may block some bits.
Click to expand...
It should be there in /var/log/apache2 anyway. Try doing a cat/tail to those files there. Or do they not give you shell access? In that case, it should be available somewhere in cpanel (not sure where.. used cpanel a long time ago lol).
 
rafark said:
Most people that block bots and crawlers without relying on the unreliable robots.txt do so by using an htaccess file.

I assume the op wants to block seo crawlers (ahrefs & co). Blocking by user agent is not 100% bullet proof (as anyone can spoof it because it's just a plain HTTP header) but it can get the job done in some cases.
Click to expand...

robots.txt definitely does not block anything.

Several well known popular crawlers fetch robots.txt and then proceed to crawl blocked pages anyway using fake UA.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top