htaccess file blocking crawler

xcodes said:

Paste your present .htaccess content here. And what do you mean by handle ? htaccess is respected by apache whatever rules you enter it should be followed by hhtpd on your server.

Click to expand...

xcodes said:

Take .htaccess backup first and replace the code with below

Spoiler: Optimized htaccess

<IfModule mod_deflate.c>
SetOutputFilter DEFLATE

AddOutputFilterByType DEFLATE text/html text/css text/plain text/xml application/x-javascript application/x-httpd-php image/svg+xml application/xhtml+xml application/xml application/rss+xml application/atom_xml

BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip

<IfModule mod_headers.c>
Header append Vary User-Agent env=!dont-vary
</IfModule>

</IfModule>

<IfModule mod_expires.c>
ExpiresActive On
ExpiresDefault "access plus 1 month"
ExpiresByType audio/ogg "access plus 1 month"
ExpiresByType image/bmp "access plus 1 month"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType video/mp4 "access plus 1 month"
ExpiresByType video/ogg "access plus 1 month"
ExpiresByType video/webm "access plus 1 month"
AddType image/x-icon .ico
ExpiresByType image/ico "access plus 1 month"
ExpiresByType image/icon "access plus 1 month"
ExpiresByType image/x-icon "access plus 1 month"
ExpiresByType text/css "access plus 1 month"
ExpiresByType text/javascript "access plus 1 month"
ExpiresByType text/html "access plus 7200 seconds"
ExpiresByType application/xhtml+xml "access plus 7200 seconds"
ExpiresByType application/javascript "access plus 1 month"
ExpiresByType application/x-javascript "access plus 1 month"
ExpiresByType application/x-shockwave-flash "access plus 1 month"
AddType application/font-woff2 .woff2
ExpiresByType application/font-woff "access plus 1 month"
ExpiresByType application/font-woff2 "access plus 1 month"
ExpiresByType application/vnd.ms-fontobject "access plus 1 month"
ExpiresByType application/x-font-ttf "access plus 1 month"
ExpiresByType font/opentype "access plus 1 month"
ExpiresByType image/svg+xml "access plus 1 month"
ExpiresByType application/x-font-opentype "access plus 1 month"
ExpiresByType application/x-font-woff "access plus 1 month"

</IfModule>

<IfModule mod_headers.c>

<FilesMatch "\.(ico|jpe?g|png|gif|swf|css|gz)$">
Header set Cache-Control "max-age=2592000, public"
</FilesMatch>

<FilesMatch "\.(js)$">
Header set Cache-Control "max-age=2592000, private"
</FilesMatch>

<filesMatch "\.(html|htm)$">
Header set Cache-Control "max-age=7200, public"
</filesMatch>

<FilesMatch "\.(pl|php|cgi|spl|scgi|fcgi)$">
Header unset Cache-Control
</FilesMatch>

</IfModule>

Click to expand...

vmaart said:

<IfModule mod_deflate.c>
SetOutputFilter DEFLATE
<IfModule mod_setenvif.c>
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
# BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html

# Don't compress images
SetEnvIfNoCase Request_URI .(?:gif|jpe?g|png)$ no-gzip dont-vary
</IfModule>

<IfModule mod_headers.c>
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</IfModule>
</IfModule>

Click to expand...

bartosimpsonio said:

Wait. Does your htaccess say Netscape 4? When was this created, year 2000 or something?

Click to expand...

vmaart said:

Thank you bro,
I'm waiting.

Can you just elaborate me about this coding.....

Click to expand...

bartosimpsonio said:

What is your indexing problem? htaccess rarely has anything to do with what Google will index.

Click to expand...

vmaart said:

Most people like me have the google index issue.

Since we use share hosting, is there a possibility to restrict the .htaccess file? It can block the crawler Or may give difficulty.
If this is true how to handle the .htaccess file correctly,

Plz advice me

Click to expand...

vmaart said:

Thank you

Yes I did backup my htaccess file and replace your code.

Let me know what the function of this code.

Click to expand...

Gogol said:

It could play a role, e.g. if the htaccess blocks bot browser agents.

Click to expand...

bartosimpsonio said:

Yeah, htaccess can almost always overwrite options. But OP hasn't told us what the indexing issue is, thus my previous post.

Click to expand...

vmaart said:

Hi bro
I already asked server log.
My hosting providers ignore my request. They didn't able to provide log files for shared hosting.
Htacceas file may block some bits.

Click to expand...

rafark said:

Most people that block bots and crawlers without relying on the unreliable robots.txt do so by using an htaccess file.

I assume the op wants to block seo crawlers (ahrefs & co). Blocking by user agent is not 100% bullet proof (as anyone can spoof it because it's just a plain HTTP header) but it can get the job done in some cases.

Click to expand...