SEMruch can't crawl my website

Mohamad Zidani

Power Member
Joined
Nov 17, 2016
Messages
530
Reaction score
65
when i started to auditing my website the crawl stop with this message :
We encountered an error that stopped us from crawling your website:
your network connection was interrupted.

Log :

IP address of Semrush Bot ( Site Audit ): XX.XXX.XXX.XX
REQUEST:
curl -i -sS -L --proto-redir -all,http,https --max-time 5 -A 'Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)' http://XXXXXXXXXX/
RESPONSE:
curl: (28) Connection timed out after 5000 milliseconds
 
Looks like you blocked the SEMrush bot from auditing your website ...

Can you take a look at your .htacess?
 
Looks like you blocked the SEMrush bot from auditing your website ...

Can you take a look at your .htacess?

i don't think so.

------------------------------------------------------------

## EXPIRES CACHING ##
<IfModule mod_expires.c>
ExpiresActive On
ExpiresByType image/jpg "access 1 year"
ExpiresByType image/jpeg "access 1 year"
ExpiresByType image/gif "access 1 year"
ExpiresByType image/png "access 1 year"
ExpiresByType text/css "access 1 month"
ExpiresByType text/html "access 1 month"
ExpiresByType application/pdf "access 1 month"
ExpiresByType text/x-javascript "access 1 month"
ExpiresByType application/x-shockwave-flash "access 1 month"
ExpiresByType image/x-icon "access 1 year"
ExpiresDefault "access 1 month"
</IfModule>
## EXPIRES CACHING ##
#Force www:
RewriteEngine on
RewriteCond %{HTTP_HOST} ^maroccontact.ma [NC]
RewriteRule ^(.*)$ http://www.XXXXXX.ma/$1 [L,R=301,NC]
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
 
Back
Top