Why cant my new website be crawled after changing robots.txt ?

[HackLady]

Hi guys,

Can you please help me with this issue regarding Google indexing in Search Console: my new website (created with WordPress) cannot be crawled after verifying the website, changing robots.txt, and even added a meta tag in the head section.

Basically, first I verified property in Search Console. By adding the txt record. All went fine.
Then, the Inspect URL and Request Indexing kept failing constantly so I read all tutorials and I did the following:

- the file robots.txt did not exist, so I created it in the root of my website. It contains this code:
User-agent: * Disallow:
- I even added a meta tag in the head section, from header.php:
<meta name="robots" content="index,follow">
- I installed the famous YOAST and configured it. It says there are no issues and that everything is working fine and that search engines have access to my website.
- uploaded the sitemaps created by YOAST into google... they have a red text that says: Can't fetch.
- in WordPress the option not allowing search engines to find the website is un checked.

Yet still, Inspect URL, AMP test, Mobile-Friendly Test , Rich Results test... all keep saying the same thing:

Page cannot be reached
This could be because the page is unavailable or blocked by robots.txt

Attached screenshot from URL inspection and Index request.

Please, can you help me with a possible fix for this issue?

Thank you very much!
Diana

 

[bossofthebosses]

Check your site here
https://technicalseo.com/tools/robots-txt/

 

[GoldenTouch]

- the file robots.txt did not exist, so I created it in the root of my website. It contains this code:
User-agent: * Disallow:

This is what's causing you indexing issues most likely. You've told google robots to not visit any pages on your website.

Check this out for more info

https://stackoverflow.com/questions/22080016/what-does-user-agent-disallow-mean

 

[Chris Newman]

I am in no way an SEO person, but I have been doing only some basic SEO for my websites since a few years. My suggestions would be :-
Is your Site is Indexed Under a www- or Non-www Domain ? Technically www is a subdomain. Thus, http://example.com is not the same as http://www.example.com. Make sure you add both sites to your Google webmaster account to ensure they are both indexed. Be sure to set your preferred domain, but verify ownership of both.
Have you turned On Privacy Settings ? If you have a WordPress site, you may have accidentally kept the privacy settings on. Go to Admin → Settings → Privacy to check.
Your webite is Blocked by .htaccess ? Your .htaccess file is part of your website’s existence on the server, which allows it to be available on the world-wide web. The .htaccess file is written in Apache. Although .htacess is handy and useful, it can be used to block crawlers and prevent indexation. So, check on that also : I am not too sure on how to check .htaccess

 

[Starblazer]

The code, "User-agent: * Disallow:" means you're requesting bots to not crawl your site. They accepted your request and abandoned your site.

You should ideally use the following code:

User-agent: *
Allow: /

Sitemap: https://your-domain .com/sitemap_index.xml

 

[NetShop ISP]

I think you need to correct code and see if that is going to help. Guys already pointed to the solution which might help you.

 

[HackLady]

Check your site here
https://technicalseo.com/tools/robots-txt/

Thank you but the site is not available. I tested other sites that offer the same service, after making the changes from this topic, and it looks like the robots.txt is free of errors and ok.
But I still have the same issue.

I really appreciate the help, any other tips?

This is what's causing you indexing issues most likely. You've told google robots to not visit any pages on your website.

Check this out for more info

https://stackoverflow.com/questions/22080016/what-does-user-agent-disallow-mean

Hi, I really appreciate the help, I have made the changes but I have the same issue, any other tips?

I am in no way an SEO person, but I have been doing only some basic SEO for my websites since a few years. My suggestions would be :-
Is your Site is Indexed Under a www- or Non-www Domain ? Technically www is a subdomain. Thus, http://example.com is not the same as http://www.example.com. Make sure you add both sites to your Google webmaster account to ensure they are both indexed. Be sure to set your preferred domain, but verify ownership of both.
Have you turned On Privacy Settings ? If you have a WordPress site, you may have accidentally kept the privacy settings on. Go to Admin → Settings → Privacy to check.
Your webite is Blocked by .htaccess ? Your .htaccess file is part of your website’s existence on the server, which allows it to be available on the world-wide web. The .htaccess file is written in Apache. Although .htacess is handy and useful, it can be used to block crawlers and prevent indexation. So, check on that also : I am not too sure on how to check .htaccess

Thank you so much for the tips,
In Search console, I have added all variations of the website. And verify them all successfully.
So, I have http://mysite.com / https://mysite.com and also http://www.mysite.com / https://www.mysite.com - still, when I try to inspect the URLS, not even one works.

Privacy is not set on my WordPress and the website is also visible.

I have no idea if it is blocked by .htaccess - can someone maybe help with this?
My htaccess has the following info:

# BEGIN WordPress
# The directives (lines) between "BEGIN WordPress" and "END WordPress" are
# dynamically generated, and should only be modified via WordPress filters.
# Any changes to the directives between these markers will be overwritten.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

Anything here that might block indexing?

In my host panel, I have turned on that automatically redirect from http to https option.

Thank you again all!

The code, "User-agent: * Disallow:" means you're requesting bots to not crawl your site. They accepted your request and abandoned your site.

You should ideally use the following code:

User-agent: *
Allow: /

Sitemap: https://your-domain .com/sitemap_index.xml

Thank you very much for the code.
I have added it but still, there is no change.

Do you maybe have other tips for me?