Tip on indexing and robots.txt

[krawling1]

Hi BHW.
I've seen a lot of shares for creating profiles or other pages where it is possible to put a link. I want you to know, that a lot of them are blocked by robots.txt and would never be indexed even if you go to church and pray for that.

Two tips on how to check whether a page is closed or no:
1) look at the source page. Then press CTRL+F and put this into the field: "content". If a webmaster doesn't want this page to be seen you'll see "content = noindex, nofollow". In this case for Google it sounds like a recommendation. He probably would miss this page but if there are pages linking to this page he would crawl it.
2) Go to https://search.google.com/test/mobile-friendly and put there a link to a page with your link
If it is blocked by robots.txt you will see an error message.

Be careful while you placing links. You can do a worthless business.


For Example phpbb.com. You can create a profile on a VERY HIGH DA forum, you put a link and feel happy, but if you run your profile page through a mobile-friendly test you'll realize that it is blocked.

 

[gregz06]

I believe robots.txt is a recommendation which can be easily neglected by crawlers, especially if some links are pointing to the page. Though, good advice for those waiting months for their profile links to get indexed.

 

[krawling1]

I believe robots.txt is a recommendation which can be easily neglected by crawlers, especially if some links are pointing to the page. Though, good advice for those waiting months for their profile links to get indexed.

I believe too. But there are some cases when the page can be totally closed at least for simple indexing links which people want to use for rankings.

 

[meandmyjetpack]

Hi BHW.
I've seen a lot of shares for creating profiles or other pages where it is possible to put a link. I want you to know, that a lot of them are blocked by robots.txt and would never be indexed even if you go to church and pray for that.

Two tips on how to check whether a page is closed or no:
1) look at the source page. Then press CTRL+F and put this into the field: "content". If a webmaster doesn't want this page to be seen you'll see "content = noindex, nofollow". In this case for Google it sounds like a recommendation. He probably would miss this page but if there are pages linking to this page he would crawl it.
2) Go to https://search.google.com/test/mobile-friendly and put there a link to a page with your link
If it is blocked by robots.txt you will see an error message.

Be careful while you placing links. You can do a worthless business.


For Example phpbb.com. You can create a profile on a VERY HIGH DA forum, you put a link and feel happy, but if you run your profile page through a mobile-friendly test you'll realize that it is blocked.

I tried method 1 on https://www.phpbb.com/community/mem...u=159755&sid=e7911ba39fd0ee2a0fd767598e5e4532

Why doesn't the source page show the "content = noindex, nofollow" tag?

ty

 

[krawling1]

I tried method 1 on https://www.phpbb.com/community/mem...u=159755&sid=e7911ba39fd0ee2a0fd767598e5e4532

Why doesn't the source page show the "content = noindex, nofollow" tag?

ty

hi
it's blocked in robots.txt
they blocked the whole directory with profiles

 

[shaggy99]

I have some profile backlinks that I suspect are no index.
I searched in their page source, but I didn't found the noindex tag.
Also, after making the mobile friendly test, I have some page resources that can't be loaded, and I have a few statuses "Googlebot blocked by robots.txt" . But this kind of error I also have on pages that I'm sure are indexed.
Where I should look more to be sure the pages are no index?