vaibhavpranjale
Newbie
- May 7, 2024
- 36
- 4
How to resolve this issue. These links are continuously increasing. This is a shopify website and I dont know how to stop to generate links. Developer is also not aware of this. Please help!!
Indexed, though blocked by Robots.txt
- Thread starter vaibhavpranjale
- Start date
How to resolve this issue. These links are continuously increasing. This is a shopify website and I dont know how to stop to generate links. Developer is also not aware of this. Please help!!
Ridlance
Junior Member
- Mar 17, 2023
- 179
- 46
Please Check and Update robots.txt file and verify that your robots.txt file is correctly configured to block the pages you don't want indexed.
Also Use Noindex Meta Tag you don't want indexed on google search results. Add a noindex meta tag to the HTML of the pages you don’t want to be indexed. This should be placed in the <head> section of the HTML:
and please ensure canonical tags are correctly implemented to prevent duplicate content issues and specify the preferred URL for indexing and this is may be main reason
After check above suggestions and then start the google search console validation to check the issue fix or not.
Also Use Noindex Meta Tag you don't want indexed on google search results. Add a noindex meta tag to the HTML of the pages you don’t want to be indexed. This should be placed in the <head> section of the HTML:
and please ensure canonical tags are correctly implemented to prevent duplicate content issues and specify the preferred URL for indexing and this is may be main reason
After check above suggestions and then start the google search console validation to check the issue fix or not.
- Feb 14, 2024
- 516
- 127
Update your robots.txt file to disallow specific paths like /search and /cart, and use noindex meta tags on pages that should not be indexed. Block unwanted URL parameters in Google Search Console and ensure canonical tags are correctly implemented on important pages. Next, submit an updated sitemap to Google Search Console.
vaibhavpranjale
Newbie
- May 7, 2024
- 36
- 4
I cant find those links in the backend.
vaibhavpranjale
Newbie
- May 7, 2024
- 36
- 4
Developer told me there is a filter app and that's why the links are generating. My main concern is I can't find that link in my shopify backend. How do I get rid of those links in console?
Seemasingh5020
Junior Member
- Apr 29, 2024
- 156
- 72
If you have a WordPress Website then you can follow these steps:
Method 1: Edit robots.txt Directly
If you have a WordPress website, you’ll probably have a virtual robots.txt file. You can visit it by searching for domain.com/robots.txt in a web browser (replacing domain.com with your domain name). However, this virtual file won’t enable you to make edits.
To start editing robots.txt, you’ll need to create a file on your server. Firstly, choose a text editor and create a new file. Be sure to name it “robots.txt”:
When you’re finished, save your changes. Then, go back to Google Search Console to see if this method resolved the error.
Method 2: Use an SEO Plugin
If you have an SEO plugin activated, you won’t have to create an entirely new robots.txt file. In many cases, the SEO tool will build one for you. Plus, it may also provide ways to edit the file without leaving the WordPress dashboard.
Yoast SEO
One of the most popular SEO plugins is Yoast SEO. It can provide a detailed on-page SEO analysis, along with additional tools to customize your search engine indexing.
To start editing your robots.txt file, go to Yoast SEO > Tools in your WordPress dashboard. From the list of built-in tools, select the File editor:
When using Yoast select Yoast SEO file editor
Similarly you can do it with Rank Math, Squirrly SEO Plugin.
Method 1: Edit robots.txt Directly
If you have a WordPress website, you’ll probably have a virtual robots.txt file. You can visit it by searching for domain.com/robots.txt in a web browser (replacing domain.com with your domain name). However, this virtual file won’t enable you to make edits.
To start editing robots.txt, you’ll need to create a file on your server. Firstly, choose a text editor and create a new file. Be sure to name it “robots.txt”:
- Create a new robots.txt file in the text editor of your choice
- Create a new robots.txt file
- Then, you’ll need to connect to an SFTP client.
- Lastly, upload the robots.txt file to your root directory (for WordPress sites, it should be called public_html). Then, you can open the file and make the necessary changes.
- You can use allow and disallow statements to customize the indexing of your WordPress site. For example, you may want a certain file to be crawled without indexing the entire folder. In this case, you can add this code:
User-agent:
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
When you’re finished, save your changes. Then, go back to Google Search Console to see if this method resolved the error.
Method 2: Use an SEO Plugin
If you have an SEO plugin activated, you won’t have to create an entirely new robots.txt file. In many cases, the SEO tool will build one for you. Plus, it may also provide ways to edit the file without leaving the WordPress dashboard.
Yoast SEO
One of the most popular SEO plugins is Yoast SEO. It can provide a detailed on-page SEO analysis, along with additional tools to customize your search engine indexing.
To start editing your robots.txt file, go to Yoast SEO > Tools in your WordPress dashboard. From the list of built-in tools, select the File editor:
When using Yoast select Yoast SEO file editor
- Select Yoast SEO file editor
- Yoast SEO will not automatically create a robots.txt file. If you don’t already have one, click on Create robots.txt file:
- Create a robots.txt file with Yoast SEO
- This will open a text editor with the content of your new robots.txt file. Similar to the first method, you can add allow statements to pages you want to be indexed. Alternatively, use disallow statements for URLs to avoid indexing:
- Edit Yoast SEO’s robots.txt file
- After making your changes, save the file. Yoast SEO will alert you when you’ve updated the robots.txt file.
Similarly you can do it with Rank Math, Squirrly SEO Plugin.
randyseoppc
Junior Member
- Feb 24, 2016
- 174
- 61
Based on the image it seems that your Shopify website has pages that are indexed by Google even though they are blocked by Robots.txt file, causing unnecessary links to be generated which is negatively impacting your website's SEO. Here are the possible solutions:
1. Remove the pages from Robots.txt file, but then these pages won't stay private.
2. Add a no-index tags to these pages, signaling Google not to index these pages even they are not blocked by Robotx.txt.
3. Resolve the issues with the Robots.txt file. I believe there might be some syntax errors on the Robots.txt file, which might be causing problems.