My website's pages (articles) are facing crawling issues. They are not getting crawled frequently. Earlier, there were no issues with crawling. However, after my tech team added some new categories for the articles & stories section. What could be the reasons?
Facing Crawling Issues
- Thread starter iliyana55
- Start date
- Feb 14, 2024
- 516
- 127
First, we need to check the robots.txt file for any "Disallow" rules blocking the new categories. Ensure the new category pages are included in the XML sitemap and submitted to search engines. Review server logs to detect changes in crawler behavior, especially for the new categories.
CallistoSEO
Supreme Member
- Jun 13, 2022
- 1,220
- 201
settting can take time to adapt force index the pages .
randyseoppc
Junior Member
- Feb 24, 2016
- 174
- 61
It seems like the new category structure might be causing the crawling issues. Often times, technical changes to the website structures confuse search engines crawlers. To pinpoint the problem, check for broken links, duplicate content, or issues with XML sitemap and robots.txt. Make sure the categories are properly structured and linked internally.
virtualize123
BANNED
- Jul 4, 2024
- 14
- 1
Your website's structure changes could be causing crawling issues. Maybe you've tweaked URLs, linked internally the wrong way, or messed up robots.txt files. Plus, you may need to freshen up your sitemap with new categories. Make sure your fresh structure is SEO-ready and important pages aren't hard for search engine crawlers to find.
sagarmalik007
BANNED
- Mar 7, 2021
- 416
- 109
Have you changed the url strucuture? If it is the case then please update the old URL structure again.
ALso try to add internal links to other pages
Build some backlinks for fast crawling
ALso try to add internal links to other pages
Build some backlinks for fast crawling