Sitemap Issue

P

pujarichestsoft

Newbie
Joined
May 30, 2024
Messages
18
Reaction score
3
My website is in Code Igniter. The present sitemap is a static file. I generated a new sitemap through online tools, but the new sitemap has 979 pages, whereas the old one has only 364. This is not the issue.

The issue is that the pages available in the older sitemap are not included in the new one. I am surprised why is this so? Although the pages are crawled and indexed, they are missing in the new sitemap, and I don't know why. Please help me understand this.
 
Advertise on BHW
These tools usually have settings on what to include/exclude in a sitemap.

They can be available for you to customize or they might be running in the background.

For example tag or author pages can be ignored. (some WordPress plugins do this by default)

I suggest to experiment with multiple tools. See if the outcome is the same.

If it the same, there is probably on-page issue. Like a bot or user-agent being blocked.
 
pujarichestsoft said:
My website is in Code Igniter. The present sitemap is a static file. I generated a new sitemap through online tools, but the new sitemap has 979 pages, whereas the old one has only 364. This is not the issue.

The issue is that the pages available in the older sitemap are not included in the new one. I am surprised why is this so? Although the pages are crawled and indexed, they are missing in the new sitemap, and I don't know why. Please help me understand this.
Click to expand...
If your older pages are missing from the new sitemap, it could be due to how the tool is detecting URLs or some restrictions in place. Double-check your site's robots.txt, canonical tags, and any settings in the generator that might be filtering certain pages out.
 
Some pages might be blocked by your robots.txt file or have meta noindex tags. Even though they’re indexed (perhaps from previous sitemaps), the tool may be ignoring them now. Do a double-checking the settings on the sitemap generator, and perhaps trying another tool to compare results.
 
I want to ask, in a website using a sitemap, besides providing directions, does it have any other uses?
 
It’s possible that the online tool you used applies different criteria for including pages in the sitemap. Here are some potential reasons why your older pages might be missing:

  1. Canonical Tags – If these pages have canonical tags pointing elsewhere, the tool might exclude them.
  2. Noindex Meta Tag – If some pages are marked as noindex, they won’t appear in the new sitemap.
  3. Blocked by Robots.txt – Check if your robots.txt is preventing those pages from being included.
  4. URL Structure Changes – If URLs were altered, the tool may not recognize them as the same.
  5. Tool Limitations – Some sitemap generators have restrictions on which URLs they detect.
Try generating the sitemap using a CodeIgniter-specific method or a more advanced tool like Screaming Frog SEO Spider. Also, compare the missing URLs to find common patterns.
 
A sitemap issue occurs when a website's sitemap is missing, incorrect, or not properly indexed by search engines, affecting SEO and discoverability.
pujarichestsoft said:
My website is in Code Igniter. The present sitemap is a static file. I generated a new sitemap through online tools, but the new sitemap has 979 pages, whereas the old one has only 364. This is not the issue.

The issue is that the pages available in the older sitemap are not included in the new one. I am surprised why is this so? Although the pages are crawled and indexed, they are missing in the new sitemap, and I don't know why. Please help me understand this.
Click to expand...
 
2lvn said:
I want to ask, in a website using a sitemap, besides providing directions, does it have any other uses?
Click to expand...
Yes it helps in indexing. Although it's not very true because a website without a sitemap can be indexed with the same efficiency but yeah, crawlers do visit a sitemap when they reach a website.
 
Map Generation Tools

Try Screaming Frog SEO Spider or Yoast SEO (if WordPress).
 
M1ll10n3rNew said:
Some pages might be blocked by your robots.txt file or have meta noindex tags. Even though they’re indexed (perhaps from previous sitemaps), the tool may be ignoring them now. Do a double-checking the settings on the sitemap generator, and perhaps trying another tool to compare results.
Click to expand...
No, there is no noindex and robots restriction.
 
neaot said:
Map Generation Tools

Try Screaming Frog SEO Spider or Yoast SEO (if WordPress).
Click to expand...
I tried but with the free version of screeming frog. Got some 500 urls and you know the same issue is there. The pages provided by screeming frog do not include some pages from the older sitemap. But there could be an exception as there are only 500 pages provided by screeming frog, we can say others may skipped out while rendering.
 
2lvn said:
I want to ask, in a website using a sitemap, besides providing directions, does it have any other uses?
Click to expand...
Besides providing directions for search engines, a sitemap also helps improve indexing efficiency, ensuring important pages are discovered.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top