1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XML sitemap: uncrawlable pages

Discussion in 'White Hat SEO' started by Cheebee, Oct 25, 2013.

  1. Cheebee

    Cheebee Newbie

    Joined:
    Apr 19, 2013
    Messages:
    25
    Likes Received:
    3
    Hey there :)

    I'm doing SEO on a bilingual website, passeportvacances.com. The issue we have is that a very important section, the coupons one, is not being indexed by google.

    We have a XML sitemap online on Webmaster Tools but it is incomplete. The sitemap generators are not able to get links from the coupon section as they consider them "broken links". Here is the jpg report of the links that are considered broken by the tool.

    Broken links.jpg

    1) Does someone have an intuition as of why this section is the only one that screws up?
    2) What if I add the URLs manually to the sitemap before submitting it, will it work anyways or Google will still consider the links are broken? If so, will it be bad for the SEO?

    Thanks a lot for your strategic input!
    Charles
     
  2. sashablack

    sashablack Elite Member

    Joined:
    Jan 8, 2010
    Messages:
    3,697
    Likes Received:
    2,071
    Gender:
    Male
    have you done this in google?

    site:yourwebsite.com

    do that and see what is indexed.
     
  3. Cheebee

    Cheebee Newbie

    Joined:
    Apr 19, 2013
    Messages:
    25
    Likes Received:
    3
    Hey,

    Thanks for the answer. I did use the command, and saw that the section I'm talking about (the dynamic coupon pages) is not there at all. What could make a section uncrawlable even by the sitemap generator if it works perfectly fine onsite?
     
  4. sashablack

    sashablack Elite Member

    Joined:
    Jan 8, 2010
    Messages:
    3,697
    Likes Received:
    2,071
    Gender:
    Male
    when did you post those pages, cuz it might just be taking a bit to index them, are you getting an error in webmaster tools regarding those pages?
     
    • Thanks Thanks x 1
  5. Cheebee

    Cheebee Newbie

    Joined:
    Apr 19, 2013
    Messages:
    25
    Likes Received:
    3
    No error in Webmaster tools, and sadly, the pages have been up and running for months... and they're directly linked to the homepage :s
     
  6. Cheebee

    Cheebee Newbie

    Joined:
    Apr 19, 2013
    Messages:
    25
    Likes Received:
    3
    Any other feeling as why an XML sitemap generator could not detect the pages? :s I'm also scared Google is not gonna like it if I manually add pages to the sitemap and he doesn't understand them.
     
  7. mrpasta

    mrpasta Newbie

    Joined:
    Apr 29, 2011
    Messages:
    25
    Likes Received:
    19
    Cheebee, what tool are you using to create your sitemap? I use Inspyder Sitemap Creator. If it won't crawl your pages, contact them for help (they are good and they helped me solve a few indexing issues my site was having).
     
    • Thanks Thanks x 1
  8. YellowHats

    YellowHats Junior Member

    Joined:
    Mar 10, 2013
    Messages:
    195
    Likes Received:
    135
    Occupation:
    Director of SEO
    Location:
    Middle East
    Just took a look at the site. Something very strange is going on. You need to check your htaccess or something. The coupons page is getting a 404 even though you can see the page. Put this link: http://www.passeportvacances [DOT] com/coupons.html in either of these tools:

    http://web-sniffer.net/
    http://validator.w3.org/

    EDIT: This is probably a PHP error. Are you using WordPress?
     
    • Thanks Thanks x 1
    Last edited: Oct 28, 2013
  9. Cheebee

    Cheebee Newbie

    Joined:
    Apr 19, 2013
    Messages:
    25
    Likes Received:
    3
    I used a few different xml sitemaps generators, Mr Pasta, but all of them couldn't retrieve the section. I guess this makes sense, considering the W3 validator is not able to retrieve it.

    Yellowhats, that's a great insight, thank you! I did not program the website myself (I'm only the SEO) but I know it's done with Coldfusion, using a homemade CMS. I can definitely update the programmer with what you just said, though... so you think that there's is a line in the PHP code that makes the pages make a 404? That's technically possible even though the CMS is able to show some content?

    Thanks a lot for the analysis,
    Cheebee
     
  10. YellowHats

    YellowHats Junior Member

    Joined:
    Mar 10, 2013
    Messages:
    195
    Likes Received:
    135
    Occupation:
    Director of SEO
    Location:
    Middle East
    Hey Cheebee,
    No problem. Yes - the problem is technically possible. It happens every once a while in WordPress if you improperly arrange some code. But more than that the developers will definitely have to figure out - no way to know without seeing the site's code.

    Good luck!