1. This website uses cookies to improve service and provide a tailored user experience. By using this site, you agree to this use. See our Cookie Policy.
    Dismiss Notice

Any tool to extract sitemap xml file of any website?

Discussion in 'Black Hat SEO Tools' started by developer1989, Sep 17, 2019.

Tags:
  1. developer1989

    developer1989 Regular Member

    Joined:
    Aug 31, 2019
    Messages:
    431
    Likes Received:
    48
    Gender:
    Male
    I want to see a website (ranking in first) sitemap xml file

    tried website.com/sitemap.xml and website.com/sitemap_index.xml

    but didn't found.

    Is there any tool to extract sitemap?
     
  2. Veil123

    Veil123 Jr. VIP Jr. VIP

    Joined:
    Jun 13, 2012
    Messages:
    1,198
    Likes Received:
    1,706
    Gender:
    Male
    Occupation:
    Artificial Intelligence
    Location:
    Macedonia
    Home Page:
    Don't know for such a tool, but you can try to find the main sitemap through domain.com/sitemap/ and domain.com/sitemap.php as well
     
  3. developer1989

    developer1989 Regular Member

    Joined:
    Aug 31, 2019
    Messages:
    431
    Likes Received:
    48
    Gender:
    Male
    Thanks for the reply.

    tried both ways and no luck

    getting
    Page Not Found
     
  4. Gogol

    Gogol Jr. VIP Jr. VIP

    Joined:
    Sep 10, 2010
    Messages:
    6,437
    Likes Received:
    6,622
    Gender:
    Male
    Occupation:
    Programmer
    Location:
    Pale Blue Dot
    Look in the robots.txt and see if it is listed there. Also try site/sitemap.php, site/sitemap.xml, site/sitemap_index.xml and so on. Additionally, try looking into the source code of the site. If it uses a cms, chances are that the xml feed will be listed in the head part of the code. If none works, probably, the site doesn't have a sitemap.
    As for a tool which does this, I am not really sure but you can cook up something in python pretty quickly...
     
  5. BetaRayOLC

    BetaRayOLC BANNED BANNED

    Joined:
    Jun 22, 2019
    Messages:
    37
    Likes Received:
    9
    Gender:
    Male
    if is it de-indexed you probably won't on search engine, if not try this-->> site:domain.com inurl:sitemap
     
    • Thanks Thanks x 1
  6. Gogol

    Gogol Jr. VIP Jr. VIP

    Joined:
    Sep 10, 2010
    Messages:
    6,437
    Likes Received:
    6,622
    Gender:
    Male
    Occupation:
    Programmer
    Location:
    Pale Blue Dot
    Umm, nice dork actually. Never thought of this. :)
     
  7. GregFromMoonsy

    GregFromMoonsy Jr. VIP Jr. VIP

    Joined:
    Aug 12, 2018
    Messages:
    598
    Likes Received:
    251
    Gender:
    Male
    Location:
    European circus
    @developer1989

    Part of webmasters don't care about it at all.
    Where more experienced webmasters don't using sitemap, because it have a lot of cons in compare to advantages.

    At first it show all pages for all kind of web scanners, making it very simple to harvest.
    At second, Google was able crawl you site without sitemap for last 20 years. Today they have much more possibilities.
    Only what Google need - it's correct html code.

    This is why you not find sitemap for many websites.

    Best regards,
    Greg.
     
  8. BetaRayOLC

    BetaRayOLC BANNED BANNED

    Joined:
    Jun 22, 2019
    Messages:
    37
    Likes Received:
    9
    Gender:
    Male
    I thought Google knows everything :) :)
     
  9. Gogol

    Gogol Jr. VIP Jr. VIP

    Joined:
    Sep 10, 2010
    Messages:
    6,437
    Likes Received:
    6,622
    Gender:
    Male
    Occupation:
    Programmer
    Location:
    Pale Blue Dot
    Yeah, typo.. ;)
     
  10. davids355

    davids355 Moderator Staff Member Moderator Jr. VIP

    Joined:
    Apr 25, 2011
    Messages:
    15,864
    Likes Received:
    15,215
    Home Page:
    Thread moved.
     
  11. Blaqstar

    Blaqstar Newbie

    Joined:
    Jan 29, 2019
    Messages:
    11
    Likes Received:
    1
    Gender:
    Male
    Try the WayBack Machine webdotarchivedot org

    Pretty sure they have all sitemaps
     
  12. Zeckie

    Zeckie Junior Member

    Joined:
    Dec 11, 2015
    Messages:
    109
    Likes Received:
    60
    you can create sitemap of your target site
     
    • Thanks Thanks x 1
  13. GregFromMoonsy

    GregFromMoonsy Jr. VIP Jr. VIP

    Joined:
    Aug 12, 2018
    Messages:
    598
    Likes Received:
    251
    Gender:
    Male
    Location:
    European circus
    Try robots.txt file - there usually you'll find the sitemap file location which all kind of "SEO gurus" recommended ;)

    Cheers, Greg.
     
  14. developer1989

    developer1989 Regular Member

    Joined:
    Aug 31, 2019
    Messages:
    431
    Likes Received:
    48
    Gender:
    Male
    Thank you Greg for the reply.

    I'm planning to build website similar to this

    https://pincode.net.in/

    This website had thousand of pages and searching for pincode in Google giving result like this https://pincode.net.in/400080

    This website doesn't have sitemap or robots.txt files.

    So if i build website similar to this can i submit all my pincodes to google through yoast sitemap?

    like mywebsite.com/000001 to mywebsite.com/999999

    Google don't spam my website right?


    Tagging @Veil123 @Gogol @Blaqstar @Zeckie for furhter help/suggestions.
     
  15. adige1431

    adige1431 Junior Member

    Joined:
    Nov 11, 2014
    Messages:
    196
    Likes Received:
    31
    upload_2019-10-15_19-39-19.png

    What your main purpose? I mean, you can scrape and build your own instead
     
    • Thanks Thanks x 1
  16. developer1989

    developer1989 Regular Member

    Joined:
    Aug 31, 2019
    Messages:
    431
    Likes Received:
    48
    Gender:
    Male
    Thanks for your reply. I want to build similar site and my main concern is how can I index such a large pages in Google? Since each page contain very less data(pincode and the place only). Please help me
     
  17. sj707

    sj707 Regular Member

    Joined:
    Jan 7, 2013
    Messages:
    300
    Likes Received:
    93
    Location:
    London U.K.
    Generate a sitemap of your target site.
    xml-sitemaps.com
     
  18. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP

    Joined:
    Mar 21, 2013
    Messages:
    17,390
    Likes Received:
    16,550
    Occupation:
    BUM
    Location:
    QUARANTEEN
    Interesting case study there. That site ranks 13k on alexa.
     
  19. developer1989

    developer1989 Regular Member

    Joined:
    Aug 31, 2019
    Messages:
    431
    Likes Received:
    48
    Gender:
    Male
    This tool didn't worked. It is creating only 500 pages for free plan.

    My main concern is how can I index such a large pages in Google? Since each page contain very less data(pincode and the place only).
     
  20. adige1431

    adige1431 Junior Member

    Joined:
    Nov 11, 2014
    Messages:
    196
    Likes Received:
    31
    Very less data isn't even relevant. Google consider it provides helpful information for others since 2010 (based on site information)
    But you can try clone with more quality content each, I personally don't think sitemap will bring much difference these days