1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Any Tool to extract all URLs of a Website ?

Discussion in 'Black Hat SEO Tools' started by webhostingproviders, Jun 1, 2017.

  1. webhostingproviders

    webhostingproviders Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 8, 2013
    Messages:
    1,479
    Likes Received:
    384
    Occupation:
    Internet Marketer
    Location:
    Planet Earth
    Home Page:
    Is there any tool through which we can extract website URLs at once i..e if a website has 300 pages and we want all the URLs in a notepad format - is there any tool available
     
  2. darulez

    darulez Jr. VIP Jr. VIP

    Joined:
    Mar 12, 2013
    Messages:
    3,067
    Likes Received:
    1,068
    Gender:
    Male
    Occupation:
    Doing Internet Warfare
    Location:
    Bad Neighborhood
  3. Repulsor

    Repulsor Power Member

    Joined:
    Jun 11, 2013
    Messages:
    774
    Likes Received:
    280
    Location:
    PHP Scripting ;)
    Doesnt scrapebox has an addon which does it? I think they have one. Link Harvester or something likely.

    I can make one too.How many pages for the website? And how many levels does it need to go deep?
     
  4. roki4ka

    roki4ka Senior Member

    Joined:
    Jun 20, 2016
    Messages:
    889
    Likes Received:
    649
    screaming frog
    scrapebox
     
  5. SBFactory

    SBFactory Newbie

    Joined:
    Mar 1, 2013
    Messages:
    3
    Likes Received:
    1
    Home Page:
    I have the tool. With it you can scrape up to 2 million URLs from a WEBsite. My Skype is sbfactory_support
     
  6. RightFootFanatic

    RightFootFanatic Regular Member

    Joined:
    May 31, 2015
    Messages:
    388
    Likes Received:
    216
    Occupation:
    DevOps
    Location:
    Whimsyshire
    you could try and grab the sitemap.xml

    otherwise scrapebox
     
  7. denchik

    denchik Registered Member

    Joined:
    Apr 5, 2016
    Messages:
    70
    Likes Received:
    14
    Gender:
    Male
    I can help you for free, PM me )
     
  8. dhia27

    dhia27 Jr. VIP Jr. VIP

    Joined:
    Jan 26, 2016
    Messages:
    529
    Likes Received:
    230
    Xenu's Link Sleuth might do it.
     
  9. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,751
    Likes Received:
    11,417
    Occupation:
    COINZ
    Location:
    BUYAH
    Home Page:
    Check the site's /robots.txt for a sitemap entry. If it exists, you can simply take that list and scrape it.
     
  10. soccerlover

    soccerlover Jr. VIP Jr. VIP

    Joined:
    Jun 12, 2014
    Messages:
    3,399
    Likes Received:
    1,773
    Gender:
    Male
    Occupation:
    Seo Analyst :D
    Location:
    ♥♥♥ BHW ♥♥♥
    Home Page:
    Step 1 Visit
    Code:
     https://xmlsitemapgenerator.org/ 
    Step 2 Enter domain name (The one for which you required URL)

    Step 3 Enter Email ID

    Step 4 Agree all terms and conditions

    Step 5 Click on Generate Sitemap

    Step 6 After generating sitemap you'll see the output in following options

    XML Sitemap
    RSS Sitemap
    HTML Sitemap
    Text Sitemap
    Robots file

    Click on download/view infront of Text Sitemap..

    Boom!!! You'll get all the URL's :)
     
    • Thanks Thanks x 6
  11. loopline

    loopline Jr. VIP Jr. VIP

    Joined:
    Jan 25, 2009
    Messages:
    3,875
    Likes Received:
    2,058
    Gender:
    Male
    Home Page:
    Yes its in the main scrapebox window under grab/check >> grab urls by crawling a site (upper right hand quadrant of the main scrpaebox window).

    The link extractor addon will only go 1 level at a time, so you just have to keep loading the levels back in and going,but you can also do it that way if you want to get a lot of sites at once.
     
    • Thanks Thanks x 1
  12. Mikaelo

    Mikaelo Regular Member

    Joined:
    Nov 10, 2013
    Messages:
    461
    Likes Received:
    109
    Location:
    Australia
    Your options are -
    Screaming frog - best
    Scrapebox
    Using SERP redux to pull indexed pages
    SEO tools for excel
    DeepCrawl
    Pulling URL's from sitemap - this will not be as good as the above as some URL's will not be in there
    Export out of WMT or GA
     
  13. themrgarycole

    themrgarycole Newbie

    Joined:
    Jun 2, 2017
    Messages:
    7
    Likes Received:
    0
    Gender:
    Male
    I remember seeing a guide where you can have an addon in scrapebox that does this. Forgot what it's called though. But I've seen great reviews about screaming frog too. You can try them to see which works best for you.
     
  14. heopas

    heopas Jr. VIP Jr. VIP

    Joined:
    Jun 22, 2012
    Messages:
    146
    Likes Received:
    45
    • Thanks Thanks x 2
  15. imagetypers

    imagetypers Jr. VIP Jr. VIP

    Joined:
    Dec 30, 2010
    Messages:
    462
    Likes Received:
    65
    Occupation:
    captcha decoding service provider
    Home Page:
  16. payal23j

    payal23j Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2009
    Messages:
    915
    Likes Received:
    115
    Gender:
    Female
    Location:
    VA and expert in data entry !!
    I am looking for free scraper through which i can scrap email id.
     
  17. Array Experts

    Array Experts Jr. VIP Jr. VIP

    Joined:
    Sep 26, 2015
    Messages:
    620
    Likes Received:
    241
    Occupation:
    I Don't Know
    Location:
    Internet
    Home Page:
    Try This Tool
    Code:
    http://www.supagrowth.com/EmailScraper
    Note : I Have Not Used That Tool But I Have Used His Other Tools.
     
  18. payal23j

    payal23j Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2009
    Messages:
    915
    Likes Received:
    115
    Gender:
    Female
    Location:
    VA and expert in data entry !!
    Okay, thanks for the info. Let me try.
     
  19. aaron416

    aaron416 Registered Member

    Joined:
    Nov 13, 2010
    Messages:
    50
    Likes Received:
    2
    Gender:
    Male
    invest in scrapebox.
     
  20. yeahhub

    yeahhub Registered Member

    Joined:
    Jun 5, 2017
    Messages:
    83
    Likes Received:
    4
    Gender:
    Male
    You can try this thing with BURP suite which can easily crawl all the internal links and can save them in a notepad file too.