1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How can I extract all URLs from my site?

Discussion in 'Black Hat SEO' started by schleprock, Apr 29, 2017.

  1. schleprock

    schleprock Regular Member

    Joined:
    Apr 10, 2014
    Messages:
    467
    Likes Received:
    33
    How can I get all the URLs to my site?
    I can go to my sitemap and get all the posts and pages but how can I get all the image links? WP hosts my images for me so each image has a URL.

    Anyway.

    I looked at link extractor for scrapebox:
    http://www.scrapebox.com/internal-and-external-link-extractor

    but could not quite figure it out.

    Is there a way to get all the URLs (whether it is Scrapebox or not)

    I just thought SB Would be easy because then I could easily delete all the URLs with "/tag" and stuff like that. Stuff that I don't want to a build a link to.
     
  2. danixD

    danixD Supreme Member

    Joined:
    Nov 29, 2014
    Messages:
    1,245
    Likes Received:
    551
    Occupation:
    Almost there
    Screaming from has that feature
     
    • Thanks Thanks x 1
  3. loopline

    loopline Jr. VIP Jr. VIP

    Joined:
    Jan 25, 2009
    Messages:
    3,875
    Likes Received:
    2,058
    Gender:
    Male
    Home Page:
    htttrack is a free program that might work for you, scrapebox will extract the pages, but its going to ignore image links etc...
     
  4. BassTrackerBoats

    BassTrackerBoats Super Moderator Staff Member Moderator Jr. VIP

    Joined:
    Mar 10, 2010
    Messages:
    17,331
    Likes Received:
    31,796
    Occupation:
    Selling CPA Sites
    Location:
    Not England
    Home Page:
    You can also use screaming frog to do that.

    j/k - I'm sure @danixD meant screaming frog.

     
  5. Jomasdf

    Jomasdf Jr. VIP Jr. VIP

    Joined:
    Jul 7, 2012
    Messages:
    458
    Likes Received:
    158
    Occupation:
    C# dev
    Location:
    Sweden
    Home Page:
    Depending on what you are looking for. If you only want the links and are willing to do it semi manually. Get an xpath addon like this and go to each of your pages giving an xpath like //a/@href.
     
  6. djp371

    djp371 Elite Member

    Joined:
    Apr 20, 2009
    Messages:
    2,082
    Likes Received:
    350
    Occupation:
    Internet marketing
    Location:
    United States
    Home Page:
    Screaming Frog is a good tool as it can crawl the entire site. Note that it only generates 500 URLs if you are not using the PRO version.
     
  7. harshmarsh

    harshmarsh Junior Member

    Joined:
    Aug 15, 2014
    Messages:
    128
    Likes Received:
    7
    Occupation:
    Website Development
    Location:
    India
  8. thebat1980

    thebat1980 Power Member

    Joined:
    Nov 19, 2013
    Messages:
    528
    Likes Received:
    102
    Sreamingfrog?
     
  9. clevelandslim

    clevelandslim Jr. VIP Jr. VIP

    Joined:
    Nov 15, 2008
    Messages:
    1,972
    Likes Received:
    191
    Page on webmasterworld.com
    Screaming Frog SEO Spider Tool

    These are just a free ones, there are many out there, not all will actually get you the same results so keep that in mind.
     
  10. Luco

    Luco Regular Member

    Joined:
    Dec 7, 2016
    Messages:
    369
    Likes Received:
    119
    Gender:
    Male
    Or search about curL on google
     
  11. jamesbond0007

    jamesbond0007 Newbie

    Joined:
    Jul 9, 2009
    Messages:
    2
    Likes Received:
    0
    I think u can check this link it will help bulkdachecker.com/url-extractor/