1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to get list of a URL's sub-pages????

Discussion in 'Black Hat SEO' started by ThopHayt, Oct 6, 2014.

  1. ThopHayt

    ThopHayt Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 25, 2011
    Messages:
    5,396
    Likes Received:
    1,644
    I'm going to be designing a rebuild of a site (not mine) and I need a quick way of getting a list of EVERY little page of the website so that I can 301 pages that won't be included in the redesign.

    What is the best way to get a full list of all the sub-pages of a URL?
     
  2. SEO Power

    SEO Power Elite Member

    Joined:
    Jul 14, 2014
    Messages:
    2,637
    Likes Received:
    680
    Occupation:
    Self employed
    Location:
    Houston, TX
    Google is the fastest way to do that (if I understand your question correctly) since they index urls very fast and crawl websites deeply. Use the site:domain.com operator to get a list of all the pages in any domain name.
     
  3. Aty

    Aty Jr. VIP Jr. VIP

    Joined:
    Jan 27, 2011
    Messages:
    5,407
    Likes Received:
    3,695
    Home Page:
  4. TrafficWizard

    TrafficWizard Junior Member

    Joined:
    Aug 22, 2014
    Messages:
    160
    Likes Received:
    26
    Not the google, yes it will index pages but not all (robots restrictions + new pages not indexed yet) ! I suggest use some crawler with save to .txt option ;)

    Or 'httrack' > google this tool there will be a first link with main page of this tool designed for copy whole website and there are a lot of other tools as well.

    Remember some pages have subdomains there is good python subdomain scanner called knock.py google it up if you need it.
     
  5. madandi

    madandi Registered Member

    Joined:
    Aug 19, 2014
    Messages:
    91
    Likes Received:
    28
    Take a look at Screaming Frog SEO Spider. It's free, can give you the info you're after, plus a lot more. It's quite a handy little tool.
     
    • Thanks Thanks x 1
  6. fc-dh

    fc-dh Elite Member

    Joined:
    Oct 20, 2012
    Messages:
    1,588
    Likes Received:
    1,224
    Occupation:
    Blackhatting
    Location:
    Den Haag | Netherlands

    Still spamming the forums i see, you must be making big money because your on the forum all the time with your open door answers
     
  7. K.H.R

    K.H.R Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 9, 2013
    Messages:
    2,931
    Likes Received:
    953
    Gender:
    Male
    Occupation:
    Study + Outsourcing
    Location:
    BHW
    But its a manual task. Is there any tools to scrape the link from this search terms ( site:domain.com ) ?
     
  8. lord1027

    lord1027 Elite Member

    Joined:
    Sep 20, 2013
    Messages:
    3,174
    Likes Received:
    2,222
    +1 for Screaming Frog SEO Spider, it does the job well and fast.