1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to get the full list of URL's of my site in a neat list?

Discussion in 'Black Hat SEO' started by theindiaphile, Dec 10, 2010.

  1. theindiaphile

    theindiaphile Senior Member

    Joined:
    Jul 26, 2010
    Messages:
    831
    Likes Received:
    245
    I've got a wordpress site that it'd be useful to have a full list of URL's of in a tet document. But how the hell to get it! Scrapebox doesn't find even a fraction of them... And Yahoo Site explorer gives me a tsv file but its full of other crap and I don't know how to clean it to just url. Would really appreciate some tips on what must be so simple....
     
  2. flymethod

    flymethod Regular Member

    Joined:
    Oct 27, 2010
    Messages:
    214
    Likes Received:
    58
    google: xml-sitemaps - I believe this site will output a text file of urls.

    It's the first site. I can't post a link yet...
     
  3. FrozenSpade

    FrozenSpade Power Member

    Joined:
    Jul 20, 2008
    Messages:
    689
    Likes Received:
    331
    Occupation:
    IM
    Location:
    EU
    scrapebox with private proxies or no proxies at all (not recommended)!
    Use site:yoursite.com
    Export list to text or excel
     
  4. cyberzilla

    cyberzilla Elite Member Premium Member

    Joined:
    Nov 15, 2009
    Messages:
    2,208
    Likes Received:
    3,367
    Location:
    zeta reticuli
    There is an add-on called "sitemap scraper" in SB. Just add your sitemap URL over there and scrape...viola!!