How to get the full list of URL's of my site in a neat list?

Discussion in 'Black Hat SEO' started by theindiaphile, Dec 10, 2010.

  1. theindiaphile

    theindiaphile Senior Member

    Joined:
    Jul 26, 2010
    Messages:
    831
    Likes Received:
    245
    I've got a wordpress site that it'd be useful to have a full list of URL's of in a tet document. But how the hell to get it! Scrapebox doesn't find even a fraction of them... And Yahoo Site explorer gives me a tsv file but its full of other crap and I don't know how to clean it to just url. Would really appreciate some tips on what must be so simple....
     
  2. flymethod

    flymethod Regular Member

    Joined:
    Oct 27, 2010
    Messages:
    214
    Likes Received:
    58
    google: xml-sitemaps - I believe this site will output a text file of urls.

    It's the first site. I can't post a link yet...
     
  3. FrozenSpade

    FrozenSpade Power Member

    Joined:
    Jul 20, 2008
    Messages:
    705
    Likes Received:
    336
    Occupation:
    IM
    Location:
    EU
    scrapebox with private proxies or no proxies at all (not recommended)!
    Use site:yoursite.com
    Export list to text or excel
     
  4. cyberzilla

    cyberzilla Elite Member Premium Member

    Joined:
    Nov 15, 2009
    Messages:
    2,224
    Likes Received:
    3,373
    Location:
    zeta reticuli
    There is an add-on called "sitemap scraper" in SB. Just add your sitemap URL over there and scrape...viola!!