Site scraper?

Discussion in 'Black Hat SEO Tools' started by turbopugsleylx, Jan 11, 2010.

  1. turbopugsleylx

    turbopugsleylx Jr. VIP Jr. VIP

    Joined:
    Jun 6, 2008
    Messages:
    3,331
    Likes Received:
    970
    Occupation:
    www.xgcmedia.com
    Location:
    www.xgcmedia.com
    Home Page:
    Hey guys I am looking for a tool or website that will scrape a site and input all the pages that site contains...any ideas? I have looked around but found nothing...

    -Turbo
     
  2. CyberWizard

    CyberWizard Junior Member

    Joined:
    Oct 21, 2009
    Messages:
    121
    Likes Received:
    108
    Try httrack. it is a full website copier.
     
  3. kerxupt

    kerxupt Newbie

    Joined:
    Nov 4, 2009
    Messages:
    17
    Likes Received:
    0
    If you're running Windows, just like CyberWizard mentioned:
    + http://www.httrack.com/

    If you're running MacOSX (even iPhone version), try:
    + http://www.sitesucker.us/home.html

    If you're running Linux or BSD, try:
    + http://www.gnu.org/software/wget/

    If you're actually trying to extract specific parts of a certain website, try using iMacros (http://www.iopus.com/iMacros/)
     
  4. turbopugsleylx

    turbopugsleylx Jr. VIP Jr. VIP

    Joined:
    Jun 6, 2008
    Messages:
    3,331
    Likes Received:
    970
    Occupation:
    www.xgcmedia.com
    Location:
    www.xgcmedia.com
    Home Page:
    Some of these are good...but I am looking for something basic...something that just scrapes the site and give u a list of links/pages within it....anything else?
     
  5. catman08

    catman08 Junior Member

    Joined:
    Jan 11, 2008
    Messages:
    170
    Likes Received:
    109
    Occupation:
    IM
    Location:
    Europe
    If you know some php - i could give you my classes i programmed once. with those classes it is a breeze to create any scraper and data analyzer you need.

    Let me know via PM if you are interested.
    Cheers
    catman
     
    • Thanks Thanks x 1
  6. CoyoteAssassin

    CoyoteAssassin Elite Member

    Joined:
    Jan 3, 2010
    Messages:
    1,869
    Likes Received:
    3,941
    Occupation:
    Full Time IMer
    Location:
    USA
  7. turokman555

    turokman555 Newbie

    Joined:
    Jan 8, 2010
    Messages:
    43
    Likes Received:
    1
    I can do some custom scraping for you using python for a fee.. PM me...

    http://www.url-directory.org
     
  8. turbopugsleylx

    turbopugsleylx Jr. VIP Jr. VIP

    Joined:
    Jun 6, 2008
    Messages:
    3,331
    Likes Received:
    970
    Occupation:
    www.xgcmedia.com
    Location:
    www.xgcmedia.com
    Home Page:
    You guys know any online only sites that do this?
     
  9. haridada

    haridada Senior Member

    Joined:
    Oct 9, 2008
    Messages:
    1,000
    Likes Received:
    383
    Location:
    Chennai