1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Site scraper?

Discussion in 'Black Hat SEO Tools' started by turbopugsleylx, Jan 11, 2010.

  1. turbopugsleylx

    turbopugsleylx Jr. VIP Jr. VIP Premium Member

    Joined:
    Jun 6, 2008
    Messages:
    3,216
    Likes Received:
    968
    Occupation:
    www.xgcmedia.com
    Location:
    www.xgcmedia.com
    Home Page:
    Hey guys I am looking for a tool or website that will scrape a site and input all the pages that site contains...any ideas? I have looked around but found nothing...

    -Turbo
     
  2. CyberWizard

    CyberWizard Junior Member

    Joined:
    Oct 21, 2009
    Messages:
    121
    Likes Received:
    107
    Home Page:
    Try httrack. it is a full website copier.
     
  3. kerxupt

    kerxupt Newbie

    Joined:
    Nov 4, 2009
    Messages:
    17
    Likes Received:
    0
    If you're running Windows, just like CyberWizard mentioned:
    + http://www.httrack.com/

    If you're running MacOSX (even iPhone version), try:
    + http://www.sitesucker.us/home.html

    If you're running Linux or BSD, try:
    + http://www.gnu.org/software/wget/

    If you're actually trying to extract specific parts of a certain website, try using iMacros (http://www.iopus.com/iMacros/)
     
  4. turbopugsleylx

    turbopugsleylx Jr. VIP Jr. VIP Premium Member

    Joined:
    Jun 6, 2008
    Messages:
    3,216
    Likes Received:
    968
    Occupation:
    www.xgcmedia.com
    Location:
    www.xgcmedia.com
    Home Page:
    Some of these are good...but I am looking for something basic...something that just scrapes the site and give u a list of links/pages within it....anything else?
     
  5. catman08

    catman08 Junior Member

    Joined:
    Jan 11, 2008
    Messages:
    170
    Likes Received:
    109
    Occupation:
    IM
    Location:
    Europe
    If you know some php - i could give you my classes i programmed once. with those classes it is a breeze to create any scraper and data analyzer you need.

    Let me know via PM if you are interested.
    Cheers
    catman
     
    • Thanks Thanks x 1
  6. CoyoteAssassin

    CoyoteAssassin Elite Member

    Joined:
    Jan 3, 2010
    Messages:
    1,862
    Likes Received:
    3,933
    Occupation:
    Full Time IMer
    Location:
    USA
  7. turokman555

    turokman555 Newbie

    Joined:
    Jan 8, 2010
    Messages:
    43
    Likes Received:
    1
    I can do some custom scraping for you using python for a fee.. PM me...

    http://www.url-directory.org
     
  8. turbopugsleylx

    turbopugsleylx Jr. VIP Jr. VIP Premium Member

    Joined:
    Jun 6, 2008
    Messages:
    3,216
    Likes Received:
    968
    Occupation:
    www.xgcmedia.com
    Location:
    www.xgcmedia.com
    Home Page:
    You guys know any online only sites that do this?
     
  9. haridada

    haridada Senior Member

    Joined:
    Oct 9, 2008
    Messages:
    1,000
    Likes Received:
    383
    Location:
    Chennai