1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Site scraper?

Discussion in 'Black Hat SEO Tools' started by turbopugsleylx, Jan 11, 2010.

  1. turbopugsleylx

    turbopugsleylx Jr. VIP Jr. VIP Premium Member

    Joined:
    Jun 6, 2008
    Messages:
    3,150
    Likes Received:
    965
    Occupation:
    www.xgcmedia.com
    Location:
    www.xgcmedia.com
    Home Page:
    Hey guys I am looking for a tool or website that will scrape a site and input all the pages that site contains...any ideas? I have looked around but found nothing...

    -Turbo
     
  2. CyberWizard

    CyberWizard Junior Member

    Joined:
    Oct 21, 2009
    Messages:
    121
    Likes Received:
    107
    Home Page:
    Try httrack. it is a full website copier.
     
  3. kerxupt

    kerxupt Newbie

    Joined:
    Nov 4, 2009
    Messages:
    17
    Likes Received:
    0
    If you're running Windows, just like CyberWizard mentioned:
    + http://www.httrack.com/

    If you're running MacOSX (even iPhone version), try:
    + http://www.sitesucker.us/home.html

    If you're running Linux or BSD, try:
    + http://www.gnu.org/software/wget/

    If you're actually trying to extract specific parts of a certain website, try using iMacros (http://www.iopus.com/iMacros/)
     
  4. turbopugsleylx

    turbopugsleylx Jr. VIP Jr. VIP Premium Member

    Joined:
    Jun 6, 2008
    Messages:
    3,150
    Likes Received:
    965
    Occupation:
    www.xgcmedia.com
    Location:
    www.xgcmedia.com
    Home Page:
    Some of these are good...but I am looking for something basic...something that just scrapes the site and give u a list of links/pages within it....anything else?
     
  5. catman08

    catman08 Junior Member

    Joined:
    Jan 11, 2008
    Messages:
    171
    Likes Received:
    109
    Occupation:
    IM
    Location:
    Europe
    If you know some php - i could give you my classes i programmed once. with those classes it is a breeze to create any scraper and data analyzer you need.

    Let me know via PM if you are interested.
    Cheers
    catman
     
    • Thanks Thanks x 1
  6. CoyoteAssassin

    CoyoteAssassin Elite Member

    Joined:
    Jan 3, 2010
    Messages:
    1,862
    Likes Received:
    3,906
    Occupation:
    Full Time IMer
    Location:
    USA
  7. turokman555

    turokman555 Newbie

    Joined:
    Jan 8, 2010
    Messages:
    43
    Likes Received:
    1
    I can do some custom scraping for you using python for a fee.. PM me...

    http://www.url-directory.org
     
  8. turbopugsleylx

    turbopugsleylx Jr. VIP Jr. VIP Premium Member

    Joined:
    Jun 6, 2008
    Messages:
    3,150
    Likes Received:
    965
    Occupation:
    www.xgcmedia.com
    Location:
    www.xgcmedia.com
    Home Page:
    You guys know any online only sites that do this?
     
  9. haridada

    haridada Senior Member

    Joined:
    Oct 9, 2008
    Messages:
    996
    Likes Received:
    382
    Location:
    Chennai