1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Site Scraper

Discussion in 'Black Hat SEO Tools' started by addbravo, Feb 14, 2012.

  1. addbravo

    addbravo Newbie

    Joined:
    Jun 20, 2010
    Messages:
    11
    Likes Received:
    0
    Hello peeps,

    What's the best site scraper out there? Looking for programs that can scrape any site; not just article directories. Thanks for the help :D
     
  2. blackhatchampion

    blackhatchampion Senior Member

    Joined:
    Dec 29, 2010
    Messages:
    1,068
    Likes Received:
    550
    Location:
    Earth
    I am looking same here.
     
  3. addbravo

    addbravo Newbie

    Joined:
    Jun 20, 2010
    Messages:
    11
    Likes Received:
    0
    No one .... really? :eek:
     
  4. christoss1959

    christoss1959 Senior Member

    Joined:
    Nov 25, 2010
    Messages:
    894
    Likes Received:
    1,150
    Home Page:
    Do you want a software than can scrap a complete website?
    Or do you want a software that can scrap the text of a website?
     
  5. kokoloko75

    kokoloko75 Elite Member

    Joined:
    Jan 1, 2011
    Messages:
    1,628
    Likes Received:
    1,936
    Occupation:
    Design director
    Location:
    Paris (France)
    HTTrack :
    Code:
    http://www.httrack.com/
    Beny
     
    • Thanks Thanks x 1
  6. addbravo

    addbravo Newbie

    Joined:
    Jun 20, 2010
    Messages:
    11
    Likes Received:
    0
    i am looking for a software that can scrap the text of a website .... any ideas?
     
  7. rallow

    rallow BANNED BANNED

    Joined:
    Aug 11, 2011
    Messages:
    138
    Likes Received:
    45
    You need a custom bot if you want to just scrap certain text or info from a site send me a pm i can help
     
  8. davids355

    davids355 Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 25, 2011
    Messages:
    8,805
    Likes Received:
    6,372
    Home Page:
    Yea you need something built for that-- need some curl and regex:)
     
  9. searchrecognition

    searchrecognition Newbie

    Joined:
    May 5, 2011
    Messages:
    3
    Likes Received:
    0
    try irobot - and if your a programmer you can get it to do loads ?
     
  10. doxxx

    doxxx BANNED BANNED

    Joined:
    Feb 13, 2011
    Messages:
    127
    Likes Received:
    61
    there is no tool out there to scrape any site :)... you need a custom bot ... you may find (or make) a bot that will scrape all text from a site but there is no tool that will scrape onlt the content of an article or a description from any site out there... Sites are different and you need to instruct your bot where to find (and how to find) the text in a page
     
  11. jkwilson78

    jkwilson78 Regular Member Premium Member

    Joined:
    Jun 24, 2010
    Messages:
    224
    Likes Received:
    311
    Like others have said, there is no tool that can scrape any site from out of the box. But there are tool you can use to scrape practically any site you just have to configure the software to do it.

    One of the easiest ones I have used (and the cheapest for what it can do) is web content extractor.
    http://www.newprosoft.com/web-content-extractor.htm

    They have a version that rotates proxies you just have to ask them for it and their support is awesome. There have been a few sites I've had trouble scraping and after contacting support with the details they have set up the scraper for me and even customized the software itself to accommodate the site I was trying to scrape.
     
  12. cledin

    cledin Registered Member

    Joined:
    Dec 10, 2012
    Messages:
    67
    Likes Received:
    50
    Occupation:
    CEO
    Location:
    Boston USA
    Home Page:
    just use some php and write a scraper, i build a few before not to hard .
     
  13. paulblack2025

    paulblack2025 Newbie

    Joined:
    Jul 8, 2016
    Messages:
    15
    Likes Received:
    1
    Gender:
    Male
    Any site??
    I'm using Octoparse | Web Scraping Software.
    But I don't think it will work for any sites.
    Writing a scraper is kinda hard for me.
     
  14. Tatterdemalion

    Tatterdemalion Newbie

    Joined:
    Jan 8, 2016
    Messages:
    37
    Likes Received:
    5
    Occupation:
    Bot Programmer
    Location:
    Googleplex
    hide me
    :p :D
     
  15. dwpg002

    dwpg002 Senior Member

    Joined:
    Dec 29, 2008
    Messages:
    919
    Likes Received:
    47
    for what purpose you need to scrape. How you need the scraper to work. You need to scrape for keyword or full site content scraping
     
  16. extremeboy

    extremeboy Jr. VIP Jr. VIP

    Joined:
    Jul 8, 2010
    Messages:
    2,993
    Likes Received:
    648
    Occupation:
    World Best RANK Tracker SERPCloud.com
    Home Page:
    That's the beauty of scraping you mostly need different scripting for each sites which are big sites like fb,G,Yt etc due to their different structures,securities etc.

    And for small sites using Wp,Joomla,etc mostly can be scrapped with once code and scrapping method normally.