1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Question about scraping - How to create an effective scraper?

Discussion in 'Black Hat SEO' started by Ampix0, Jan 15, 2014.

  1. Ampix0

    Ampix0 Power Member

    Joined:
    Jan 10, 2012
    Messages:
    525
    Likes Received:
    60
    Home Page:
    I have had experience with WebHarvy before and I absolutely love it (high recommend if anyone was wondering). However with this current situation (which I run into a lot actually) I can not use it.

    With a program like Webharvy, you can scrape multiple fields on a page and tell it where the "next page" button is. This only works if the page contains the information you need scraped.

    What I have is more like a directory.

    [​IMG]

    Each item in this list links to a page which I want to scrape. So I need to scrape the inner pages of everything in this table, then go to the next page and scrape those inner pages and so on.

    Any ideas?


    I basically want a clone of this Database for leads.
     
  2. dannyhw

    dannyhw Senior Member

    Joined:
    Jul 16, 2008
    Messages:
    980
    Likes Received:
    462
    Occupation:
    Software Engineer
    Location:
    New York City Burbs
    You should be able to do that in like 20 lines of javascript or python. Writing a universal scraper is damn near impossible but writing a specific one is actually easy. You should be able to get someone on elance to do it for a few bucks
     
  3. Beven

    Beven Elite Member

    Joined:
    Aug 30, 2011
    Messages:
    1,810
    Likes Received:
    937
    Location:
    United Kingdom
    Post this request in the WTB/HAF section and see what offers pop up :
     
  4. Ampix0

    Ampix0 Power Member

    Joined:
    Jan 10, 2012
    Messages:
    525
    Likes Received:
    60
    Home Page:
    I would probably buy something like this if it was for me, but unfortunately its for the job I work at in sales, and they would never give me a penny to spend on this kind of stuff, let alone understand it. What if I want to scrape the same information from a list of URLS. I can scrape the links first. Is there a simple way to do this with software that is pre-made?
     
  5. MrBeastsOnToast

    MrBeastsOnToast Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 17, 2011
    Messages:
    921
    Likes Received:
    536
    Location:
    The Internetz
    There will not be a premade bot for this. Your best bet is hiring a coder, or buying ubot and making it yourself.
     
  6. Schvamp

    Schvamp Power Member

    Joined:
    Feb 13, 2012
    Messages:
    684
    Likes Received:
    549
    Location:
    Hogwarts
    Is it only 3 rows x 23 pages as in the printscreen? Send me the details and I can do it for free.
    Edit; sent you a skype request.
     
    • Thanks Thanks x 1
    Last edited: Jan 16, 2014