Question about scraping - How to create an effective scraper?

I have had experience with WebHarvy before and I absolutely love it (high recommend if anyone was wondering). However with this current situation (which I run into a lot actually) I can not use it.

With a program like Webharvy, you can scrape multiple fields on a page and tell it where the "next page" button is. This only works if the page contains the information you need scraped.

What I have is more like a directory.



Each item in this list links to a page which I want to scrape. So I need to scrape the inner pages of everything in this table, then go to the next page and scrape those inner pages and so on.

Any ideas?


I basically want a clone of this Database for leads.

You should be able to do that in like 20 lines of javascript or python. Writing a universal scraper is damn near impossible but writing a specific one is actually easy. You should be able to get someone on elance to do it for a few bucks

Post this request in the WTB/HAF section and see what offers pop up :

I would probably buy something like this if it was for me, but unfortunately its for the job I work at in sales, and they would never give me a penny to spend on this kind of stuff, let alone understand it. What if I want to scrape the same information from a list of URLS. I can scrape the links first. Is there a simple way to do this with software that is pre-made?

Ampix0 said:

I would probably buy something like this if it was for me, but unfortunately its for the job I work at in sales, and they would never give me a penny to spend on this kind of stuff, let alone understand it. What if I want to scrape the same information from a list of URLS. I can scrape the links first. Is there a simple way to do this with software that is pre-made?

Click to expand...

There will not be a premade bot for this. Your best bet is hiring a coder, or buying ubot and making it yourself.

Is it only 3 rows x 23 pages as in the printscreen? Send me the details and I can do it for free.
Edit; sent you a skype request.