Looking for recomendation for a scraper website

Hi there,
I would like to create a deal or search engine type website which will scrape data from other e-com websites and show the result on my website. my plan is to monetize the site using an affiliate program.

Most of the other website doesn't have API so I am a little confused about which type of program or code I will use to scrape data. Do I need to store all data on my server-side or I can make the result real-time?

Can you recommend any place to find the programmer for this job? I have my front end ready.


Thanks in advance.

you could do it either way. Doing it real time will be slower but will be more up to date. Faster will be to scrape the sites and store the data to a database you can query, but you will need to be constantly scraping the entire site which will raise it's own challenges

Can you recommend any place to find the programmer for this job? I have my front end ready.

Click to expand...

Hire a freelancer on BHW
Or any other freelance site

import.io => make a scraper yourself, it is easy
scrapingrobot.com => they have some built in modules

Hi,
I can assist with some high level design if you like, besides that I've vast experience with web scraper
I found scrapy to be a robust framework for crawlers creation, you can either code your own spiders
with their respective pipelines and middlewares, keep in mind like noted above that can be a very challenging task considering that:
1. Some websites have in place a guarding mechanism to prevent scarping.
2. Of course you should respect fair crawling (robots.txt).
3. Website schema changes can break your spiders crawls and effect accurate data extraction.

and some more obstacles along the way.
By the way scrapy is both open source project and a cloud based sass as well at scrapinghub.com
you can choose your own way.

Success