[Advice Needed] Large Scale Web Scraping

Holzr

Holzr

Elite Member
Joined
Sep 4, 2016
Messages
2,236
Reaction score
2,666
Hi all,

I'm starting a project with a friend in which we will be needing to scrape web store data daily to upload to a database and display on our own site. I need to keep everything in categories also; so basically taking a carbon copy from other sites to use.

We will be adhering to T&C and robots.txt of the sites.

To use an example: https://www.diy.com/ (B&Q - A UK DIY, decorating and gardening supplies warehouse chain).

We would be looking at scraping all the categories and formatting everything so it's easy to upload to our own site.

I was thinking of scaping myself but I realise this is probably not ideal as I am looking to scrape hundreds-thousands of pages a day (scheduled) and I'' run into performance issues and IP blocking potentially.

So I'm left with using:

a) Cloud based scraping tool like Scrapinghub or Octoparse.
b) Using a scraping service company.

Question: Is it worth skipping straight to a scraping service or would I be better off hiring someone to write something like a scrapy code for each of the sites so I can upload that to something like scrapinghub to use instead. I'd imaging using a cloud scraper like scrapinghub is going to costs me a lot less then paying for the service and they set up their own crawler/scraper etc.

If anyone has any advice or recommendations to services/companies I'd be very grateful.

Thanks in advance.

Edit: Was not to sure on the best thread/forum to use. Please move if so. :)
 
Advertise on BHW
If it's one time work than hire scraping company or if you want to it regular than it's good to hire someone to write code for so you can run that whenever you need to scrape data
 
Array Experts said:
If it's one time work than hire scraping company or if you want to it regular than it's good to hire someone to write code for so you can run that whenever you need to scrape data
Click to expand...

We need data to be regularly updated daily (for price info. and stock etc.). Yeah I was leaning on the idea of hiring someone to write a script then using a cloud scraping platform like scrapinghub to do the work (easy to schedule and they use their own IP's etc.)

Thanks
 
Hire someone, that's a better option.
 
As I understand your thread your Unique selling proposition is that you scrape and consolidate data from others. I would suggest to keep the knowledge with you.
The algorithm to map and match categories between different shops etc. is the value of your company for the next few years.
The scraping is easy done but needs a lot of vps-performance and proxies etc.

For first setup I would suggest a simple setup to learn:

Step 1: get a VPS (e.g free AWS Tier, size doesn’t matter for now.)
Step 2: install a rotating proxy on vps (e.g. Tor roaring proxy docker image available)
Step 3: develop scraper for example in python and use celery for distributed work (need an rabbitmq server also available as docker image)
Step 4: put scraper worker on vps
Step 5: add worktasks (urls) to your local celery (later an own server for sure)
Step 6: let the worker do his business, store data and change random to a new Tor-IP
Step 7: consolidate received data (in Json or whatever)
Need more speed repeat step 1, 3 and 4

Be happy and name your first born child after me :-)

Best luck
Mf
 
Growth: while growing think about putting your vps setup (few pip3 installs, files, etc.) in an own docker container that you push into your own private registry. As an result you need only to install docker on the vps and pull your own image. If you have to change things (write more complex scraping functions, etc.) you only need to update the image and pull it again. So you reduce the work.

Later you could go with tools like rancher. The got a own distribution only for docker use. Install it on different vps and control the workers from a main console. Easier scaling and updates...
 
That's a lot of work to do and it's not something you can do by yourself. I would definitely hire someone to do it, if I were you.
 
I agree, the most time consuming when scraping is updating the html structure and rotating proxy.
If you're planning daily scraping, hiring is a smart move.
 
Hire someone with scraping knowledge, it's a better option if this is a one time job.
 
Holzr said:
Hi all,

I'm starting a project with a friend in which we will be needing to scrape web store data daily to upload to a database and display on our own site. I need to keep everything in categories also; so basically taking a carbon copy from other sites to use.

We will be adhering to T&C and robots.txt of the sites.

To use an example: https://www.diy.com/ (B&Q - A UK DIY, decorating and gardening supplies warehouse chain).

We would be looking at scraping all the categories and formatting everything so it's easy to upload to our own site.

I was thinking of scaping myself but I realise this is probably not ideal as I am looking to scrape hundreds-thousands of pages a day (scheduled) and I'' run into performance issues and IP blocking potentially.

So I'm left with using:

a) Cloud based scraping tool like Scrapinghub or Octoparse.
b) Using a scraping service company.

Question: Is it worth skipping straight to a scraping service or would I be better off hiring someone to write something like a scrapy code for each of the sites so I can upload that to something like scrapinghub to use instead. I'd imaging using a cloud scraper like scrapinghub is going to costs me a lot less then paying for the service and they set up their own crawler/scraper etc.

If anyone has any advice or recommendations to services/companies I'd be very grateful.

Thanks in advance.

Edit: Was not to sure on the best thread/forum to use. Please move if so. :)
Click to expand...


If you gonna have a lot of custom work, Hiring someone good would be a nice choice - I would suggest looking at scrapy or Django dynamic scraper - it has pretty good documentation - We managed to scrape 100's of websites with structured information in past with that
 
Simplest and best way (I did that many times):
1) make simple customized scraping script for a specified site
2) when scrapper finish, make simple parsing script to extract data you need directly into MySql.
 
You probably need rotating proxies, a crawler or even automation. Apify scrape in JavaScript have all these. There's Scrapy if you prefer Python, but I don't know if they have consultants that can help you set everything up.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top