[Suggestions needed] eCommerce Products Search Engine

Hello,

I am creating this thread with the intention to get some useful insights from the community members who have at least
once faced similar issues and selected one or another way to solve them.

Problem: Imagine you want to build some sort for eCommerce products aggregator (search engine for products) from stores like Amazon, eBay, Macys, Newegg and etc..
I mean something similar to Google Shopping, but, of course, at smaller scale. This involves tens or hundreds of millions of products that
need to be scraped and aggregated daily.

Main issues:

• Right method to get product catalog data. We all know that we should only fallback to the scraping if there is no possibility to utilize APIs. However it is extremely rare for retailers to offer product catalog APIs. Or am I missing something? For instance, Amazon product data. I know there is Amazon Affiliate API but I am not sure this API involves product catalog data for millions of products stored in Amazon. Also, Amazon Affiliate API is available only to verified and accepted partners who are already established businesses and generate consistent volumes of traffic and leads. I do not have this possibility yet as there is no production system so far. So the only option is scraping the data as I am now doing.

• Quality of scraped products data. Since in (1.) I made a conclusion that it is not possible to utilize APIs, scraped product data will always be in much more poor quality. So far I have selected the method only to scrape product search result pages and not go to the concrete product pages. This leaves me with the few data points of product data:
  • Title (often very long and machine-generated from various product attributes)
  • Price
  • Brand (sometimes)
  • Discounted price (sometimes)
  • Short description (sometimes)
  • Product image(s)

Later I am using full-text search engine that I offload product data to, but this poor product data leads to poor search relevancy when searching for concrete models, brands. So do you think it is essential to be scraping concrete product page to extract more data points about the product? Would this contribute much to increasing search relevancy later on? I mean I guess it could work if I manage to extract such data points as model or brand which often are missing in product search results page grid.

• Correct storage type for product data. Since product data is in varying forms, it should not be relational for sure, so I have picked Cloud-provisioned PaaS service for NoSQL database for now. But I have worries it will cost too much when I consider tens of hundreds of millions of products updated daily. Should I provision infrastructure manually, like hosting MongoDB on my own VPS?

• Scalability issues of headless browser-based scraper

Since most of retailers use dynamic Javascript based data loading mechanics and lazy-loading tactics when scrolling, I am forced to use browser emulation technologies to get data.
For that purpose I have selected puppeteer with stealth-plugin. I am currently facing quite a big scalability issues when running multiple puppeteer instances. My architecture for scraper
looks something like that:

• API - accepting URLs to be stored for scraping
• SQL database - storing URLs to scrape and scraping statuses.
• Coordinator - central orchestrating app which runs in a loop and checks for existing URLs to scrape in database. Then it lookups for available proxies in Proxy service and later on calls Puppeteer scrape. This is very poor component as I cannot easily scale it with Kubernetes (various race-condition issues arise with DB)
• Proxy service - manages available proxies and their last usage times for concrete domains.
• Express API - Node.JS application containing Puppeteer code. Puppeteer is not hosted in this app but instead app is connecting to separately hosted Puppeteer instances via DevTools Protocol (web socket connection) using puppeteer.connect().
• Puppeteer - docker image containing puppeteer executable along with useful configuration values for it.
• Kubernetes - whole infrastructure is contained in Kubernetes cluster as dockerised microservices. So I could easily adjust scale for separate components.

What do you think about this architecture? My main concern when it comes to scalability is having this central SQL storage and non-scalable Coordinator
component which obviously has some vertical-scaling limits. Should I seek for some alternative design solution? Like utilizing messaging brokers and avoiding
SQL-based and coordinator-based solution?

• Avoiding blocks from eCommerce sites

I have already observed quite frequent blocks even though I currently use these things to lower chances of getting blocked:

• Using 4g LTE residential proxies
• Applying some sort of politeness policy for concrete domain (not scraping more frequently than 10 seconds)
• Using puppeteer-stealth-plugin

Would you have something to suggest to avoid these frequent detections and blocks?


As I mentioned, if you would have any experience with at least one of these concerns I mentioned above, I would be more than happy to hear experience/insights/suggestions from
you here