Hi, I have a quick question. I want to do a price comparison for computer parts, but I'm unsure about the correct way to scrape data. Should I scrape the brand and all the product series below it, or should I simply scrape the titles from websites like Amazon and let the user choose? The reason I'm uncertain is that Amazon often lists the same product with different titles due to multiple sellers. Have anyone of you encountered this issue before, and if so, how did you handle it? I ask because when I try to organize products by brand, I find myself not scraping anything and having to manually input the information.
Scraping data for a price comparison website
- Thread starter evaworld9
- Start date
- Oct 24, 2022
- 1,135
- 1,378
I prefer to scrape the brand and all the product series below it and save the data into a CSV file then you can filter the duplicated data from your CSV
- Aug 7, 2011
- 1,577
- 1,432
You need way to identify product - lets say part id and manufacter name. For example for this part it would be 7C56-002R and MSI. Thats how you can easily find duplicate parts etc. Another approach is ASIN (Amazon Standard Identification Number).
Edit:
Doing proper scraper is easy part, you will get into real troubles later. How to keep up with price changes (rescraping daily?), how to scrape data in bulk. Proxies may burn your budget. Also for bulk scraping you need to find a way to scrape data without a browser, if you right now use selenium etc for scraping - dump this asap, thats not tool for bulk scraping.
Terimouse
Newbie
- Feb 14, 2021
- 35
- 21
You have to reverse engineer the private API the website is using if you want to scrape in bulk.
Don't forget to use proxies