Problems when scraping Amazon

4440

4440

Elite Member
Jr. Executive VIP
Jr. VIP
Joined
Jun 23, 2021
Messages
1,510
Reaction score
1,557
Hi guys,

I have a problem when scraping amazon with selenium in python.

The problem is that, sometimes the asin page does not show the actual content completely. It will not load/render some parts such as product description, product detail tables.

I tried scrolling down step by step and wait, but the missing part just won't load.

How to force it to load the full page?

** I have API and know how to use the API but I need to scrape the content without the API.
 
Advertise on BHW
You should watch the video with tucker Wilson on YouTube, store sellers who sell on other platforms within 15 minutes Amazon suddenly automatically indefinitely removes their for sale message unless they reinstate the price on all other platforms including Amazon.


Now if that guy's struggling to make a living I'm sure scraping Amazons gonna do wonders, infact scraping is the new espionage of ai, every company, business, research, institute is scraping but everyone who is buying up ai tech is also investigating in intrusion detection.


Come on dude what are you really doing.
 
Gants-P said:
You should watch the video with tucker Wilson on YouTube, store sellers who sell on other platforms within 15 minutes Amazon suddenly automatically indefinitely removes their for sale message unless they reinstate the price on all other platforms including Amazon.


Now if that guy's struggling to make a living I'm sure scraping Amazons gonna do wonders, infact scraping is the new espionage of ai, every company, business, research, institute is scraping but everyone who is buying up ai tech is also investigating in intrusion detection.


Come on dude what are you really doing.
Click to expand...
Come on dude what are you trying to say?
 
4440 said:
Come on dude what are you trying to say?
Click to expand...
Since your desperate I'll give you a free one, burpsuit pentest, or kalilinux can be installed on a usb nothing fancy, their practically scrapers, hacking, what ever you want.
 
Try increasing wait times, scrolling in intervals, or waiting for specific elements to load. To handle potential anti-bot measures, use random delays, rotate user agents, and employ proxies.
 
OnlineProxy said:
Try increasing wait times, scrolling in intervals, or waiting for specific elements to load. To handle potential anti-bot measures, use random delays, rotate user agents, and employ proxies.
Click to expand...
I use tones of random sleep, delay and implicit_wait and do multiple scroll downs, but the elements just don't load. It seems like amz knows you are bot and serves you different content on purpose. Even some of the already displayed/rendered content is different than what you see in real browsers.
 
4440 said:
Hi guys,

I have a problem when scraping amazon with selenium in python.

The problem is that, sometimes the asin page does not show the actual content completely. It will not load/render some parts such as product description, product detail tables.

I tried scrolling down step by step and wait, but the missing part just won't load.

How to force it to load the full page?

** I have API and know how to use the API but I need to scrape the content without the API.
Click to expand...
I think this should be in the programming section. I've notified the mods.
 
Try libraries like botsaurus that already have more human interactions built in.
 
You say that pages sometimes load. Try this logic: if selenium finds an element, follow scenario 1 and exit the cycle. If there is no content on the page, restart the page
 
4440 said:
Hi guys,

I have a problem when scraping amazon with selenium in python.

The problem is that, sometimes the asin page does not show the actual content completely. It will not load/render some parts such as product description, product detail tables.

I tried scrolling down step by step and wait, but the missing part just won't load.

How to force it to load the full page?

** I have API and know how to use the API but I need to scrape the content without the API.
Click to expand...

Not all product pages are the same on Amazon. There are different product category that are available per country, device type etc.
Could you please give some sample ASINs that gives you troubles?
Some things to consider:
- using different user agents (for example such that are suitable for Fire devices)
- using proxies for different countries (residential/mobile)
- You may not need a full browser/js automation like selenium, sometime a simple GET request and DOM parsing will get you what you need (i recently wrote a crawler/parser for ASIN pages for certain type of products - no JS rendering engine was needed)
 
newcat said:
Not all product pages are the same on Amazon. There are different product category that are available per country, device type etc.
Could you please give some sample ASINs that gives you troubles?
Some things to consider:
- using different user agents (for example such that are suitable for Fire devices)
- using proxies for different countries (residential/mobile)
- You may not need a full browser/js automation like selenium, sometime a simple GET request and DOM parsing will get you what you need (i recently wrote a crawler/parser for ASIN pages for certain type of products - no JS rendering engine was needed)
Click to expand...
i saved screenshots so I can see amazon actually served different content.
Anyways thanks for the reply.
I will re-enter this project soon and by then i'll try different solutions like driverless anti-detect and other methods. I think scraping AMZ is not that hard.
 
I tried scraping Amazon in a previous life before.....

They do all manner of things to combat this. It's super difficult.

They don't straight up ban you or anything... they just quietly introduce errors/rendering faults etc.

I would honestly reconsider what you are doing..especially if it depends on scraping Amazon, your idea may not be viable.

I also tried to build auto checkout using Selenium before and even that was difficult... which is crazy, considering I was trying to automate a way to give them money.
 
Hi, I am a little bit late but scraping the product page is the wrong way to go, you'll quickly be flagged by amazon and it takes a lot of resources. I currently have a project that bypasses all this by using private endpoints and returns all product info in +- 200 ms. Which part of the product page do you need? Good luck!
 
Demoniak said:
Hi, I am a little bit late but scraping the product page is the wrong way to go, you'll quickly be flagged by amazon and it takes a lot of resources. I currently have a project that bypasses all this by using private endpoints and returns all product info in +- 200 ms. Which part of the product page do you need? Good luck!
Click to expand...
i need all of them actually, every part helps, even all the reviews.
 
4440 said:
i need all of them actually, every part helps, even all the reviews.
Click to expand...
You can achieve this without having to scrape the web page. What your main goal behind this ? Botting products, Affiliate marketing, etc ?
 
Last edited:
Demoniak said:
You can achieve this without having to scrape the web page. What your main goal behind this ? Botting products, Affiliate marketing, etc ?
Click to expand...
It is mostly affiliate but i basically do need the entire content of the page.
 
Demoniak said:
You can achieve this without having to scrape the web page. What your main goal behind this ? Botting products, Affiliate marketing, etc ?
Click to expand...

You are allowed to share your methods btw :)

Are you using the Product Advertising API ?
 
paddy_proxies said:
You are allowed to share your methods btw :)

Are you using the Product Advertising API ?
Click to expand...
Unfortunalty this is a private method/endpoint, hope you can understand it. I don't use Product Advertising API as I am not even using it for affiliate marketing but I use it for botting.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top