Anyone manage to scrape bing?

spectrejoe

Elite Member
Jr. VIP
Joined
Sep 25, 2013
Messages
4,705
Reaction score
2,701
Has anyone managed to scrape bing lately using scripts? It seems to only grab results from the first page exclusively even if you try to do pagination (going to the next page)

So even if u go to page 10 the script will still only get the first page results because Bing detects you as a "bot".

I'm using python to code my scripts, I've tried requests, I've tried playwright and the only one that actually works is running playwright but without being headless. Otherwise every other option seems to be triggering the anti botting measures of Bing.
 
I'm using A-Parser for scraping nowadays and it's scraping multiple Bing results pages with no issue - I think they use Playwright in at least part of their app, it's in their dist directory along with Puppeteer. If you're still having issues you can maybe give them a try.
 
Back
Top