Who's scraped yandex.com before?

jamie3000

Elite Member
Executive VIP
Jr. VIP
Joined
Jun 30, 2014
Messages
13,741
Reaction score
5,218
I'm trying to find an accurate way of doing an exact match search query on domains for example to see how many results there are in a search engine to "volkswagenlover.tumblr.com" but on a large scale at the lowest cost.

Here are my findings so far

Google, didn't even bother trying because the scale I need to do it at wouldn't work with how good their blocking is, I've scraped google tons before.

Bing, they let you send ton's of requests but their exact match queries don't work

Yahoo, same as bing

DuckDuckGo, exact match for domains also doesn't work

Baidu, seems to have limited results for English queries

Yandex.ru, exact match queries work well with domains but I've never scraped it before on a large scale.

So before I start poking Yandex.ru does anyone know what their anti scraping is like? Bing just lets you go nuts, Google is the opposite, where does Yandex.ru sit?
 
I can't speak directly on Yandex, but I've hit a lot of barriers effectively scraping Craigslist for a client. Residential IPs within the US are what I have the best luck on by a pretty wide margin. If I were you, I'd try my luck with the same except with Russian IP's. Local addresses always seem to fly under the radar better than foreign.

HTH
 
Back
Top