- Jun 30, 2014
- 13,741
- 5,218
I'm trying to find an accurate way of doing an exact match search query on domains for example to see how many results there are in a search engine to "volkswagenlover.tumblr.com" but on a large scale at the lowest cost.
Here are my findings so far
Google, didn't even bother trying because the scale I need to do it at wouldn't work with how good their blocking is, I've scraped google tons before.
Bing, they let you send ton's of requests but their exact match queries don't work
Yahoo, same as bing
DuckDuckGo, exact match for domains also doesn't work
Baidu, seems to have limited results for English queries
Yandex.ru, exact match queries work well with domains but I've never scraped it before on a large scale.
So before I start poking Yandex.ru does anyone know what their anti scraping is like? Bing just lets you go nuts, Google is the opposite, where does Yandex.ru sit?
Here are my findings so far
Google, didn't even bother trying because the scale I need to do it at wouldn't work with how good their blocking is, I've scraped google tons before.
Bing, they let you send ton's of requests but their exact match queries don't work
Yahoo, same as bing
DuckDuckGo, exact match for domains also doesn't work
Baidu, seems to have limited results for English queries
Yandex.ru, exact match queries work well with domains but I've never scraped it before on a large scale.
So before I start poking Yandex.ru does anyone know what their anti scraping is like? Bing just lets you go nuts, Google is the opposite, where does Yandex.ru sit?