I get a pretty good 1500 urls/s scraping bing lately, and get about 350-400 urls/s on google, but the problem is I get a ton of dupes scraping Bing. I am talking like 90% dupes.
Does anyone else have the same experience? Anyone found a good way to lower the amount of dupes you get when scraping bing for targets for GSA SER?
Does anyone else have the same experience? Anyone found a good way to lower the amount of dupes you get when scraping bing for targets for GSA SER?