Using TOR as opposed to VPS/Proxies for Web Scraping?

[protonx85]

There is some talk of this online, but surprisingly very little talk or discussion.

As far as staying anonymous goes, correct me if I'm wrong, but isn't TOR the most powerful in this regard?

That, and it is free.

Anyone have any thoughts or experience with this?

I've just recently started delving more into web scraping.. none of the sites I am scraping have explicit policies regarding this, and the scraping is done respectfully and without much server push.

Still, just as a matter of keeping things separate (and who wouldn't want to remain anonymous if you could and there was no caveat), who knows about this and I'm curious why it's not talked about more.

 

[Chris.Roark]

Indeed TOR is great for privacy, but there are a few things to consider before using it for automation or scraping:

1. TOR is a privacy tool, not an automation tool and it was created for this purpose
2. Most TOR exit nodes have flagged IPs, so on some major websites you risk getting captcha-ed often (just try to open google.com while connecting to TOR - you might get a captcha).
3. TOR is slow because it passes your traffic through 3 different nodes before reaching a website.
4. Even if your use TOR you still need a VPS to hold your scraping tool/script so you can automate it 24-7.

You can use TOR for scraping, and you can even find scripts ready-to-use online, but it doesn't mean it will work perfectly (otherwise everybody will do it).

 

[protonx85]

Indeed TOR is great for privacy, but there are a few things to consider before using it for automation or scraping:

1. TOR is a privacy tool, not an automation tool and it was created for this purpose
2. Most TOR exit nodes have flagged IPs, so on some major websites you risk getting captcha-ed often (just try to open google.com while connecting to TOR - you might get a captcha).
3. TOR is slow because it passes your traffic through 3 different nodes before reaching a website.
4. Even if your use TOR you still need a VPS to hold your scraping tool/script so you can automate it 24-7.

You can use TOR for scraping, and you can even find scripts ready-to-use online, but it doesn't mean it will work perfectly (otherwise everybody will do it).

Right, those were my thoughts.
Might be ideal in conjunction with VPS to scrape the non-behemoths or a list of URL's that don't require captcha or what not.
Obviously scraping google will create some issues with captchas, but for sites that have nothing of the sort.. say scraping like ... oh, idk, mywebstore.com (just made that up, but you know what I mean), could work well I'd think?

Thanks for the input though I appreciate it, it is an interesting subject and I suppose comes down to an application type basis.
Proxies and paid-VPNs both have their own downside as well as far as complete anonymity. TOR as far as I know is basically untraceable, and has rotating IPs automatically, is free... or you'd have to be doing something really shady for them to go through all that work and probably falling short anyhow. Just my thoughts in the name of trying to stay anonymous

 

[HoNeYBiRD]

Right, those were my thoughts.
Might be ideal in conjunction with VPS to scrape the non-behemoths or a list of URL's that don't require captcha or what not.
Obviously scraping google will create some issues with captchas, but for sites that have nothing of the sort.. say scraping like ... oh, idk, mywebstore.com (just made that up, but you know what I mean), could work well I'd think?

Thanks for the input though I appreciate it, it is an interesting subject and I suppose comes down to an application type basis.
Proxies and paid-VPNs both have their own downside as well as far as complete anonymity. TOR as far as I know is basically untraceable, and has rotating IPs automatically, is free... or you'd have to be doing something really shady for them to go through all that work and probably falling short anyhow. Just my thoughts in the name of trying to stay anonymous

You'd be better off with some free public proxies, some of those are really fast, if you want to scrape Google, then Google-passed ones. Tor is really slow.

 

[surgee]

I've tried scraping with TOR before. For my use case the website I was scrapping from didn't care it was TOR. But google will captcha you like crazy if you have a TOR ip list. It easy to get a list of all the TOR exit nodes so most website especially ones that use CloudFlare constantly block TOR ips.

 

[itz_styx]

depending on what kind of scraping you want to do, TOR works fine if you keep switching nodes. its just a bit slow.

 

[Chris.Roark]

You'd be better off with some free public proxies, some of those are really fast, if you want to scrape Google, then Google-passed ones. Tor is really slow.

I agree, this is the best first option. There's no need to send automatic request through TOR because the network can be quite slow sometimes.

Try some free proxies, they will do the job