Scraping SimilarWeb & Data Extraction

B

blacksoy

Newbie

Feb 9, 2016

11

0

• Jan 27, 2017

• #1

Hello BHW,

I think every white, grey and black SEOs here know about Screaming Frog.

Maybe some of you don't know it's data extraction feature (CSS or XPath) which I really like.

Buuuut.. the problem is when you are crawling websites like SimilarWeb, you need to use proxies.

Screaming Frog has only one proxy option. -You can see in the attachment-

How to overcome this problem? Is it possible to send different requests to SimilarWeb with only one User Proxy Server?

Waiting your help!

 

cherub

Senior Member

Dec 18, 2006

973

577

• Jan 27, 2017

• #2

I suppose you could use one of the rotating proxy providers here, they tend to give you just one IP : PORT which rotates proxy every request/every few mins. Plenty of providers in the Proxies for Sale section of the marketplace.

 

E

ebiz101

Power Member

Feb 16, 2010

739

174

• Jan 27, 2017

• #3

Yup.. I agree with @cherub this will solve your problem.. usually you have the option to change on every request, 3 min, 10 min, etc.

 

Gogol

Administrator

Staff member

Moderator

Executive VIP

Jr. VIP

Sep 10, 2010

11,811

26,653

• Jan 27, 2017

• #4

You could even use a service like Tor may be? AdvOr can be of help in that case.

 

outscrape

Junior Member

Nov 23, 2016

181

145

• Jan 28, 2017

• #5

Might be able to use something like HMA or a VPN (someone said Tor) that auto-rotates.

 

E

extremeboy

Elite Member

Jr. VIP

Jul 8, 2010

3,994

823

• Jan 28, 2017

• #6

if you want to do an scraping get private proxies will be best for that purpose and rotate with some randomization technique will be good.

 

domainmadness

Supreme Member

Jun 22, 2011

1,313

474

• Feb 23, 2017

• #7

cherub said:

I suppose you could use one of the rotating proxy providers here, they tend to give you just one IP : PORT which rotates proxy every request/every few mins. Plenty of providers in the Proxies for Sale section of the marketplace.

Click to expand...

Which services gives you new ip on every request.

 

cherub

Senior Member

Dec 18, 2006

973

577

• Feb 23, 2017

• #8

Off the top of my head there are stormproxies and P2P proxies (both in the proxies section here), and some smaller outfits such as privateproxies.org and proxy-connect.com