1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scraping SimilarWeb & Data Extraction

Discussion in 'Black Hat SEO Tools' started by blacksoy, Jan 27, 2017.

  1. blacksoy

    blacksoy Newbie

    Joined:
    Feb 9, 2016
    Messages:
    11
    Likes Received:
    0
    Hello BHW,

    I think every white, grey and black SEOs here know about Screaming Frog.

    Maybe some of you don't know it's data extraction feature (CSS or XPath) which I really like.

    Buuuut.. the problem is when you are crawling websites like SimilarWeb, you need to use proxies.

    Screaming Frog has only one proxy option. -You can see in the attachment-

    [​IMG]

    How to overcome this problem? Is it possible to send different requests to SimilarWeb with only one User Proxy Server?

    Waiting your help!
     
  2. cherub

    cherub Regular Member

    Joined:
    Dec 18, 2006
    Messages:
    285
    Likes Received:
    123
    Gender:
    Male
    Occupation:
    Boss
    Location:
    UK
    I suppose you could use one of the rotating proxy providers here, they tend to give you just one IP : PORT which rotates proxy every request/every few mins. Plenty of providers in the Proxies for Sale section of the marketplace.
     
  3. ebiz101

    ebiz101 Jr. VIP Jr. VIP

    Joined:
    Feb 16, 2010
    Messages:
    580
    Likes Received:
    135
    Yup.. I agree with @cherub this will solve your problem.. usually you have the option to change on every request, 3 min, 10 min, etc.
     
  4. Gogol

    Gogol Jr. VIP Jr. VIP

    Joined:
    Sep 10, 2010
    Messages:
    3,476
    Likes Received:
    3,103
    Gender:
    Male
    You could even use a service like Tor may be? AdvOr can be of help in that case.
     
  5. outscrape

    outscrape Jr. VIP Jr. VIP

    Joined:
    Nov 23, 2016
    Messages:
    118
    Likes Received:
    77
    Might be able to use something like HMA or a VPN (someone said Tor) that auto-rotates.
     
  6. extremeboy

    extremeboy Jr. VIP Jr. VIP

    Joined:
    Jul 8, 2010
    Messages:
    3,220
    Likes Received:
    673
    Occupation:
    World Best RANK Tracker SERPCloud.com
    Home Page:
    if you want to do an scraping get private proxies will be best for that purpose and rotate with some randomization technique will be good.
     
  7. domainmadness

    domainmadness Senior Member

    Joined:
    Jun 22, 2011
    Messages:
    1,119
    Likes Received:
    354
    Which services gives you new ip on every request.
     
  8. cherub

    cherub Regular Member

    Joined:
    Dec 18, 2006
    Messages:
    285
    Likes Received:
    123
    Gender:
    Male
    Occupation:
    Boss
    Location:
    UK
    Off the top of my head there are stormproxies and P2P proxies (both in the proxies section here), and some smaller outfits such as privateproxies.org and proxy-connect.com