Looking for tool to scrape SimilarWeb

[chunklady]

Hi,

I have been looking for a while for a tool to scrape similar web - i have a stack of domains i need to update on a yearly basis (around 10k domains), but they seem to have a rather efficient bot-detection system.

In my last attempt I tried doing some scraping with Puppeteer / Headless Chromium, but even that it seems to be able to recognise.

So, I thought there might be a tool or service out there, that scrapes the data already?

 

[Deleted member 877043]

What happens when you use headless mode or Chrome?

Do you use sloMo?

How many requests are you sending? I.e. try to scrape 100 pages in 10 sec.

Do you use proxies?

 

[chunklady]

After 3 requests from the same IP, i got the bot-detection trigger. I have tried with/without SlowMo.

I scraped 3 single URLs with 10 seconds apart. I have tried with/without proxies.

 

[Deleted member 877043]

After 3 requests from the same IP, i got the bot-detection trigger. I have tried with/without SlowMo.

I scraped 3 single URLs with 10 seconds apart. I have tried with/without proxies.

Hhmm, okay.

And scraping 10.000 domains/pages that way would take forever too.

Have you tried the similarweb api?
https://www.similarweb.com/corp/developer/

 

[thetrustedzone]

I already have similar web scraper will be launched end of this month in bhw marketplace... what exactly results you want to get ?

 

[iki]

I already have similar web scraper will be launched end of this month in bhw marketplace... what exactly results you want to get ?

I would like to check similarweb traffic for domains in bulk, is that possible with your tool?

 

[GrandScraperG]

It is protected by distil-network.

 

[thetrustedzone]

I would like to check similarweb traffic for domains in bulk, is that possible with your tool?

yes , follow me once product launched you will be informed .

 

[chunklady]

I already have similar web scraper will be launched end of this month in bhw marketplace... what exactly results you want to get ?

I was looking for basically all the information on the default (free) page, that you see on web. Without the api / login.

Have you tried the similarweb api?
https://www.similarweb.com/corp/developer/

Yes - and no; the API isn't free, and their service is quite pricy.

 

[thetrustedzone]

I was looking for basically all the information on the default (free) page, that you see on web. Without the api / login.

okay ...

 

[chunklady]

It is protected by distil-network.

Distil seems extremely effective compared to some of the other services I have tried to scrape through.

 

[Striker21]

yes , follow me once product launched you will be informed .

Did this/your service/tool ever happen?

I could need to scrape a few thousand entries myself every month or two. A rather tiny batch i know... but as the original poster said their protection is annoying without a proper trick or two to bypass the captcha (well sometimes I don't get that, just blocked a 100% after 3-4 attempt). Optimally I'd would have loved help to get a something based on puppeteer or similar working on a my own VPS box, pulling the list of sites from pgsql (and updating), with a proxy and/or captcha service (e.g. 2captcha) if an integrated solution is not possible/easily done... been experimenting with puppeteer, headless etc but struggling to get things working. Would be willing to pay some quid for help or a service.

 

[thetrustedzone]

@chunklady

Did this/your service/tool ever happen?

I could need to scrape a few thousand entries myself every month or two. A rather tiny batch i know... but as the original poster said their protection is annoying without a proper trick or two to bypass the captcha (well sometimes I don't get that, just blocked a 100% after 3-4 attempt). Optimally I'd would have loved help to get a something based on puppeteer or similar working on a my own VPS box, pulling the list of sites from pgsql (and updating), with a proxy and/or captcha service (e.g. 2captcha) if an integrated solution is not possible/easily done... been experimenting with puppeteer, headless etc but struggling to get things working. Would be willing to pay some quid for help or a service.

similarweb scraper part of Teknife Multi Software -submitted to BHW marketplace under review now - , could be found here with free trial , but don't buy now please because not approved yet by BHW + sale thread contains discount offer .