Looking for tool to scrape SimilarWeb

chunklady

chunklady

Newbie
Joined
Apr 7, 2016
Messages
44
Reaction score
11
Hi,

I have been looking for a while for a tool to scrape similar web - i have a stack of domains i need to update on a yearly basis (around 10k domains), but they seem to have a rather efficient bot-detection system.

In my last attempt I tried doing some scraping with Puppeteer / Headless Chromium, but even that it seems to be able to recognise.

So, I thought there might be a tool or service out there, that scrapes the data already?
 
Advertise on BHW
What happens when you use headless mode or Chrome?

Do you use sloMo?

How many requests are you sending? I.e. try to scrape 100 pages in 10 sec.

Do you use proxies?
 
After 3 requests from the same IP, i got the bot-detection trigger. I have tried with/without SlowMo.

I scraped 3 single URLs with 10 seconds apart. I have tried with/without proxies.
 
thetrustedzone said:
I already have similar web scraper will be launched end of this month in bhw marketplace... what exactly results you want to get ?
Click to expand...
I would like to check similarweb traffic for domains in bulk, is that possible with your tool?
 
thetrustedzone said:
I already have similar web scraper will be launched end of this month in bhw marketplace... what exactly results you want to get ?
Click to expand...

I was looking for basically all the information on the default (free) page, that you see on web. Without the api / login.

TrayShawn said:
Have you tried the similarweb api?
https://www.similarweb.com/corp/developer/
Click to expand...

Yes - and no; the API isn't free, and their service is quite pricy.
 
thetrustedzone said:
yes , follow me once product launched you will be informed .
Click to expand...
Did this/your service/tool ever happen?

I could need to scrape a few thousand entries myself every month or two. A rather tiny batch i know... but as the original poster said their protection is annoying without a proper trick or two to bypass the captcha (well sometimes I don't get that, just blocked a 100% after 3-4 attempt). Optimally I'd would have loved help to get a something based on puppeteer or similar working on a my own VPS box, pulling the list of sites from pgsql (and updating), with a proxy and/or captcha service (e.g. 2captcha) if an integrated solution is not possible/easily done... been experimenting with puppeteer, headless etc but struggling to get things working. Would be willing to pay some quid for help or a service.
 
@chunklady

Striker21 said:
Did this/your service/tool ever happen?

I could need to scrape a few thousand entries myself every month or two. A rather tiny batch i know... but as the original poster said their protection is annoying without a proper trick or two to bypass the captcha (well sometimes I don't get that, just blocked a 100% after 3-4 attempt). Optimally I'd would have loved help to get a something based on puppeteer or similar working on a my own VPS box, pulling the list of sites from pgsql (and updating), with a proxy and/or captcha service (e.g. 2captcha) if an integrated solution is not possible/easily done... been experimenting with puppeteer, headless etc but struggling to get things working. Would be willing to pay some quid for help or a service.
Click to expand...

similarweb scraper part of Teknife Multi Software -submitted to BHW marketplace under review now - , could be found here with free trial , but don't buy now please because not approved yet by BHW + sale thread contains discount offer .
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top