Scraping similarweb

Hello,
I am looking for a way to scrap similarweb.com
But just once, around 200 000 urls. I tries many things without success. I want to try to use proxy but don’t know if this is the solution. I read that similarweb is using distil-network. So even with proxy and ip pool, its not going to work ?

What are you using right now?
200k URL can't be done semi-automated via browser extensions (you can but that would take you ages), you need a bot for that! There is plenty of frameworks, some are free some are paid!
Proxies/Random User agents/Unique Sessions, are very important so you don't get blocked! using web browser automation is probably best in this case to avoid getting detected by anti-bots systems (but you can give HTTP a go)!
If you still not sure how to do that, i can write a small tutorial on the concept of scraping large amount of data from similar sites!

Regards,

Thank you for your reply. Ive just saw that scraping similarweb is impossible due to high bot detection. In case someone thinks this is possible, what Would you suggest ? Specificaly for similarweb.com

Been there, done that. Even doing it now as we speak.

Http requests simply won't cut it. Buy their corporate package or hire a developer/data miner to do it for you if you're on a budget.

What would be the budget for that ? (Data miner)

Post a thread in this section if you want to hire someone to do it for you : https://www.blackhatworld.com/forums/hire-a-freelancer.76/

I don't know about others but I'd charge $1k for mining 200k domains data.

Gerardfromhere said:

What would be the budget for that ? (Data miner)

Click to expand...

If you post in hire freelancer section ... for 200k i will do it for $200

I'm actually using an API I found on RapidAPI. I'm paying the $19USD plan and I'm getting structured data, pretty good tbh.

Here's the link: https://rapidapi.com/apifactory/api/similarweb2