LinkedIn scraper

Bonvi

Power Member
Joined
Jul 15, 2019
Messages
756
Reaction score
419
I recently tried to scrape LinkedIn for testing purposes and my requests were blocked after a short time. I think it's some captcha issue or something because when I logged in with my throwaway LinkedIn account I got a warning about automation. Yet I know that many people scrape LinkedIn.

How?
 
Set it up simply. Logged in using my credentials, then searched for what I need, click the profiles one by one and extract the desired data. I used selenium and scrapy.
 
Set it up simply. Logged in using my credentials, then searched for what I need, click the profiles one by one and extract the desired data. I used selenium and scrapy.
The selenium framework is too old to be easily recognised by websites
 
You can try playwright, I've been using playwright to automate google lately and it works pretty well.
 
You can try playwright, I've been using playwright to automate google lately and it works pretty well.
Okay, sounds interesting. I haven't looked at it in detail yet, jsut quickly googled it. What are the main advantages of playwright over selenium?
 
Okay, sounds interesting. I haven't looked at it in detail yet, jsut quickly googled it. What are the main advantages of playwright over selenium?
For me, compared to selenium, it doesn't require a webdriver to be installed, you don't need to adapt to manage different versions, and it's a bit simpler.

For me, compared to selenium, it doesn't require a webdriver to be installed, you don't need to adapt to manage different versions, and it's a bit simpler.
Most importantly, it is an latest framework that is relatively less likely to be detected by websites
 
Puppeteer is another good headless browser you could use. Have you looked into using proxies?
 
Puppeteer is another good headless browser you could use. Have you looked into using proxies?
Puppeteer and playwright drive the browser in a similar way, playwright+antidetect browser, and it's easy to manage the ip through the antidetect browser as well
 
Puppeteer and playwright drive the browser in a similar way, playwright+antidetect browser, and it's easy to manage the ip through the antidetect browser as well
How do you set up anti-detect browser and how does that work with linkedin?

I did look into proxies btw but something went wrong with my program last time I tried and I ended up paying 10 dollars worth of proxy in 5 seconds somehow :/
 
Using selenium or any scraper it will get your account suspend !

Better to build your own code to manipulate browser behaviour as human to pass those verification checks on linkedin.
 
Using selenium or any scraper it will get your account suspend !

Better to build your own code to manipulate browser behaviour as human to pass those verification checks on linkedin.
How?
 
Puppeteer and playwright drive the browser in a similar way, playwright+antidetect browser, and it's easy to manage the ip through the antidetect browser as well
how do you connect playwright+antidetect browser?
 
When I was doing something similar, the first thing I did was to delay the requests. In this case, the system does not recognize the activity as questionable, and therefore does not ban or freeze the account.
 
I recently tried to scrape LinkedIn for testing purposes and my requests were blocked after a short time. I think it's some captcha issue or something because when I logged in with my throwaway LinkedIn account I got a warning about automation. Yet I know that many people scrape LinkedIn.

How?
You try making a chrome extension instead and inject js directly this will reduce a lot of your problems and mimicking human behaviour would be easy. Professional LinkedIn scrapers mostly inject js after scrolling and loading all page data.
 
You try making a chrome extension instead and inject js directly this will reduce a lot of your problems and mimicking human behaviour would be easy. Professional LinkedIn scrapers mostly inject js after scrolling and loading all page data.
It's not as easy as you're saying, because for faking human behavior, you need to look at parameters, and I've recently prepared a scraping script. Mostly, LinkedIn changes its parameters frequently and keeps a tracking ID, which causes problems in automation.
 
Okay, this sounds very difficult. Are there any alternatives to linkedin for leads?
 
Hello. I developed a linkedin scraper and selling it for a year. I can tell that it is not easy to scrape data on LinkedIn.
First of all you cant see all data if you arent logged in.
 
Last edited by a moderator:
Back
Top