Scraping Linkedin Jobs

iddqd · Jan 3, 2018

Hi Guys,

I'm working on a personal project, I want to create a local web portal that implements a big repositoy for job posts scraped from different local website...

I have already scraper 5 local portals and now I'm facing the big challenge: Scraping Linkedin Job Positions...

I'm working with .Net using a common component called Selenium , it is available also for Java, Python and other languages...

I simulate the login, I scrap the jobs but after 4-5 actions it log me off asking for login again and for a captcha control....

If I try to load the job positions manually it never happens, but if I try to do it by my bot, I get kicked off...

I have tryed also to implement some random actions like scroll the page and clicking random links but I doesn't works... I get kicked off...

Can you help me ?

Thanks

Pruvia · Jan 8, 2018

Are you adding wait times between each action? It could be doing each action too fast.

DaVinci Tech · Jan 9, 2018

Could be user-agent and could be the wait times and can also be your pattern of behavior (logging in and instantly looking for jobs).

Jobdollarr · Jan 9, 2018

Actually you just need a software designer for it.

And you can see it around google.
There is always a solution in every problem

knaitas · Jan 9, 2018

linkedin is extremely difficult to scrape, you will have a hard time scraping it

but lots of luck to you!

DaVinci Tech · Jan 9, 2018

knaitas said:
linkedin is extremely difficult to scrape, you will have a hard time scraping it but lots of luck to you!

I actually made a bot to scrap LinkedIn and didn't face any problem but the Captcha on login.
Why is it difficult to scrap?

knaitas · Jan 9, 2018

DaVinci Tech said:
but the Captcha on login.

and the captcha after they log you out

and the log outs every x actions

thats why

and the captcha on login can be avoided by saving cookies and reusing them for each account individually

DaVinci Tech · Jan 9, 2018

knaitas said:
and the captcha after they log you out

and the log outs every x actions

thats why and the captcha on login can be avoided by saving cookies and reusing them for each account individually

I'm pretty sure they don't log you out after some number of actions.
Are you using selenium perhaps?
I used to scrape jobs and mass apply for groups.

knaitas · Jan 9, 2018

DaVinci Tech said:
I'm pretty sure they don't log you out after some number of actions.
Are you using selenium perhaps?
I used to scrape jobs and mass apply for groups.

When did you do that? Since microsoft bought linkedin they became very strict about this website, and keep the data secure.

DaVinci Tech · Jan 9, 2018

knaitas said:
When did you do that? Since microsoft bought linkedin they became very strict about this website, and keep the data secure.

Two to four months ago, I remember it having the new design and the big JSON responses drove me mad when coding.

Kuem · Jan 10, 2018

DaVinci Tech said:
Two to four months ago, I remember it having the new design and the big JSON responses drove me mad when coding.

This was in c# correct?

knaitas · Jan 10, 2018

Kuem said:
This was in c# correct?

it does not matter what language it is... everyone still uses the same libraries for scraping web, no matter the language >.>

Kuem · Jan 10, 2018

knaitas said:
it does not matter what language it is... everyone still uses the same libraries for scraping web, no matter the language >.>

That's not true... i usually scrape or automate with ruby or python and i never used selenium or sny of that. It's such a hassle to scraoe in .net framework

knaitas · Jan 10, 2018

Kuem said:
That's not true... i usually scrape or automate with ruby or python and i never used selenium or sny of that. It's such a hassle to scraoe in .net framework

yes, because .net is not for scraping!! why would you use it to scrape?

Kuem · Jan 10, 2018

knaitas said:
yes, because .net is not for scraping!! why would you use it to scrape?

No language is made for scraping... you would use it to scrape if you are creating a program with a gui easily

knaitas · Jan 10, 2018

Kuem said:
No language is made for scraping... you would use it to scrape if you are creating a program with a gui easily

yes, but some are much easier to do one task then to do another one... and you can make gui interfaces even with php..... .net is over-f***-rated

iddqd · Jan 21, 2018

DaVinci Tech said:
I'm pretty sure they don't log you out after some number of actions.
Are you using selenium perhaps?
I used to scrape jobs and mass apply for groups.

Hi Man,
Yes, I'm using Selenium and I only need to scrap jobs...
After 2-3 pages it logs me out and asks captcha...

I try to put random delay, random scroll, but nothing... It kick me off...

DaVinci Tech · Jan 24, 2018

iddqd said:
Hi Man,
Yes, I'm using Selenium and I only need to scrap jobs...
After 2-3 pages it logs me out and asks captcha...

I try to put random delay, random scroll, but nothing... It kick me off...

There you have it, you have no control of JavaScript and log requests, moreover the user agent is shows as selenium for whatever driver you're using.
Can easily identify that you're a bot.

BuyProxies.com · Jan 29, 2018

Are you using proxies? You'll need to change your IP after every few requests to avoid captchas.

DaVinci Tech · Jan 29, 2018

BuyProxies.com said:
Are you using proxies? You'll need to change your IP after every few requests to avoid captchas.

I'm not sure if this is a bad joke given that you're a proxy seller

.

but that's not true.
If you do it the right way you, you can use one account to scrape thousands of jobs.

Scraping Linkedin Jobs

Newbie

Junior Member

Newbie

Elite Member

BANNED

Newbie

BANNED

Newbie

BANNED

Newbie

Power Member

BANNED

Power Member

BANNED

Power Member

BANNED

Newbie

Newbie

Newbie

Newbie

Main Menu

Marketplace

Making Money

BlackHat World