[RUBY] LinkedIn Scraper for getting more profile hits and connections

plopster

Newbie
Joined
Feb 13, 2012
Messages
16
Reaction score
3
I use this script to search for other IT professionals in my area and view their profiles. Especially useful for visiting recruiters, etc who then go to view your profile and send you future job prospects. If you are part of a business that uses LinkedIn and lists their staff, it also helps raise your ranking with that.

With the code below you need to replace the hxxp://xxx.linkedin.xxx part with the correct address because it wouldn't let me post links.

Code:
require 'rubygems'
require 'mechanize'
require 'open-uri'
require 'json'

user = 'linkedinusername or email'
pass = 'sdsdsd'
delay = 10

keywords = [
  'information technology brisbane',
  'it brisbane',
  'cio brisbane',
  'cto brisbane'
]

url = 'hxxp://xxx.linkedin.xxx/vsearch/f?type=all&trk=vsrp_people_sel&keywords='
profileurl = 'hxxp://xxx.linkedin.xxx/profile/view?id='

results = []
loggedin = false

a = Mechanize.new { |agent|
  agent.user_agent_alias = 'Mac Safari'
}
a.agent.http.verify_mode = OpenSSL::SSL::VERIFY_NONE

a.get('hxxp://xxx.linkedin.xxx') do |page|
  loggedin_page = page.form_with(:id => 'login') do |form|
    form.session_key = user
    form.session_password = pass
  end.submit
  
  if loggedin_page.title == 'Welcome! | LinkedIn' then loggedin = true end
end

if not loggedin then print "Couldn't logon to LinkedIn.\n"; exit; end

keywords.each { |x|
  a.get(url + URI::encode(x)) do |page|
    pp url + URI::encode(x)
    
    begin
    
      begin
      jsons = page.body.scan(/\{"person"\:\{.*?\}\}\}/i).each do |match|
        json = JSON.parse(match)
        if json["person"]["fmt_name"] == nil then next end
        
        print "Visiting profile for #{json["person"]["fmt_name"]}\n"
        a.get(profileurl + json["person"]["id"].to_s)
        print "Waiting #{delay} seconds before continuing.\n"
        sleep(delay)
      end
      rescue
      end
      
      is_next_page = false
      next_url = ""
      
      begin
      jsons = page.body.scan(/"resultPagination"\:\{.*?\}\]/i).each do |match|
        json = JSON.parse('{' + match + "}}")
        if json["resultPagination"]["nextPage"] != nil then
          is_next_page = true;
          next_url = "hxxp://xxx.linkedin.xxx" + json["resultPagination"]["nextPage"]["pageURL"]
        end
      end
      rescue
      end

      if is_next_page then
        sleep(delay)
        print "Getting next page: #{next_*****\n"
        page = a.get(next_url)
      end
    end while (is_next_page)
  end
}
 
Last edited:
Update the following vars with your information:

Code:
user = 'linkedinusername or email'
pass = 'sdsdsd'
delay = 10

keywords = [
  'information technology brisbane',
  'it brisbane',
  'cio brisbane',
  'cto brisbane'
]

url = 'hxxp://xxx.linkedin.xxx/vsearch/f?type=all&trk=vsrp_people_sel&keywords='
profileurl = 'hxxp://xxx.linkedin.xxx/profile/view?id='

Then run with ruby:

Code:
ruby ./filename.rb
 
how do you use this script?

1. Name the script link.rb
2. Replace the variables that OP said
2. Put it on your server that has Ruby installed
3. In the command like run ruby ./link.rb

Correct?
 
After chaning line 74 to
Code:
print "Getting next page:"

Im just getting the keywords url posted in my console. Any ideas? (Links are ok, I changed the "xxx" to post here)

C:\Sites>ruby C:\Users\usuario\Desktop\linkedinruby.rb
"xxx.linkedin.xxx/vsearch/f?type=all&trk=vsrp_people_sel&keywords=information%20technology%20brisbane"
"xxx.linkedin.xxx /vsearch/f?type=all&trk=vsrp_people_sel&keywords=it%20brisbane"
"xxx.linkedin.xxx/vsearch/f?type=all&trk=vsrp_people_sel&keywords=cio%20brisbane"
"xxx.linkedin.xxx/vsearch/f?type=all&trk=vsrp_people_sel&keywords=cto%20brisbane"
 
I'm getting the following error when running on Linux Mint 17.1:

$ ruby ./linkedinscript.rb
./linkedinscript.rb:28:in `block (2 levels) in <main>': undefined method `session_key=' for nil:NilClass (NoMethodError)
from (eval):23:in `form_with'
from ./linkedinscript.rb:27:in `block in <main>'
from /usr/lib/ruby/vendor_ruby/mechanize.rb:434:in `get'
from ./linkedinscript.rb:26:in `<main>'
 
I'm getting the following error when running on Linux Mint 17.1:

$ ruby ./linkedinscript.rb
./linkedinscript.rb:28:in `block (2 levels) in <main>': undefined method `session_key=' for nil:NilClass (NoMethodError)
from (eval):23:in `form_with'
from ./linkedinscript.rb:27:in `block in <main>'
from /usr/lib/ruby/vendor_ruby/mechanize.rb:434:in `get'
from ./linkedinscript.rb:26:in `<main>'


The linkedin page has likely changed since the code was written, so you'll need to update that section of the program. This happens with a lot of scrapers eventually.
 
Back
Top