1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to Scrape using NodeJS and JQuery: (Get your IP, Host and User-Agent)

Discussion in 'Programming' started by rodvan, Sep 2, 2016.

  1. rodvan

    rodvan Jr. VIP Jr. VIP

    Joined:
    Jul 27, 2010
    Messages:
    1,313
    Likes Received:
    495
    Occupation:
    developer, marketing, automation, machine learning
    Location:
    Wizard of Bots
    Home Page:
    Here is a chunk of code that will go to a get my ip domain and extract using CSS selectors thanks to JQuery library for NodeJS called Cheerio.
    The code is properly explained on comments:
    Code:
    var request = require('request'); // we need request library
    var cheerio = require('cheerio'); // and cheerio library/ JQuery
    // set some defaults
    req = request.defaults({
      jar: true,                 // save cookies to jar
      rejectUnauthorized: false,
      followAllRedirects: true   // allow redirections
    });
    // scrape the page
    req.get({
        url: "http://www.whatsmyip.org/",
        headers: {
            'User-Agent': 'Google' // You can put the user-agent that you want
         }
      }, function(err, resp, body) {
     
      // load the html into cheerio
      var $ = cheerio.load(body);
     
      // get the data and output to console
      console.log( 'IP: ' + $('#ip').text() );  //scrape using CSS selector
      console.log( 'Host: ' + $('#hostname').text() );
      console.log( 'User-Agent: ' + $('#useragent').text() );
    });
     
    • Thanks Thanks x 1
  2. Mr.Charles

    Mr.Charles Jr. VIP Jr. VIP

    Joined:
    Apr 6, 2012
    Messages:
    407
    Likes Received:
    35
    Gender:
    Male
    Occupation:
    Web Developer
    Location:
    Google
    Home Page:
    Great contribution. Cheerio is a very powerful library, Just now I'm doing tests with it.
     
  3. rodvan

    rodvan Jr. VIP Jr. VIP

    Joined:
    Jul 27, 2010
    Messages:
    1,313
    Likes Received:
    495
    Occupation:
    developer, marketing, automation, machine learning
    Location:
    Wizard of Bots
    Home Page:
    Im glad this worked for you, getting CSS selectors this easy is very cool instead of using xpath.