1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scrape with NodeJS and JQuery: Get your IP, Host and User-Agent

Discussion in 'Programming' started by rodvan, Dec 5, 2016.

  1. rodvan

    rodvan Jr. VIP Jr. VIP

    Joined:
    Jul 27, 2010
    Messages:
    1,324
    Likes Received:
    505
    Occupation:
    developer, marketing, automation, machine learning
    Location:
    Wizard of Bots
    Home Page:
    In this post I am going to teach you how to screen scrape using NodeJS and JQuery (cheerio). Its relatively easy, here is the code:

    Code:
    var request = require('request'); // we need request library
    var cheerio = require('cheerio'); // and cheerio library/ JQuery
    // set some defaults
    req = request.defaults({
      jar: true,                 // save cookies to jar
      rejectUnauthorized: false,
      followAllRedirects: true   // allow redirections
    });
    // scrape the page
    req.get({
        url: "http://www.whatsmyip.org/",
        headers: {
            'User-Agent': 'Google' // You can put the user-agent that you want
         }
      }, function(err, resp, body) {
     
      // load the html into cheerio
      var $ = cheerio.load(body);
     
      // get the data and output to console
      console.log( 'IP: ' + $('#ip').text() );  //scrape using CSS selector
      console.log( 'Host: ' + $('#hostname').text() );
      console.log( 'User-Agent: ' + $('#useragent').text() );
    });
    
     
  2. kahuna74

    kahuna74 Regular Member

    Joined:
    Aug 19, 2014
    Messages:
    270
    Likes Received:
    102
    Gender:
    Male
    Occupation:
    Software Developer
    Location:
    Grand Rapids, MI
    I just use an apache env page on a random server to do this. no external libraries required, and it gives you more info than whatismyip.org.