How useful is Javascript (Node.js)?

[BBulldog]

Now, before anything, I'd like to say that I'm a total newbie in anything programming related. I know some very basic python and that's about it. However, I see a lot of talk about Node.js in the context of automation. Some web scraper I was looking at, for instance, used Selenium, but afterwards, the creator said that he rewrote the project in Node.js as he thinks Selenium is outdated for this task and Node.js is the go-to choice nowadays.

I do find myself needing scraping quite often, so I was wondering is this something worth investing time into and are my assumptions correct in regards to Node.js being the winner? Thanks for your time.

 

[ulaho]

Now, before anything, I'd like to say that I'm a total newbie in anything programming related. I know some very basic python and that's about it. However, I see a lot of talk about Node.js in the context of automation. Some web scraper I was looking at, for instance, used Selenium, but afterwards, the creator said that he rewrote the project in Node.js as he thinks Selenium is outdated for this task and Node.js is the go-to choice nowadays.

I do find myself needing scraping quite often, so I was wondering is this something worth investing time into and are my assumptions correct in regards to Node.js being the winner? Thanks for your time.

Tried scrapping in node.js it is not difficult, I was learning python at first, great language, also has lots of libraries. But I have chosen node.js. It has so many packages at npm, also you can do so much with it. Same language for front-end and back-end. Lots of tutorial available on how to do stuff.

 

[BBulldog]

Tried scrapping in node.js it is not difficult, I was learning python at first, great language, also has lots of libraries. But I have chosen node.js. It has so many packages at npm, also you can do so much with it. Same language for front-end and back-end. Lots of tutorial available on how to do stuff.

Good to hear your positive experience. Yes, I forgot to mention that the other reason I want to get familiar with JS is web design, thanks for mentioning that.

 

[satyr85]

Python have scrapy libary for scraping. I code alot in python and i dont see a reason to switch to other language. Unless you write code that need to be distributed on 100s of servers python is good choice because you can write code very fast. In 100s servers scenario node for example can be better because of lower cpu usage(do you can save on hardware). On other hand Instagram backend is written python so...

 

[BBulldog]

Python have scrapy libary for scraping. I code alot in python and i dont see a reason to switch to other language. Unless you write code that need to be distributed on 100s of servers python is good choice because you can write code very fast. In 100s servers scenario node for example can be better because of lower cpu usage(do you can save on hardware). On other hand Instagram backend is written python so...

The words from the developer I talked about above:

1. Python is not the language/framework for modern scraping. Node/Javascript is. The reason is puppeteer. puppeteer is the de-facto standard for controlling and automatizing web browsers (especially Chrome). This project uses Selenium. Selenium is kind of old and outdated.

I asked in regards to that. I also saw a few users here on the forum saying that they use Node.js for some automation.

 

[satyr85]

I do find myself needing scraping quite often

I see a lot of talk about Node.js in the context of automation

Scraping and web automation are 2 different things.
You dont want to use puppeteer or selenium for scraping - its just headless browser and because of that it use a ton of resources.
You want to scrape with pure reguests, in this way you can scrape with speed of light lol.
When you cant scrape site because it have some antibot protection then you try selenium puppeteer/selenium or something similar.

Selenium is not old, its not outdated (you can use selenium with latest version of firefox) but its detectable. Puppeteer is also detectable by some have antibot networks so... same/similar pair of shoes. Ive made successful bots for some big sites with selenium + python and i didnt faced any problems tbh.

Python is not the language/framework for modern scraping.

Thats bulshit, any modern language (C#, python, node etc) is perfect for scraping. No matter what language you choose you will face same/similar problems if you will try to scrape sites with heavy antispam protection (distill networks for example)

Rule of thumb - dont complicate your life, dont complicate your code. Try the easiest solution, than move to something more complex if easy solution dont work. Scraping with selenium/puppeteer is like trying to kill a fly with a hammer - sometimes its obligatory, but mostly not.

P.S.
Use language you "feel", language you like, language you understand.

 

[alice252293]

Learning a programming language to create bot is not that easy. Especially for a newbie like you. Why dont you hire someone to do it ?

 

[BBulldog]

P.S.
Use language you "feel", language you like, language you understand.

Thanks for the detailed write-up. Yeah, that's just the quote from the project page, like I said, I'm a total newbie to programming. I just noticed a lot of people mentioning Node.js recently, so I was wondering what's the deal. He mentioned there's a lot of anti-bot protection these days, so scraping with requests isn't as good anymore. Guess I'll check out Node.js and see how it resonates with me. Thanks again!

Learning a programming language to create bot is not that easy. Especially for a newbie like you. Why dont you hire someone to do it ?

Did you even read what I wrote or did you just comment for the sake of commenting?

 

[alice252293]

Thanks for the detailed write-up. Yeah, that's just the quote from the project page, like I said, I'm a total newbie to programming. I just noticed a lot of people mentioning Node.js recently, so I was wondering what's the deal. He mentioned there's a lot of anti-bot protection these days, so scraping with requests isn't as good anymore. Guess I'll check out Node.js and see how it resonates with me. Thanks again!



Did you even read what I wrote or did you just comment for the sake of commenting?

oh, of course I read your comment. "I'm a total newbie to programming"

 

[litportnet]

I would go for nodejs. Because of crazy amount of available packages and because of headless chrome library.
On the other hand, python is good because of scrapy library. But it's not a rocket science and you can achieve the same with nodejs and it will be probably easier and cheaper to find somebody to maintain and develop it in the future.

 

[TasDePixels]

I would go for nodejs. Because of crazy amount of available packages

That's the typical mistake made by new-comers.
First you should learn the core concepts of node : The event loop, Callbacks & Events, error management, clustering, deploying and how to handle data.
you won't learn anything by relying on third-party libraries. Don't be that guy, the self-proclaimed "node ninja" who had fun building a todo app with 700 mb node_modules folder. Seriously, don't be that guy.

Sure, it's cool as hell to skip the headache of understanding how handle an API by downlaoding a wrapper from NPM and some people will argue that you should never reinvent the wheel.
I believe in understanding the wheel before using it. Always check the source of code of the libraries ur using. check this article : https://medium.com/s/silicon-satire/i-peeked-into-my-node-modules-directory-and-you-wont-believe-what-happened-next-b89f63d21558

just stick to Wordpress if u love that plug'n'play mentality.

 

[ulaho]

That's the typical mistake made by new-comers.
First you should learn the core concepts of node : The event loop, Callbacks & Events, error management, clustering, deploying and how to handle data.
you won't learn anything by relying on third-party libraries. Don't be that guy, the self-proclaimed "node ninja" who had fun building a todo app with 700 mb node_modules folder. Seriously, don't be that guy.

Sure, it's cool as hell to skip the headache of understanding how handle an API by downlaoding a wrapper from NPM and some people will argue that you should never reinvent the wheel.
I believe in understanding the wheel before using it. Always check the source of code of the libraries ur using. check this article : https://medium.com/s/silicon-satire...-wont-believe-what-happened-next-b89f63d21558

just stick to Wordpress if u love that plug'n'play mentality.

You stated good points.

I even checked my current React projects size too, and it was ~200mb, if we would think more about that, we will make a production version of our project who won't take so much space. I even built the project to ensure how much space it would take and it is 485kb.

Of course this is more relevant when developing back-end applications.

 

[supereek]

If you master nodejs you can do both frond-end and back-end (for node then) as it is all in the same scripting language. Which is a good thing. Furthermore, nodejs is a really fast framework imho.

 

[kiron lessi]

Node.JS is a craze now. But start with python first.

 

[ebalda]

I think lot of people heard that Javascript is asynchronous and often problem that newbies experiencing with NodeJS is slow execution and etc.
It is happening because not everyone understand the basics on how to write Non Blocking code(takes time). So consider read more about it and you will be able to build a powerful applications. After clustering and etc (pm2)

Cheers

 

[jazzc]

Well, let's see:

* You can code anything you want in a reasonable timeframe without pulling your hair out:
* There's tons of "other peoples quality work" for you to leech off, thus saving you time and money:
* Same language for front-end and back-end without DSLs:
* Has high market demand if you want to sell your time:

What more do you want?

As for the node_modules file size, disk space is cheap. Unless you're making mobile apps minimizing disk space is a relic of the past and more or less and ideological crutch for some people nowadays. Don't spend time optimizing for things that don't pay you, unless your time is worth nothing.

 

[Deleted member 923340]

As for the node_modules file size, disk space is cheap.

Only matters when u develop.
You don't pack the node_modules directory when you ship the software.

Edit: to answer op
Entirely depends on what you want to do.
Working with wordpress/drupal -> php
Building a SAAS kind-of app -> Nodejs (React/Angular/Express and what not)
Want build bots -> C#/Python . (best option for you i guess)
Work in enterprise software -> Java
this list goes on and on...

 

[BBulldog]

Well, let's see:

* You can code anything you want in a reasonable timeframe without pulling your hair out:
* There's tons of "other peoples quality work" for you to leech off, thus saving you time and money:
* Same language for front-end and back-end without DSLs:
* Has high market demand if you want to sell your time:

What more do you want?

As for the node_modules file size, disk space is cheap. Unless you're making mobile apps minimizing disk space is a relic of the past and more or less and ideological crutch for some people nowadays. Don't spend time optimizing for things that don't pay you, unless your time is worth nothing.

I appreciate the detailed response! Your points definitely reassured me that this is something worth picking up and will prove a valuable tool in my marketing arsenal. Thanks!

 

[Gogol]

What, so python isn't a good choice for scrapping? Well, best of luck learning a completely new language when you can do it with python only lol.
Also to the person who said frontend js = backend js.. Nah bro, it isn't. Even the context is different. You don't use js for the same reason in front-end, as you do in backend.. ...

 

[jazzc]

You don't use js for the same reason in front-end, as you do in backend

Obviously you don't, just like you don't use wood the same way in a house and in a boat. The point is that you're proficient in woodwork.