Node.js or Python for scraping?

Node.js or Python

  • Node.js

    Votes: 4 33.3%

  • Python

    Votes: 8 66.7%
  • Total voters
    12
Status
Not open for further replies.
dennsen

dennsen

Newbie
Joined
Apr 6, 2018
Messages
15
Reaction score
0
I am currently learning HTML & CSS and then I‘ll learn Javascript.

I want to built a Website that is going to use the jpg‘s from a website for my own.

I am asking myself if I should learn Node.js or Python. I think Python is more powerful, but if I would use Node.js I would only need to know JavaScript...

How do YOU scrape websites?
And how does it even work?

Are you coding a desktop app? Or a web app? A script?? And please talk about the security?

I’ve read that they know when someone is scraping their websites. And they will block your IP adress.
 
Advertise on BHW
The answer always comes down to what language you know best.
If you don't know both. I'd start with python
You can use frameworks like selenium, scrapy, beautifulsoup (based on your need) to do your stuff.

dennsen said:
And how does it even work?
Click to expand...
No one will spoon-feed you here. I'd start with this video to learn python.

Good luck.
 
good advice .

Deleted member 923340 said:
The answer always comes down to what language you know best.
If you don't know both. I'd start with python
You can use frameworks like selenium, scrapy, beautifulsoup (based on your need) to do your stuff.


No one will spoon-feed you here. I'd start with this video to learn python.

Good luck.
Click to expand...
 
Scrapy (python), is a good starting point. You will get more done in short time. They have a brief tutorial to get you started.

dennsen said:
I’ve read that they know when someone is scraping their websites. And they will block your IP adress.
Click to expand...

Unless you are making too many requests, like 1000 calls every minute. You are safe.
 
dennsen said:
I am currently learning HTML & CSS and then I‘ll learn Javascript.

I want to built a Website that is going to use the jpg‘s from a website for my own.

I am asking myself if I should learn Node.js or Python. I think Python is more powerful, but if I would use Node.js I would only need to know JavaScript...

How do YOU scrape websites?
And how does it even work?

Are you coding a desktop app? Or a web app? A script?? And please talk about the security?

I’ve read that they know when someone is scraping their websites. And they will block your IP adress.
Click to expand...

You ask way too much questions and each and every one of them is no way related to another.

Let me answer one by one...

It doesn't really matter which language you choose as long as the job gets done. Especially when you're a starter.

You want to build a website that is going to use jpg's ~ You can do this with plain HTML & CSS and you can sprinkle in some javascript to do some animations, etc.,

You think Python is more powerful but WHY? What makes you think node.js is inferior than Python?

How do I scrape websites?

Using both node.js and python.

And how does it even work?

There are a piece of code called Libraries(Python) and Modules(node.js) that helps you get the task done. If you were to scrape a page/website, http://docs.python-requests.org/en/master/ and/or https://github.com/request/request will get the job done.

Are you coding a desktop app? Or a web app? A script?? And please talk about the security?

I build desktop, web and mobile apps with node.js react but mostly web apps. Securing your app is whole another topic and it doesn't lie within the app itself. You gotta do some server hardening which you'll understand after reading some network administration books/articles.

I’ve read that they know when someone is scraping their websites. And they will block your IP adress.

True. You can either limit/dripfeed your requests or use proxies.
 
Your question is too broad. Both the languages should do the trick. To avoid IP ban you can use proxy IPs.
Finally it all depends on the target website which you want to scrape. If its a plain html website it should be fairly easy. While scraping content from big sites like Amazon could be really challenging.
 
I feel @SpoonFeeder has made a very good point

All of it depends what kind of data you want to scraping.
I mean, for static content you should definlity go for Python stuff for this kind of work.
Why? because it's simple to got and it have a lot madure software like Beautifulsoup, Scrappy and so-on.

Now for dynamic content data like ajax results. You can check Puppeteer, it's a new and niece Javascript node library to work with Chrome headless.

But, if you don't know HTML/CSS very well, you will stuck with it when you are trying doing Scrapping stuff. (xpath, click behaviour). Learn fundations first.

Hope this helps.
 
dennsen said:
I am currently learning HTML & CSS and then I‘ll learn Javascript.

I want to built a Website that is going to use the jpg‘s from a website for my own.

I am asking myself if I should learn Node.js or Python. I think Python is more powerful, but if I would use Node.js I would only need to know JavaScript...

How do YOU scrape websites?
And how does it even work?

Are you coding a desktop app? Or a web app? A script?? And please talk about the security?

I’ve read that they know when someone is scraping their websites. And they will block your IP adress.
Click to expand...

When it comes to these kinds of questions it comes down to what you personally are more drawn to. Both languages will do exactly what you need, and there are plenty of jobs that would hire you for knowing either language too, if you'd want to be a developer at some point. If you have never written software before, I suggest going with Python. Javascript uses a few concepts (callback functions, concurrent programming, etc) that make it a bit more challenging to use if you don't understand those concepts. Hell, I write Javascript nearly every day and I still have trouble sometimes with dealing with callback hell. That's not to say that Javascript isn't a good choice though. It is INCREDIBLY powerful, and if you know Javascript well you can literally write software for any platform in it. It's just getting over the initial learning curve. Both have great libraries for web scraping and parsing HTML and bot will get the job done.

I personally use Python whenever I can because it is the language that I know best, but it's not always the most convenient. There is no answer to your question, it is going to come down to the task that you are trying to complete.

Web scraping works by first writing a program that downloads a webpage with the expected HTML. The same HTML that you get in your regular web browser. This is easier said than done because you'll need to use Proxies do to websites that try to block your scraper. The next program looks at the HTML and picks out the information that you want then you do stuff with that data. Typically, a web scraper is a small part in a bigger application. That could be a desktop app, a web app, or just a script. It all depends on what you are trying to do.
 
I'm vote both. They're useful tools for scraping websites or others.
Another recommend for you, you can use PHP. It's a popular programming language. I usually PHP to scrape website easily and fast
 
Python is probably easier to use, and although you can use selenium in both, it has a few more features in python. Node.js has some things about it that I find tedious to work around even with quite a lot of experience.
 
Gazo said:
When it comes to these kinds of questions it comes down to what you personally are more drawn to. Both languages will do exactly what you need, and there are plenty of jobs that would hire you for knowing either language too, if you'd want to be a developer at some point. If you have never written software before, I suggest going with Python. Javascript uses a few concepts (callback functions, concurrent programming, etc) that make it a bit more challenging to use if you don't understand those concepts. Hell, I write Javascript nearly every day and I still have trouble sometimes with dealing with callback hell. That's not to say that Javascript isn't a good choice though. It is INCREDIBLY powerful, and if you know Javascript well you can literally write software for any platform in it. It's just getting over the initial learning curve. Both have great libraries for web scraping and parsing HTML and bot will get the job done.

I personally use Python whenever I can because it is the language that I know best, but it's not always the most convenient. There is no answer to your question, it is going to come down to the task that you are trying to complete.

Web scraping works by first writing a program that downloads a webpage with the expected HTML. The same HTML that you get in your regular web browser. This is easier said than done because you'll need to use Proxies do to websites that try to block your scraper. The next program looks at the HTML and picks out the information that you want then you do stuff with that data. Typically, a web scraper is a small part in a bigger application. That could be a desktop app, a web app, or just a script. It all depends on what you are trying to do.
Click to expand...

Thank you @Gazo for the great advice. I guess I‘ll choose Python.

It would be great if you guys could answer one last question,
I dont know how to code yet, but I think I understand the philosophy behind it.

How does a script work?
Does it work like this:
  • Go to these websites
  • Look for comics/manga with a certain tag
  • Take their info and links
  • Put the info and links in my database
  • Now gonto my websites CMS and create pages and put the comics in them
Does it work like that? And would those steps I‘d take be alright or should I do it differently?

If you would scrape something, for instance comics, manga, how would you do that? Would you scrape the links and put them in your database or download them and put them on a img-hoster?
 
Python may be faster but node.js is definitely easier to learn. I think you shouldn't concern yourself with performance differences from the beginning because in web scraping you will encounter several other problems that need solving before hitting a performance bottleneck.
I personally would start with node because you can use the knowledge for buiding websites (both front end and backend). Also, you can find tons of tutorials and npm's for webscraping. For example you can make a basic one just with 2 libraries: 'request' and 'cheerio'. Cherio is extremely easy to learn because is basicaly just like JQuery (using css selectors). You could use request, cherio for scraping and for storage go with mongodb (json like storage) with mongoose because they are very easy to use.

I personally know both python and node but for any type of scraping project I prefer nodejs because I know it better and is fast enough for anything I needed so far.

Some aspects to consider while building a scraper are:
- rate limiting for specific domain
- ip changer (depending on the website you are scraping)
- user agent changer
- very good error handling in case something goes wrong
- ability to resume from where it stopped in case of any crashes

Some websites have better protection for scraping, some have cloudflare which is harder to scrape because they will ban your ip after couple of hundreds of requests.

Now for the secret ingredient for webscraping: try to make it look like you are the googlebot which will give you way less bans because nobody wants to ban the googlebot.
 
Apparently i am the only old school guy who still uses Java +selenium to scrape. Works like a charm for me. Just to say, use whatever you are comfortable with. Good luck!
 
I prefer Python cause is more for Data Science.
So, thats it.
 
Advertise on BHW
Status
Not open for further replies.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top