Learning to write scraping software R vs Python?!

[Cleast Intwood]

I have some programming experience from before, id say im somewhat intermediate in C#, ASP.Net and java. Im now looking to learn how to code scraping software and im considering in what language i should learn to write it in. Would R or Python be best? Or something else?

What i want is basically the path of least resistance, so as easy/closely related to what ive coded before and effective long term.

Would really apprechiate some pointers here!

 

[naskootbg]

I love to do it on php. curl or file_get_contents + preg_match,preg_replace,str_replace and strpos. Or with simple_html_dom.php. The php problem is the recaptchas v2, but if it is possible on any language it will be possible on php. There have awesome free software php-descktop wich is much better than wamp and xamp for me. Ofcource the new cpanel redactor is cool as well. Php can do multithread and proxy work as well. PHP is very easy and for scrapping tasks need to learn about 15 comands - tons of ready made classes and functions + answers on stackoverflow.

Happy New Year

 

[Cleast Intwood]

Hmm ok that would be awesome since i need to learn PHP for wordpress anyways.. Will look into this, cheers!

Anyone else? Is PHP a good alternative?

 

[neuris]

I wouldn't use R for scraping purposes. It's language mainly designed for statistical computation.

Python is good though. The syntax is great and there are fantastic libraries available for scraping, e.g. scrapy. But if you have already experience with Java, why not use it? It's got plethora of libraries: JSoup, Selenium, HtmlUnit, etc. And of course writing multithreaded scrapers is very easy in Java. I've personally used Java for almost every scraping project I've done, and it's served me well.

 

[Cleast Intwood]

I wouldn't use R for scraping purposes. It's language mainly designed for statistical computation.

Python is good though. The syntax is great and there are fantastic libraries available for scraping, e.g. scrapy. But if you have already experience with Java, why not use it? It's got plethora of libraries: JSoup, Selenium, HtmlUnit, etc. And of course writing multithreaded scrapers is very easy in Java. I've personally used Java for almost every scraping project I've done, and it's served me well.

Thanks for the input. I googled some more and found many Python recommendations, some Java.. So ur saying java gets the job done in a simple way? That would be awesome, so i dont have to put time and effort into learn a whole new language.

Is developing Java as efficient as python in scraping? Does it take about same time to code the scraper? I have overkill server so performance isnt really a huge issue for me atm.

 

[handmadebots]

Stick with python.
I've been doing automation for ... some years now. Been working in C#, Python, PHP, Java, NodeJS, etc. but my favorite one is still python.
In python you can have a fairly complex bot up and running in minutes. It has lots of libraries already built by others ready to be used, syntax is easy to understand and type.

PS: Python is not necessarily the best in all cases. Normally you pick a language depending on what software/job you have to do. In general though, being cross-platform, I would still pick python.

 

[Donald J. Trump]

Python hands down! Many libraries available also easy to understand and code!

 

[Cleast Intwood]

Stick with python.
I've been doing automation for ... some years now. Been working in C#, Python, PHP, Java, NodeJS, etc. but my favorite one is still python.
In python you can have a fairly complex bot up and running in minutes. It has lots of libraries already built by others ready to be used, syntax is easy to understand and type.

Thanks for the reply! Ok i think your right.. Time to learn python then I have done a little bit of python before but time to get deeper into it.. Found a guide on Python scraping with scrapy on udemy, gotta get up to intermediate level before that tho..

Lets do this!

 

[neuris]

Thanks for the input. I googled some more and found many Python recommendations, some Java.. So ur saying java gets the job done in a simple way? That would be awesome, so i dont have to put time and effort into learn a whole new language.

Is developing Java as efficient as python in scraping? Does it take about same time to code the scraper? I have overkill server so performance isnt really a huge issue for me atm.

I wouldn't say developing in Java is necessarily simple or very fast, but if you want something more complex than just scraping a few pages then in my opinion it's the way to go. I've used it to concurrently scrape millions of web pages. But it all depends on what you want to do.

If you want something simple up and running quickly, I agree with handmadbots that Python is probably the way to go.