C# Vs Python For Advanced Web Scraping?

B

black_hat_newbie

Newbie
Joined
Jan 12, 2009
Messages
28
Reaction score
1
hi guys,
I want to be able to scrape google maps, google places, amazon, facebook and other websites like yellowpages, yelp, realtor.com, etc. and save the data in an excel sheet or database.

Which would be a better one for me to learn? or both would be same and it wont matter?
 
Advertise on BHW
Good luck. I would point you in the direction of python because it is more lightweight and flexible.
 
thanks trance.

Would c# be able to do the same things efficiently and does c# also have good scraping libraries like python?
 
+1 for Python. There are already many libraries and code snippets available related to scraping and data collection.
 
Ive been writing scrapers and small bots over the last few weeks. I've found you should decide what scraping tools you want to use first then look at what language you can use with it. C# has htmlagility pack which a few people use for twitter bots etc. I'm currently using a combination of selenium and phantomjs with php/python/casperjs. Add me on Skype if you want to discuss mate :-) jmcostello21
 
If the learning curve is fine for me for c#, does c# has as good libraries as python for advanced web scraping?
is htmlagility as good as the ones that are available for python.

I was considering c# if it would let me do advanced scraping and at the same time build it into desktop apps easily. But only if it can do advanced webscraping well. If not as easily as python but with medium ease. If c# is going to be a pain and a lot lot more effort, then i feel python would be better.

I wanted to learn a proper programming language too which is systematic has graphical UI and powerful if it can be used for advanced we scraping well.

What's your take?
 
black_hat_newbie said:
If the learning curve is fine for me for c#, does c# has as good libraries as python for advanced web scraping?
is htmlagility as good as the ones that are available for python.

I was considering c# if it would let me do advanced scraping and at the same time build it into desktop apps easily. But only if it can do advanced webscraping well. If not as easily as python but with medium ease. If c# is going to be a pain and a lot lot more effort, then i feel python would be better.

I wanted to learn a proper programming language too which is systematic has graphical UI and powerful if it can be used for advanced we scraping well.

What's your take?
Click to expand...

Do you want to just scrape and parse or scrape and render pages? When I say render I mean execute JavaScript and display CSS.
 
at the moment, I just need to store the data in excel/database. But in the future might need to display it.

can you do advanced web scraping with c# well without much pain like you do in python?
 
You're comparing 2 languages with great support.

Python is interpreted and C# is compiled.

Python will be quicker to code, but C# is easier to protect.

So, Python for personal jobs, C# for something you plan to distribute.

Just a few comparisons. So if you're just scraping for your own good, Python is the clear winner.
 
I myself use Node.js and Ruby. It depends on your goals. With a site like import.io you can scrape without really knowing how to code. If you're learning to develop the skill, unless the site is advanced at thwarting scraping you can use a high level language. This of course would be a scripting language; so yes Python with Beautiful Soup or Scrapy would be great ways to quickly write something where you can buy a proxy on a server, run your script on schedule with a cron, and get regular data with minimal maintenace. I like csv/json. Have fun!
 
black_hat_newbie said:
snip
Click to expand...

C# is compiled and thus highly unnecessary for parsing/scraping. Learn python, it will be faster to learn and well coded python is actually pretty good performance if at some point down the line you need speed. (You won't until and if you need to scrape thousands of sites every minute or so)
 
Having pretty good knowledge of both Python and C#, I'd definitely go for C#. Python will be great for small projects, but if you want a language you can develop further then C# is the way. I also found C# easier to learn than Python, but maybe that's just me
 
I've used both C# and Python for scraping projects (and exporting them to excel). Both languages are more than capable for the advanced scraping you speak of. So, I'd say you may need to think about other factors to determine which one you choose. Python, in general, cuts down production time by quite a lot compared to C#. Both languages have their different libraries you can use for scraping, so you may want to consider which library is easier for you to learn. You may also wish to think of the long-term, perhaps you want to get hired as a programmer one day. Currently, both languages are in demand, however this, mostly, depends on your local area. Both C# and Python have a great community. I'm sure there are other factors you may wish to take into consideration too. I hope that has helped somewhat.

I'd research and explore both languages before making a concrete choice.

P.S: If you're wondering which language I personally prefer then it's mostly C# for bigger and stable projects, and Python for quicker and prototyping. However, bare in mind that's just my personal usage and opinion.
 
Python
Selenium & Mechanize & Beautifulsoup under Python will help you to manage everything
 
as for me, python is better for the scraping tasks, cause u`ll spend less time making scraping soft in python
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top