Writing a crawler

BetterKnow

BetterKnow

Senior Member
Jr. VIP
Joined
Mar 17, 2018
Messages
1,117
Reaction score
1,291
I need to write a crawler for a project of mine that needs to be extremely scalable and effecient.

I was advised to use a lower level language, specifically C++

Anyone with experience doing something similar, I would love to hear from you
 
Advertise on BHW
Better Know said:
I need to write a crawler for a project of mine that needs to be extremely scalable and effecient.

I was advised to use a lower level language, specifically C++

Anyone with experience doing something similar, I would love to hear from you
Click to expand...
So you have a high level problem and you are choosing a low level(mid-level) language! Why? Why create more problems? Just write it in Perl or Python or Ruby!
 
For anything scraper related, I would go with python. For scalable crawlers, you can use something like java.
 
I write mine in c#. Works well so far.
 
Who told you C++ would be best?! No way I'd recommend that. I'd say Python (or Ruby) is perfect for this and really easy, and if you want something a more low-level with better performance, use Go.
 
Agree with the avoid c++ sentiments. You can always drop back into c++ from pretty much any scripting language if you really need to. You will just waste weeks/months coding in c++ something that could be done in half the time in something else.
 
Go with Python instead of C++, because is easy to code web crawlers.
 
Would recommend python or dotnet as well, however if you want to focus on high performance then c++ might be a good option.

Although the difference might not be very significant, especially if you take dotnet's overhead out of the equation, I'm still certain that Big crawlers such as Google's are all written in c++.
 
Check scrapy. Dont go for low level language. Its just stupid idea. Software written i low level language like C++ will use less resources but resources make no difference unless you need to run your crawler on thousands of dedicated servers. Only in such scenario low level language give some real profits because you can save money on hardware costs.
 
Python would be a great choice but if you are skeptical then use Golang instead of c++
 
I would recommend python as it will be much faster to develop and they have a ton of libraries to make it easier. ( Beautiful Soup + urllib)
 
Do you have an experience working with C++? If not, there is a pretty steep learning curve, so it may be best if you used a language that you are already familiar with. As others have mentioned above, fast and scalable crawler can be implemented using many programming languages like Java, Python, C#, Ruby and the list goes on. My advice is to go with the one that you are most familiar with.
 
I understand that this webclawler is to go through websites for e-mail addresses? I will never be amazed WHY people recommend to such things languages like C++. I think it's based on mass opinins like "C++ good for everything". I'll tell you something, buddy. I have pretty good experience with Delphi and there is nothing that I couldn't write in Delphi what is made in C++. And I can tell that writing crawler with Delphi will be 100 times harder than in Python. Here you have tutorial how to write crawler in python:

Have a nice one ;)
 
As other people have implied, C++ is literally the last language you should consider writing this in. Any of the other languages mentioned would be better. C#, Python, Go, etc
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top