Which language for spider/crawler

F

Flusel

Newbie
Joined
Mar 20, 2010
Messages
1
Reaction score
0
Hi,

I need some advice in which language to choose for programming a spider/crawler. I need some basic functionality like URL parsing and remote file access and I want to do it with multithreading. I know Java has some Issues regarding URL Parsing functions and multithreading.
 
Advertise on BHW
Any modern language can cope with that (but it 's harder to do with PHP due to lack of proper threading). Python, Java, Ruby even C++ are good choices.
 
for multithreading i would go with vb.net or c++. they also have great 3rd party libraries to deal with parsing..
 
How come no one mentions Perl anymore :D
Go Perl, has all the modules you will need and does multithreading.....nice easy scripting.

Who needs these new fangled languages when you have Perl :D

By the way - I suggest Perl :D
 
Flusel, don't think java have issues, I have used it with htmlunit and it worked just as expected.

You can scrap with most programming and scripting languages this days some will give you best performance, other readability it really depends on how you make it, to what youre going to use it.

If I had to recommend a language to you I would say C# or PHP they are more commonly used, have lots of resource, tutorials and other things available for learning including but not limited in regards crawling.

C# have libraries like HTMLAgilityPack and others for parsing HTML at easy, libraries like JINT for javascript, you can also implement multithreaded browsers for contents that need it and there is a lot more to it, it also does well with multithreading.

PHP can be used for multithreaded too, it is a different way as spawning childs but works very well and stable.

It all depends on the knowledge of the developer on the tool he is using to ensure that the code is well made to its usage.
 
How come no one mentions lisp anymore
clj-html, has all the modules you will need and does multithreading.....nice easy scripting.

Who needs these new fangled languages when you have lisp

By the way - I suggest lisp
 
Code: 
(DEFUN BECAUSE ()  "TOO MANY ()")

:D j/k I know nothing about it. :o Only time I used lisp was about two decades ago in Autocad to make a triangulation function :)
 
j/k I know nothing about it. Only time I used lisp was about two decades ago in Autocad to make a triangulation function
Click to expand...

Hahaha! That just made my day. Valid syntax makes it even better lol.

In all seriousness, I never understood the parenthesis hate to be honest... Maybe I just got used to them, not sure haha.
I've switched to clojure a year or so ago (not because of the reduced parenthesis mind you) but find myself going back to SBCL every now and then (we also got some codebase back at work running on lisp as well so I'm happily hacking away).

Autocad + lisp sounds like good fun btw.
 
I'd suggest using Python for any http automation. It uses barely any resources, and has a few nice libraries that update the response buffer in real time (in case content is updated via an XHR). It also will run on any operating system (which means you can host it on your home connection, your server, etc).
 
You can create that in javascript and run in any browser.. that would be easiest way..
Other options are C#/Python/iMacro blah blah
 
Perl is the king of this field, and always will be. The language was designed for this sort of thing.
 
Java or c#.Java has no problem either with multithreading or scraping. New fork framework, with usual thread, of java is more than enough you need for concurrency. There are very few languages which can provide better multithreading than java, erlang is one of them.
For scraping there is jsoup and for web automation you can use selenium, which can drive original firefox, ie or chrome. As it drives your original explorer it is very accurate and consistent with complex ajax calls and javascript.
Other benifit of java's are platform independent codes, sure you need to twik your codes a bit, very mature library and robust language.
 
Please check out this book by sybex.
docsgoogle/file/d/0B39j2WChNGOuSExfb0ZNLWxRUHc/edit

This is my personal file on my google drive, feel free to check em out
 
Depends on the approach - You can do it in any lang. Theoretically speaking , but the discussion is which is better :

You can Thread PHP by spawning children - The advantage of php is , well , simplicity if you are going to tie it up with MYSQL

Java works as a better language , and if you are already using JVM then the performance is well. Threading and connections are all well - but you need some info on how to optimize it (SQL pools , thread caches )

C# is the same as java , but has a better regex library - How ever if you are scaling and running from a VPS , drop this one - icky MS tech

Python - Simple and effective , didnt try it , but most crawlers are moving to it. Ask someone else

C++ - Effective and more controllable - If you know your way around it- Advanced stuff mainly

Javascript : An option , but not too well given it's security policies .

Ruby : No Threads - some other alt called green threads.

Note : You can always replace threads by a queuing system some way or another

My 2 cents :) any help needed PM me
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top