Using Node.js to scrape websites

Node js is a pretty cool new server-side tool that lets you run javascript on the server.

We've been using it to scrape data from websites for a couple of months.

Does anybody here ever use it?

what do you do with the scraped data? does node.js have any database functionality?

ive been looking at php's simple dom parser.

but, i hear node.js and jquery make traversing the dom, pretty easy.

I also just got started with Node.js. Here are some resources to get you started with scraping in Node.js.

How do I create a HTTP Client Request with a cookie?
http://stackoverflow.com/questions/4579757/how-do-i-create-a-http-client-request-with-a-cookie

What is the most elegant way to do screen scraping in node.js?
http://stackoverflow.com/questions/...-elegant-way-to-do-screen-scraping-in-node-js

Modules:

node.io
http://www.node.io
https://github.com/chriso/node.io
http://www.coderholic.com/scraping-the-web-with-node-io

http-agent:
https://github.com/indexzero/http-agent
http://blog.dtrejo.com/scraping-made-easy-with-jquery-and-selectorga

request
https://github.com/mikeal/request

abstract-http-request
https://github.com/coolaj86/abstract-http-request

XMLHttpRequest
https://github.com/driverdan/node-XMLHttpRequest

Spider
https://github.com/mikeal/spider

node-cookiejar
https://github.com/bmeck/node-cookiejar

jsDOM/jQuery
https://github.com/tmpvar/jsdom
https://github.com/coolaj86/node-jquery


If you have any questions just PM me