Logging into a website and scraping content

[davids355]

Is there any software or method that would allow you to log into a website and fill and form and scrape some data from a page, all via php or similar (rather than by running a local application)?

 

[sunsilk]

Yes, simple CURL with PHP can do it.
Just search google with CURL PHP Scraping and you will find many examples.

 

[YouFeelMeDawg?]

Yes, simple CURL with PHP can do it.
Just search google with CURL PHP Scraping and you will find many examples.

[sarcasm]Why curl, just go straight sockets and build an http wrapper request[/sarcasm]
Obviously he was asking for some sort of software, or perhaps a framework would do good.

Look into scrapy.org, is a python library for scraping. Really good scraping framework. Now for filling forums and logging in, I find it is a lot easier to use slybot(which is based on scrapy and scrapely)
http://scrapy.org
https://github.com/scrapy/slybot

Now if you want to take it a lil bit further, you might want that scraper deployed on a public cloud(amazon ec2, rackspace,azure etc) or private cloud(your own hypervisors like vmware,kvm etc)

 

[davids355]

Thanks guys.