Logging into a website and scraping content

davids355

Super Moderator
Moderator
Executive VIP
Jr. VIP
Joined
Apr 25, 2011
Messages
19,849
Reaction score
28,315
Is there any software or method that would allow you to log into a website and fill and form and scrape some data from a page, all via php or similar (rather than by running a local application)?
 
Yes, simple CURL with PHP can do it.
Just search google with CURL PHP Scraping and you will find many examples.
 
Yes, simple CURL with PHP can do it.
Just search google with CURL PHP Scraping and you will find many examples.

[sarcasm]Why curl, just go straight sockets and build an http wrapper request[/sarcasm]
Obviously he was asking for some sort of software, or perhaps a framework would do good.

Look into scrapy.org, is a python library for scraping. Really good scraping framework. Now for filling forums and logging in, I find it is a lot easier to use slybot(which is based on scrapy and scrapely)
http://scrapy.org
https://github.com/scrapy/slybot

Now if you want to take it a lil bit further, you might want that scraper deployed on a public cloud(amazon ec2, rackspace,azure etc) or private cloud(your own hypervisors like vmware,kvm etc)
 
Thanks guys.
 
Back
Top