1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Logging into a website and scraping content

Discussion in 'BlackHat Lounge' started by davids355, Apr 27, 2013.

  1. davids355

    davids355 Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 25, 2011
    Messages:
    8,805
    Likes Received:
    6,372
    Home Page:
    Is there any software or method that would allow you to log into a website and fill and form and scrape some data from a page, all via php or similar (rather than by running a local application)?
     
  2. sunsilk

    sunsilk Newbie

    Joined:
    Dec 2, 2011
    Messages:
    48
    Likes Received:
    3
    Yes, simple CURL with PHP can do it.
    Just search google with CURL PHP Scraping and you will find many examples.
     
  3. YouFeelMeDawg?

    YouFeelMeDawg? BANNED BANNED

    Joined:
    Aug 10, 2011
    Messages:
    266
    Likes Received:
    371
    [sarcasm]Why curl, just go straight sockets and build an http wrapper request[/sarcasm]
    Obviously he was asking for some sort of software, or perhaps a framework would do good.

    Look into scrapy.org, is a python library for scraping. Really good scraping framework. Now for filling forums and logging in, I find it is a lot easier to use slybot(which is based on scrapy and scrapely)
    http://scrapy.org
    https://github.com/scrapy/slybot

    Now if you want to take it a lil bit further, you might want that scraper deployed on a public cloud(amazon ec2, rackspace,azure etc) or private cloud(your own hypervisors like vmware,kvm etc)
     
  4. davids355

    davids355 Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 25, 2011
    Messages:
    8,805
    Likes Received:
    6,372
    Home Page:
    Thanks guys.