Scrape the Web: Strategies for programming websites that don't expected it

[MrBlue]

An interesting lecture on scraping/browser automation methods and strategies using Python.

* Choosing a parser: BeautifulSoup, lxml, HTMLParse, and html5lib.
* Extracting information, even in the face of bad HTML: Regular expressions, BeautifulSoup, SAX, and XPath.
* Automatic template reverse-engineering tools.
* Submitting to forms.
* Playing with XML-RPC.
* Countermeasures, and circumventing them:
o IP address limits
o Hidden form fields
o User-agent detection
o JavaScript
o CAPTCHAs
* Plenty of full source code to working examples:
o Submitting to forms for text-to-speech.
o Downloading music from web stores.
o Automating Firefox with Selenium RC to navigate a pure-JavaScript service.
* Q&A; and workshopping
* Use your power for good, not evil.

Video:

http://python.mirocommunity.org/video/1616/pycon-2010-scrape-the-web-stra

 

[sw1344]

Mmm.. Nice..

I'd better brush on my python.

Many thanks

 

[wu1239]

the video is fine and i found the guy's site:
google: asheesh laroia pycon
(i can not post links now)

 

[Frogserv]

* Use your power for good, not evil.

Evil is good enough

 

[wu1239]

i also found the guy's slides links:
i just could not post the link, so I give google keyword again:
google: "stats pop quitz" ext:pdf