I mostly use
Zend Framework 's classes for scraping things. I 'm posting some of my code that 's
easy to read and will help you (or anyone) start with
web scraping.
Study it and make it yours, especially the second part.
Here 's my
grabber class, it returns the
HTML code and stores the
time it took to grab it and the
HTTP response code (e.g. 200, 404 etc). See
Zend_Http for details.
PHP Code:
class Networking
{
public $pageLoadTime = 0;
public $respstatus;
function getPage($url)
{
$client = new Zend_Http_Client($url, array('maxredirects' => 2, 'timeout' => 10,
'useragent' => 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.3) Gecko/20010801 '));
$time_start = microtime(true);
$code = $client->request("GET");
$time_end = microtime(true);
$time = $time_end - $time_start;
$this->pageLoadTime = $time;
$this->respstatus = $code->getStatus();
return $code;
}
}
After the HTML code has been fetched, I create the
DOM document with
Zend_Dom
Here 's my class for scraping
Google Blogs
PHP Code:
<?php
class BlogResults
{
public $url;
public $anchor;
}
class GoogleBlogSearch
{
public $pageBody = '';
public $pageDom = '';
function getPage($url)
{
$url = noWWW(noHTTP($url));
$worker = new Networking();
$page = "http://blogsearch.google.com/blogsearch?hl=en&q=" . noVars($url);
$code = $worker->getPage($page);
$this->pageBody = $code->getBody();
$this->pageDom = new Zend_Dom_Query($this->pageBody);
}
function parseResults()
{
$results = $this->pageDom->query('a');
if (0 < count($results)) {
$tmp_links = array();
foreach ($results as $result) {
if (!$this->validLink($result->getattribute("href")))
continue;
if (in_array($result->getattribute("href"), $tmp_links))
continue;
$tmp = new BlogResults();
if ($result->hasattribute("href"))
$tmp->url = $result->getattribute("href");
$tmp->anchor = $result->textContent;
$tmp_links[] = $tmp->url;
$ret[] = $tmp;
}
}
return $ret;
}
function validLink($url)
{
if (preg_match('#google\.#', $url)) {
return false;
}
if (!preg_match('#http:\/\/#', $url)) {
return false;
}
if (preg_match('#youtube\.#', $url)) {
return false;
}
return true;
}
}
?>
Enjoy
