Google Scraper

WORK@HOME

Senior Member
Joined
Apr 25, 2013
Messages
1,013
Reaction score
479
Hey guys.I have this section of code I can't get to work to scraoe google search results.I need help to make this work big time.

Any help will be awesome.

Code:
function get_google($kw, $res=10, $geo='US') {
    $urls = array();
    if ($geo == '') $geo = 'US';
    $url  = 'http://www.google.com/search?hl=en&as_qdr=all&gl='.$geo.'&q='.urlencode($kw).'&num='.$res;
    
    try {
        $html = curl_fetch($url);
    
        $dom = new DOMDocument();
        @$dom->loadHTML($html);
        $xpath = new DOMXPath($dom);
        $hrefs = $xpath->evaluate("/html/body//a");
        for ($i = 0; $i < $hrefs->length; $i++) {
            $href = $hrefs->item($i);
            if ($href->getAttribute('class') == 'l') {
                $link = explode('?', $href->getAttribute('href'));
                $urls[] = $link[0];
            }
        }
    } catch (Exception $e) {}
    return $urls;
}
 
Are you trying to do it many times in a short period of time? because I know that they have a captcha that pops up in that case..
 
Actually, depending on your search amount, if it is not a huge amount, then you can consider using the custom search api
developers.google.com/custom-search
where the data is much better formatted for your manipulation.

By scraping, google is smart enough to detect you are doing it through bot and disallow you for that.
Thus, theoretically speaking, it's limited in the same way as custom search.
 
this works for me
Code:
function get_google($kw, $res=10, $geo='US') {
        
        $urls = array();
        if ($geo == '') $geo = 'US';
        $url  = 'http://www.google.com/search?hl=en&as_qdr=all&gl='.$geo.'&q='.urlencode($kw).'&num='.$res;

        try {
                $html = curl_fetch($url);

                $dom = new DOMDocument();
                @$dom->loadHTML($html);
                $xpath = new DOMXPath($dom);
                $hrefs = $xpath->evaluate("/html/body//h3/a");
                for ($i = 0; $i < $hrefs->length; $i++) { 
                        $href = $hrefs->item($i);
                        if ( preg_match('/url\?q=(.*?)&sa/', $href->getAttribute('href'),$result) ) {
                                $link = urldecode( $result[1] );
                                $urls[] = $link;        
                        }
                }
        } catch (Exception $e) {}
        return $urls;
}
 
Hey guys...Thanks for the help.I actually found out that I could do it with Scrapebox.
 
Back
Top