1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scraping google SERPs

Discussion in 'Black Hat SEO' started by stash, Aug 26, 2008.

  1. stash

    stash Newbie

    Joined:
    Mar 28, 2008
    Messages:
    10
    Likes Received:
    0
    looking for script that will scrape top 10 google search results for a keyword and then insert the results in text format to pages. would work on pages that are php and would need to look up the results based on the php ID.

    any help appreciated
     
  2. Chris Devon

    Chris Devon Power Member

    Joined:
    Jul 2, 2008
    Messages:
    509
    Likes Received:
    192
    Can you be more specific. Maybe I can help
     
  3. unknown_zero

    unknown_zero Jr. VIP Jr. VIP

    Joined:
    May 15, 2008
    Messages:
    2,130
    Likes Received:
    766
    make your own dude, it's very easy
    hxxp://www.oooff.com/php-scripts/php-scraping-scripts.php
     
    • Thanks Thanks x 1
  4. bps_

    bps_ Newbie

    Joined:
    Jul 30, 2008
    Messages:
    1
    Likes Received:
    1
    Here is a script that scrapes top 10 results from google for the desired keyword and stores the results in a text file.
    Code:
    $keyword = urlencode('keyword you want to search for');
    
    $url = "http://www.google.com/search?q=".$keyword."&num=10&hl=en&start=0&sa=N";
    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_USERAGENT , "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16");
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    $webpage = curl_exec($ch);
    curl_close($ch);
    preg_match_all("/<h3 class=r><a href=\"(.*?)\" class=l onmousedown=\"(.*?)\">(.*?)<\/a>/",$webpage,$match);
    
    $size = (sizeof($match[1]) -1);
    for ($i = 1; $i <$size ; $i++)     //60 Articles
      {
    		$foundLinks[] = $match[1][$i];	
      		
      }
      foreach ($foundLinks as $links)
      {
      	fwrite($fh, $links);
      	fwrite($fh, "\r\n");
      }
      fclose($fh);
    
     
    • Thanks Thanks x 1
  5. inggik2

    inggik2 Registered Member

    Joined:
    Oct 29, 2008
    Messages:
    58
    Likes Received:
    18
    Location:
    USD / EUR / JPY / IDR
    @ unknow_zero.

    Yeah i think i find a lot of shit over there. thanks anyway
     
  6. headspin

    headspin Regular Member

    Joined:
    Jun 3, 2008
    Messages:
    234
    Likes Received:
    143
    Home Page:
    If this is for automatic content generation, maybe you should consider the MSN Live Search API. It is much more flexible and you won't get banned for "automatic" requests like Google. You can get more info at
    http://dev.live.com/livesearch/
     
  7. the_demon

    the_demon Jr. Executive VIP

    Joined:
    Nov 23, 2008
    Messages:
    3,218
    Likes Received:
    1,591
    Occupation:
    Search Engine Marketing
    Location:
    The Internet
    Google banned me in 3 different states! just for gathering data 3x/ 4days a week. They also banned a friend of mine in antoher state. I wasn't scraping jsut using an automaeted tool(s) to gather serp data.

    NOTE: By banned i mean they screw with the SERP really badly (for me only).

    For example: I'm on "page 20"... however if you check from any other computer it's page 3.

    They do this so I can't see which SEO techniques are working. Though they have only done this to 1 website of mine???
     
  8. ukescuba

    ukescuba Senior Member

    Joined:
    Feb 24, 2008
    Messages:
    994
    Likes Received:
    635
    Occupation:
    Mobile Marketer & QR Code Junkie
    Location:
    San Antonio, TX
    Home Page:
    sign up for a free account with google alerts - the rest will become obvious

    demon you possibly experiencing the google dance...
     
  9. macpaulos

    macpaulos Regular Member

    Joined:
    Oct 14, 2009
    Messages:
    295
    Likes Received:
    53
    2 Firefox addons

    Download SEOquake to provide data and download into txt or CSV Files
    Download FLEM This loads txt files into Firefox

    Step 1 - Type any keyword into google or a few keywords

    Step 2 - Copy google's URL and paste it into an excel Sheet
    I tried putting the full search string but didn't come up for some reason

    Step 3 - Split the URL, copy the first part of the URL up until your first keyword and put in a seperate cell
    http://www.google.com/search?hl=en&num=100&q=

    Step 4 make cells for each keyword all starting with a + sign

    Step 5 paste the end of the google URL into the cell.
    &cts=1266480684079&aq=f&aqi=&oq=

    Step 6 - place all keywords in those cells and copy and paste google's URL to match the amount of keywords you have

    Step 7 - Copy and paste the whole lot into notepad and save

    Step 8 - Load your new text file into FLEM and push next

    Step 9 - Use SEOQuake to save the results (should be above the first result)

    Step 10 - Enjoy

    Sorry for the amount of steps just wanted to make it easy to understand. If you know imacro's or something then you could possibly make this fully automated. FLEM also allows a slideshow which means it will press next for you at intervals you setup.
    You can also set SEOquake to show as much or as little data you want. So if you only want URL's or if you want the full works (PR, backlinks ect) it's up to you.
     
    • Thanks Thanks x 1
    Last edited: Feb 18, 2010
  10. kaeen

    kaeen Newbie

    Joined:
    Dec 25, 2009
    Messages:
    17
    Likes Received:
    18
    Thanks man! Can you share link download SEOquake tools, i finding it along time but not see anywhere!
     
  11. anodyne

    anodyne Newbie

    Joined:
    Apr 5, 2008
    Messages:
    19
    Likes Received:
    6
    Did you happen to check seoquake.com? ;)
     
    • Thanks Thanks x 1
  12. justone

    justone Elite Member

    Joined:
    Oct 12, 2008
    Messages:
    1,519
    Likes Received:
    1,037
    Occupation:
    -
    Location:
    Europe
    http://google-scraper.squabbel.com

    This contains all information you need to scrape google, tells you how to avoid their detection and contains an advanced scraper (PHP) that was already used to scrape a few million hits without any banning problem.

    That's the right source for people who want to professional scrape SERP data and either know a bit PHP or have a programmer to customize the work.
    I'd add mysql support for example.
     
    • Thanks Thanks x 1
  13. Bostoncab

    Bostoncab Elite Member

    Joined:
    Dec 31, 2009
    Messages:
    2,256
    Likes Received:
    514
    Occupation:
    pain in the ass cabbie
    Location:
    Boston,Ma.
    Home Page:
    Uhhm.. have you tried logging out of google and clearing your cookies? also Some ISPs now offer the ability to update your ip id ask yours.
     
  14. Bostoncab

    Bostoncab Elite Member

    Joined:
    Dec 31, 2009
    Messages:
    2,256
    Likes Received:
    514
    Occupation:
    pain in the ass cabbie
    Location:
    Boston,Ma.
    Home Page:
    Im looking for a list of forums I can join that have either the words Boston,Taxi,Cab or a combo or 2 or all three in their domains. IF you get really bored you can list them in a spread sheet with a direct link to the signup pages and if you are really bored you can list them in descending order of page rank of the main page.
     
  15. macpaulos

    macpaulos Regular Member

    Joined:
    Oct 14, 2009
    Messages:
    295
    Likes Received:
    53
    • Thanks Thanks x 1
  16. cbnoob

    cbnoob Senior Member

    Joined:
    Sep 27, 2010
    Messages:
    967
    Likes Received:
    456
    Thanks bro but I think your code has stopped working. the reason could be now google use ajax to generate the results, not plain html anymore.
     
  17. soma56

    soma56 Regular Member

    Joined:
    Jun 16, 2009
    Messages:
    276
    Likes Received:
    155
    Home Page:
    Technically, you can still use cURL to scrape Google serps. While Google has modified their algorithm their search results are still in plain html. As a result you can use PHP's cURL to scrape the results.