Scraping google SERPs

macpaulos said:

2 Firefox addons

Download SEOquake to provide data and download into txt or CSV Files
Download FLEM This loads txt files into Firefox

Step 1 - Type any keyword into google or a few keywords

Step 2 - Copy google's URL and paste it into an excel Sheet
I tried putting the full search string but didn't come up for some reason

Step 3 - Split the URL, copy the first part of the URL up until your first keyword and put in a seperate cell
http://www.google.com/search?hl=en&num=100&q=

Step 4 make cells for each keyword all starting with a + sign

Step 5 paste the end of the google URL into the cell.
&cts=1266480684079&aq=f&aqi=&oq=

Step 6 - place all keywords in those cells and copy and paste google's URL to match the amount of keywords you have

Step 7 - Copy and paste the whole lot into notepad and save

Step 8 - Load your new text file into FLEM and push next

Step 9 - Use SEOQuake to save the results (should be above the first result)

Step 10 - Enjoy

Sorry for the amount of steps just wanted to make it easy to understand. If you know imacro's or something then you could possibly make this fully automated. FLEM also allows a slideshow which means it will press next for you at intervals you setup.
You can also set SEOquake to show as much or as little data you want. So if you only want URL's or if you want the full works (PR, backlinks ect) it's up to you.

Click to expand...

kaeen said:

Thanks man! Can you share link download SEOquake tools, i finding it along time but not see anywhere!

Click to expand...

the_demon said:

Google banned me in 3 different states! just for gathering data 3x/ 4days a week. They also banned a friend of mine in antoher state. I wasn't scraping jsut using an automaeted tool(s) to gather serp data.

NOTE: By banned i mean they screw with the SERP really badly (for me only).

For example: I'm on "page 20"... however if you check from any other computer it's page 3.

They do this so I can't see which SEO techniques are working. Though they have only done this to 1 website of mine???

Click to expand...

bps_ said:

Here is a script that scrapes top 10 results from google for the desired keyword and stores the results in a text file.

Code:

$keyword = urlencode('keyword you want to search for');

$url = "http://www.google.com/search?q=".$keyword."&num=10&hl=en&start=0&sa=N";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT , "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$webpage = curl_exec($ch);
curl_close($ch);
preg_match_all("/<h3 class=r><a href=\"(.*?)\" class=l onmousedown=\"(.*?)\">(.*?)<\/a>/",$webpage,$match);

$size = (sizeof($match[1]) -1);
for ($i = 1; $i <$size ; $i++)     //60 Articles
  {
        $foundLinks[] = $match[1][$i];    
          
  }
  foreach ($foundLinks as $links)
  {
      fwrite($fh, $links);
      fwrite($fh, "\r\n");
  }
  fclose($fh);

Click to expand...

cbnoob said:

Thanks bro but I think your code has stopped working. the reason could be now google use ajax to generate the results, not plain html anymore.

Click to expand...