Google waiting to be raped!!!

Can someone scrape the profiles using this to scrape loads of profiles with buzz on:

Code:
t:buzz site:http://profiles.google.com
 
there is approximately 17.5 million urls in this list. That is a fair few.
 
For anyone interested here is a bit of code for you.

Collect all links and store to a text file(This gets rather big, ~700mb or so).
PHP:
<?php

$file = "gprofiles.txt";
$fh = fopen($file, 'a') or die("can't open file");

$i = 000;
while ($i <= 3449) {
    if($i < 10) { $i = str_pad($i, 3, '0', STR_PAD_LEFT); }
    if($i < 100) { $i = str_pad($i, 3, '0', STR_PAD_LEFT); }
    $text = file_get_contents("http://www.gstatic.com/s2/sitemaps/sitemap-$i.txt");
    $i++;
    $profiles = $text."\r\n";
    fwrite($fh, $profiles);
}


?>

Once you scrape the links, if you're interested in emails:
PHP:
<?php
ini_set('memory_limit', '128M');
$file = "gprofiles-emails.txt";
$fh = fopen($file, 'a') or die("can't open file");

function extract_emails_from($text){
  preg_match("/[\._a-zA-Z0-9-]+@[\._a-zA-Z0-9-]+/i", $text, $matches);
  return $matches[0];
}

$lines = file("gprofile-001.txt");

foreach($lines as $line)
{
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $line);
    curl_setopt($ch, CURLOPT_USERAGENT, 'Opera/9.23 (Windows NT 5.1; U; en)');
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST,  1);
    $a = curl_exec($ch);
    $emails = extract_emails_from($a);
    if(empty($emails)) {
        echo "empty\r\n";
    }else{
        echo "found\r\n";
        $mails = $emails."\r\n";
        fwrite($fh, $mails);
    }
}
?>

You need to split the text file generated with the first PHP script into smaller chunks, you may have to adjust the memory_limit in the 2nd piece of code.

These are written to be ran from the command line, if you want to run them from a browser you'll need to add "set_time_limit(0);" at the start of each file. If you want to scrape other data from the pages it shouldn't be hard to modify the code I've provided.
 
All these lists of profiles were offered on the same day, 2010-07-21.
Their URLs are listed here (between the <loc> and </loc> tags):
http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml

As f0rked correctly identified, the file names go from sitemap-000.txt to sitemap-3449.txt. There is no file 3500.txt and above but there certainly are profiles created after July 2007. It seems like a one day sting operation.

Just be careful of a possible honey trap as 8 months after the list came online it's still not spammed to death.
 
When I posted that code, it hadn't finished gathering all of the URLs, it's now finished and the file size is a bit bigger than I said. It's actually 867mb, and there are 17,173,113 URLs.
 
This could quite easily be manipulated to get a lot of spam messages out there, even if backlinks are easily deleted.
 
Don't just look at it from the SEO point of view... its a data goldmine... think creative..
 
When I posted that code, it hadn't finished gathering all of the URLs, it's now finished and the file size is a bit bigger than I said. It's actually 867mb, and there are 17,173,113 URLs.

Have you tried doing emails? I don't see any profiles with plaintext emails up there.
 
Attention:
The content on the sites is NOT in the google index!
So it may just count as unique in the Big Gs eyes....
 
Have you tried doing emails? I don't see any profiles with plaintext emails up there.

Yes, there are emails listed in some of the profiles. The regex used in the 2nd bit of code isn't the best, so it returns a lot of junk as well, but I pulled ~650 real emails before I stopped it from harvesting any further. If you a VPS/Dedi that you can run it on, set the memory limit to something pretty high and pass it a rather large list of URLs and just let it gather data. I'd strongly suggest changing that regex to something else though.
 
This was posted months ago. I'm sure if you snoop around you'll find an already compiled list of scraped urls from it.
 
Back
Top