Looking for SERP content scraper

stugz said:

Perl LWP is my favorite.

But you can also use curl with PHP or C.

I'm sure Python and Ruby will have simple solutions for this as well.

Click to expand...

python retriever-multi.py <file with URLs to fetch> <# of concurrent connections>

Click to expand...

<?php
function get_contents($url){
$r = file_get_contents($url);
return $r;
}


function save_contents($content, $filename){
$f = fopen($filename . ".txt", "wb");
$write = fwrite($f, $content);


if($f && $write){
fclose($f);
return true;
} else {
return false;
}


}


$arr = ['http://www.facebook.com', 'http://www.google.com', 'http://www.yahoo.com'];


for($i = 0; $i < count($arr); $i++){


print "Getting content for $arr[$i]! \n";
$body = get_contents($arr[$i]);


print "Got content! Now saving it. \n";


if(save_contents($body, $i)){
print "Saved content to {$i}.txt! \n";
} else {
print "Failed to save content! \n";
}
}


?>

Click to expand...

apoorv said:

https://github.com/blackstonetech/pycurl/blob/master/examples/retriever-multi.py would work if you are okay with Python. It's a little bit of an overkill for something like this, but it's fast, serves the purpose, and it's already written.

The usage is something like this:



You can add proxies, etc. to this script, too, if you are scraping on a large scale. It's a few lines of code, let me know if you want me to add it?

Also, doing something like this would be pretty simple in PHP, too. Something like this:



This isn't very robust or very fast, but it will get the job done. I tested it out with a couple of URLs. You can also use cURL with PHP, that will be faster.

Click to expand...

hadoken said:

I have not used them but check these out to see if they fit your needs

http://www.blackhatworld.com/blackh...-matter-what-your-keyword-take-challenge.html

Click to expand...