Working on a crawler for PR Storm (need suggestions)

[xpro]

I have started to word on a crawler so it can crawl pages for PRStorm, the part I need help on is that I don't know which sites publish their logs, Is there anyway to find those sites with Google search? something like "Powered by awstats"?

 

[entreprenuer]

well why don't you just crawl for the logs, and If you find the log then it has to be published right?

Im not smart on this kind of stuff so correct me if Im wrong.
hope I can help

 

[xpro]

well why don't you just crawl for the logs, and If you find the log then it has to be published right?

Im not smart on this kind of stuff so correct me if Im wrong.
hope I can help

yea, that would be the best, but how can I find sites that have logs?

 

[spaceinvader]

you mean stuff like this?

intext:?Created by awstats?
?Generated by Webalizer Version 2.01″
?Generated by Webalizer Version *?
?Top * of * Total Referrers?
?usage statistics? ?Summary Period: August 2008″ (change the month to get more results)

inurl:yourkeyword +usage_200804 (enter your keyword and change the date for more results)
inurl:/webalizer/usage_200806.html (change the date for more results)
inurl:.com/webalizer/usage_200806.html (change the date and .com for more results like)
inurl:.edu/webalizer/usage_200805.html
 inurl:.net/webalizer/usage_200804.html
 inurl:.org/webalizer/usage_200803.html

 

[entreprenuer]

you mean stuff like this?

intext:?Created by awstats?
?Generated by Webalizer Version 2.01″
?Generated by Webalizer Version *?
?Top * of * Total Referrers?
?usage statistics? ?Summary Period: August 2008″ (change the month to get more results)

inurl:yourkeyword +usage_200804 (enter your keyword and change the date for more results)
inurl:/webalizer/usage_200806.html (change the date for more results)
inurl:.com/webalizer/usage_200806.html (change the date and .com for more results like)
inurl:.edu/webalizer/usage_200805.html
 inurl:.net/webalizer/usage_200804.html
 inurl:.org/webalizer/usage_200803.html

wow! so If I search those strings in a google serp scraper I will find a list of links that can be pinged?