Working on a crawler for PR Storm (need suggestions)

xpro · May 5, 2009

I have started to word on a crawler so it can crawl pages for PRStorm, the part I need help on is that I don't know which sites publish their logs, Is there anyway to find those sites with Google search? something like "Powered by awstats"?

entreprenuer · May 5, 2009

well why don't you just crawl for the logs, and If you find the log then it has to be published right?

Im not smart on this kind of stuff so correct me if Im wrong.
hope I can help

xpro · May 5, 2009

entreprenuer said:
well why don't you just crawl for the logs, and If you find the log then it has to be published right?

Im not smart on this kind of stuff so correct me if Im wrong.
hope I can help

yea, that would be the best, but how can I find sites that have logs?

spaceinvader · May 5, 2009

you mean stuff like this?

Code:

intext:?Created by awstats?
?Generated by Webalizer Version 2.01â€³
?Generated by Webalizer Version *?
?Top * of * Total Referrers?
?usage statistics? ?Summary Period: August 2008â€³ (change the month to get more results)

Code:

inurl:yourkeyword +usage_200804 (enter your keyword and change the date for more results)
inurl:/webalizer/usage_200806.html (change the date for more results)
inurl:.com/webalizer/usage_200806.html (change the date and .com for more results like)
inurl:.edu/webalizer/usage_200805.html
 inurl:.net/webalizer/usage_200804.html
 inurl:.org/webalizer/usage_200803.html

entreprenuer · May 25, 2009

spaceinvader said:

you mean stuff like this?

Code:

intext:?Created by awstats?
?Generated by Webalizer Version 2.01â€³
?Generated by Webalizer Version *?
?Top * of * Total Referrers?
?usage statistics? ?Summary Period: August 2008â€³ (change the month to get more results)

Code:

inurl:yourkeyword +usage_200804 (enter your keyword and change the date for more results)
inurl:/webalizer/usage_200806.html (change the date for more results)
inurl:.com/webalizer/usage_200806.html (change the date and .com for more results like)
inurl:.edu/webalizer/usage_200805.html
 inurl:.net/webalizer/usage_200804.html
 inurl:.org/webalizer/usage_200803.html

wow! so If I search those strings in a google serp scraper I will find a list of links that can be pinged?

Working on a crawler for PR Storm (need suggestions)

xpro

Regular Member

entreprenuer

Junior Member

xpro

Regular Member

spaceinvader

Regular Member

entreprenuer

Junior Member

Main Menu

Marketplace

Making Money

BlackHat World