What kind of Bot / Web Scraper do you need ??

[DrGonzo]

Hi

I want to building a Bot/Web scraper and i would like to get some ideas from the forum of what type of information they would like to get scraped from websites, forums,blogs etc etc.

Throw your suggestions at me please.

Thanks

_drgonzo_

 

[stan]

Hey dude

You know what would be awsome. If you build a software that will allow your html and htm sites to be scrapped by thousands of other sites. Rather than building rss feeds to accomplish this. Also the software should have a built in function or attachment that will crawl your websites and extract other links within it.

Now that would be cool

 

[xXKingdom_SEOXx]

that would be awesome. stan suggestion is great.

 

[stan]

Once again thank you bro. If I can get a list of sites that you can ping html to for scrapping I may try my hand at building one of these myself and share it among my new friends here at BHW.

 

[Wombat]

Prstorm hits a lot of sites in the hope that some of them make their stats page public, which most don't.

If you Google
http://www.google.com/search?hl=en&q=webalizer+intitle%3A%E2%80%9Dusage+statistics+for+www%E2%80%9D&btnG=Google+Search
it shows 148,000 sites that do.

Is there some way to get those 148,000 URLs ?
That would be very handy.

 

[asgdude]

Yes, this would be a very nice idea.

 

[stan]

I wouldn't mess with PRStorm if I were you. If your ISP won't ban you for using up all their bandwidth for running this program or for the complaints the
webmasters will send to them for you referral log spamming their sites. You probably will still have to be worried about being banned from google, yahoo and msn once they realize where your backlinks are coming from. Or the hacker sites automatically sending surprises back to you once you start spamming them.

To each his own. If you still wanna try it have fun.

 

[Denise]

I'd like a scraper bot that scrapes the feeds that you tell it to scrape, automatically rewrites the articles and posts them to your blog. lol

 

[Wombat]

I'd like a scraper bot that scrapes the feeds that you tell it to scrape, automatically rewrites the articles and posts them to your blog. lol

Blog Solution does all that and more.