Brand new to BH - looking to scrape statistics

[Mr Salt]

Hello, and thanks in advance for fielding my questions. This may come across as naive, and for that, I apologize. I have a good idea of what I want to do, but don't quite know how to get there. I am not a programmer, so that's the foreword.

Basically, I have a new site design that I want to launch. As part of the site, I want to compile sports statistics. However, I don't have the resources to compile them, myself. These stats need to be updated weekly. I am not using the stats as the entirety of my site - it's just an "in addition to" my content. Bloat the site, keep readers around, etc...

Basically, I understand the concept of scraping, but I don't know the ins and outs. Don't know what is the best way to accomplish it. And I'd prefer not to have a site finding its way back to my main site, and blocking me for scraping data. (even if it's just numbers, and technically not copyright) I was thinking that there might be a way to run a scraper program from a unique URL, and have the data compiled into a database, which could be used to refresh the data on my page via cron job. Is that thinking reasonable?

You guys are the experts. I am here to learn. I await your replies...

 

[Panther28]

you could start by getting someone to build a bot for you to scrape the sites, and then input the data into your own site. That way you could test the idea for the site much quicker and if its looks good, you can spend more time coding it up properly to run auto pilot.

 

[Mr Salt]

So let me go out on a limb and ask, are there any bots out there that are pre-made, and customizable? (preferably free) This is an experimental site, and I'm not ready to go all in and invest tons of $ just yet...

Thank you for your reply.

 

[Mr Salt]

Still looking for a good solution for this. There must be a way to do what I'm asking without re-inventing the wheel?

Thank you.

 

[Mr Salt]

To the person who sent me a PM... I cannot PM before I reach 15 posts. Feel free to send me another PM with an alternate means of contacting you.

Thanks!

 

[bartosimpsonio]

There's a section here called hire a freelancer. Unless you got the programming foo you'll need someome to develop a custom solution for that.

 

[Wilson Grant Fisk]

Welcome to BHW, Mr Salt

Scraping is a lot of fun and there are several ways to do it.

Paste the following into google and you should find threads that will interest you.

site:blackhatworld.com website scraping

Alternatively, if you would like a trusted member to give you a quote on a custom built bot that has the features you require then click on the link in my signature space or search out tompots

http://www.blackhatworld.com/blackhat-seo/members/246112-tompots.html

 

[Mr Salt]

Hey, guys... Thanks a lot for your replies. I will definitely explore this angle.

"Scraping is a lot of fun", he said... That's funny.

 

[Wilson Grant Fisk]

Hey, guys... Thanks a lot for your replies. I will definitely explore this angle.

"Scraping is a lot of fun", he said... That's funny.

Seriously Mate, when you see all the data that can be yours very easily and all the evil genius ways to use that data, you will love it.

 

[Mr Salt]

Several of you have tried to send me PM, but I don't have the ability to send them back, at the moment. If you want to get ahold of me, send me your info in a PM. I don't have enough question to "fluff" my minimum post count, at the moment.

 

[Ptrick125]

If you need a bot, the best tool to use is: www.ubotstudio.com

If you were looking to hire someone, just make a thread at www.ubotstudio.com/forum in the Buy Sell Trade section.

Posted via Topify using iPhone/iPad

 

[Mr Salt]

I'm not a cheapo, I would love to hire someone. Problem is, I'm flat broke, and trying to work my way into something from, literally, nothing. Hoping to find a very simple solution. I already know where I want to get my data from, and I already know what I want to do with it. Problem is, I just don't know how to accomplish it. I don't need to scrape vast amounts of data from all over the internet. I just need a couple of sites, and dump the data into a database. (preferably mySQL) The resulting data would be updated in my main site once a week.

 

[Wilson Grant Fisk]

I don't have enough question to "fluff" my minimum post count, at the moment.

I like that.

I'm not a cheapo, I would love to hire someone. Problem is, I'm flat broke, and trying to work my way into something from, literally, nothing. Hoping to find a very simple solution. I already know where I want to get my data from, and I already know what I want to do with it. Problem is, I just don't know how to accomplish it. I don't need to scrape vast amounts of data from all over the internet. I just need a couple of sites, and dump the data into a database. (preferably mySQL) The resulting data would be updated in my main site once a week.

Anyway have a look at some of these.

http://sourceforge.net/projects/web-harvest/
http://sourceforge.net/projects/websitescraper/
http://sourceforge.net/projects/webscraper-plus/
http://www.irobotsoft.com/
http://www.iopus.com/iMacros/
http://www.gnu.org/software/wget/
http://www.httrack.com/
http://curl.haxx.se/
https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd?hl=en
http://www.reporterslab.org/browser-scrapers/
https://docs.google.com/document/d/18Q2THQvYCG2_n6nKVsZRHlaPG9iJ9NvLezOOQbEuAJs/edit?hl=en&pli=1
http://www.outwit.com/
http://www.poynter.org/how-tos/digi...websites-for-data-without-programming-skills/
https://addons.mozilla.org/en-us/firefox/addon/outwit-hub/
http://scrapy.org/
http://doc.scrapy.org/en/latest/topics/firefox.html
https://addons.mozilla.org/en-US/firefox/addon/html-regex-data-extractor/
http://simpletest.org/

 

[Mr Salt]

Wow, thanks for the reading material, W130SN. That's what I'm talking about. I will scour it thoroughly.

Great to know their are still some friendlies out there.

 

[Wilson Grant Fisk]

Wow, thanks for the reading material, W130SN. That's what I'm talking about. I will scour it thoroughly.

Great to know their are still some friendlies out there.

No problem Mate, Good Luck

 

[dgruergerugerhiye]

If you're completely non-technical, you could pirate a macro recorder and record yourself doing the task once manually. Provided the sites you're scraping stats from don't change, you could run your macro once per week with minimal additional work.

This isn't an ideal solution for a serious IMer, but it's a start, and it works.

 

[Mr Salt]

If you're completely non-technical, you could pirate a macro recorder and record yourself doing the task once manually.

That's not a bad idea. Can it be done with VB framework?

I'm still triggering the alerts when posting... There should be a [.] and a [net] in there...

 

[Mr Salt]

Nevermind that last question. The "irobot" scraper that W130SN provided a link to, seems to be the ticket. It has a macro recorder, and it works pretty well for my simple task. I will use this as my starting point, and follow where my education leads.

Thank you everyone for your input. Very much appreciated!