1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Brand new to BH - looking to scrape statistics

Discussion in 'Black Hat SEO' started by Mr Salt, Sep 17, 2013.

  1. Mr Salt

    Mr Salt Newbie

    Joined:
    Sep 17, 2013
    Messages:
    15
    Likes Received:
    0
    Occupation:
    self employed
    Location:
    Paradise
    Hello, and thanks in advance for fielding my questions. This may come across as naive, and for that, I apologize. I have a good idea of what I want to do, but don't quite know how to get there. I am not a programmer, so that's the foreword.

    Basically, I have a new site design that I want to launch. As part of the site, I want to compile sports statistics. However, I don't have the resources to compile them, myself. These stats need to be updated weekly. I am not using the stats as the entirety of my site - it's just an "in addition to" my content. Bloat the site, keep readers around, etc...

    Basically, I understand the concept of scraping, but I don't know the ins and outs. Don't know what is the best way to accomplish it. And I'd prefer not to have a site finding its way back to my main site, and blocking me for scraping data. (even if it's just numbers, and technically not copyright) I was thinking that there might be a way to run a scraper program from a unique URL, and have the data compiled into a database, which could be used to refresh the data on my page via cron job. Is that thinking reasonable?

    You guys are the experts. I am here to learn. I await your replies...
     
  2. Panther28

    Panther28 Elite Member

    Joined:
    May 2, 2010
    Messages:
    2,268
    Likes Received:
    3,405
    Occupation:
    Internet.
    Location:
    Internet.
    you could start by getting someone to build a bot for you to scrape the sites, and then input the data into your own site. That way you could test the idea for the site much quicker and if its looks good, you can spend more time coding it up properly to run auto pilot.
     
  3. Mr Salt

    Mr Salt Newbie

    Joined:
    Sep 17, 2013
    Messages:
    15
    Likes Received:
    0
    Occupation:
    self employed
    Location:
    Paradise
    So let me go out on a limb and ask, are there any bots out there that are pre-made, and customizable? (preferably free) This is an experimental site, and I'm not ready to go all in and invest tons of $ just yet...

    Thank you for your reply.
     
  4. Mr Salt

    Mr Salt Newbie

    Joined:
    Sep 17, 2013
    Messages:
    15
    Likes Received:
    0
    Occupation:
    self employed
    Location:
    Paradise
    Still looking for a good solution for this. There must be a way to do what I'm asking without re-inventing the wheel?

    Thank you.
     
  5. Mr Salt

    Mr Salt Newbie

    Joined:
    Sep 17, 2013
    Messages:
    15
    Likes Received:
    0
    Occupation:
    self employed
    Location:
    Paradise
    To the person who sent me a PM... I cannot PM before I reach 15 posts. Feel free to send me another PM with an alternate means of contacting you.

    Thanks!
     
  6. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    8,835
    Likes Received:
    7,445
    Occupation:
    ZLinky2Buy SEO Services
    Location:
    ⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩
    Home Page:
    There's a section here called hire a freelancer. Unless you got the programming foo you'll need someome to develop a custom solution for that.
     
  7. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,112
    Likes Received:
    28,526
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
    Welcome to BHW, Mr Salt

    Scraping is a lot of fun and there are several ways to do it.

    Paste the following into google and you should find threads that will interest you.

    Alternatively, if you would like a trusted member to give you a quote on a custom built bot that has the features you require then click on the link in my signature space or search out tompots

    http://www.blackhatworld.com/blackhat-seo/members/246112-tompots.html
     
  8. Mr Salt

    Mr Salt Newbie

    Joined:
    Sep 17, 2013
    Messages:
    15
    Likes Received:
    0
    Occupation:
    self employed
    Location:
    Paradise
    Hey, guys... Thanks a lot for your replies. I will definitely explore this angle.

    "Scraping is a lot of fun", he said... That's funny.
     
  9. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,112
    Likes Received:
    28,526
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
    Seriously Mate, when you see all the data that can be yours very easily and all the evil genius ways to use that data, you will love it.
     
  10. Mr Salt

    Mr Salt Newbie

    Joined:
    Sep 17, 2013
    Messages:
    15
    Likes Received:
    0
    Occupation:
    self employed
    Location:
    Paradise
    Several of you have tried to send me PM, but I don't have the ability to send them back, at the moment. If you want to get ahold of me, send me your info in a PM. I don't have enough question to "fluff" my minimum post count, at the moment.
     
  11. Ptrick125

    Ptrick125 Regular Member

    Joined:
    Mar 4, 2013
    Messages:
    428
    Likes Received:
    113
    Occupation:
    Going To School
    Location:
    Near Austin, Texas
    Home Page:
  12. Mr Salt

    Mr Salt Newbie

    Joined:
    Sep 17, 2013
    Messages:
    15
    Likes Received:
    0
    Occupation:
    self employed
    Location:
    Paradise
    I'm not a cheapo, I would love to hire someone. Problem is, I'm flat broke, and trying to work my way into something from, literally, nothing. Hoping to find a very simple solution. I already know where I want to get my data from, and I already know what I want to do with it. Problem is, I just don't know how to accomplish it. I don't need to scrape vast amounts of data from all over the internet. I just need a couple of sites, and dump the data into a database. (preferably mySQL) The resulting data would be updated in my main site once a week.
     
  13. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,112
    Likes Received:
    28,526
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
    I like that.


    Anyway have a look at some of these.

    http://sourceforge.net/projects/web-harvest/
    http://sourceforge.net/projects/websitescraper/
    http://sourceforge.net/projects/webscraper-plus/
    http://www.irobotsoft.com/
    http://www.iopus.com/iMacros/
    http://www.gnu.org/software/wget/
    http://www.httrack.com/
    http://curl.haxx.se/
    https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd?hl=en
    http://www.reporterslab.org/browser-scrapers/
    https://docs.google.com/document/d/18Q2THQvYCG2_n6nKVsZRHlaPG9iJ9NvLezOOQbEuAJs/edit?hl=en&pli=1
    http://www.outwit.com/
    http://www.poynter.org/how-tos/digi...websites-for-data-without-programming-skills/
    https://addons.mozilla.org/en-us/firefox/addon/outwit-hub/
    http://scrapy.org/
    http://doc.scrapy.org/en/latest/topics/firefox.html
    https://addons.mozilla.org/en-US/firefox/addon/html-regex-data-extractor/
    http://simpletest.org/
     
    • Thanks Thanks x 1
  14. Mr Salt

    Mr Salt Newbie

    Joined:
    Sep 17, 2013
    Messages:
    15
    Likes Received:
    0
    Occupation:
    self employed
    Location:
    Paradise
    Wow, thanks for the reading material, W130SN. That's what I'm talking about. I will scour it thoroughly.

    Great to know their are still some friendlies out there.
     
  15. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,112
    Likes Received:
    28,526
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
    No problem Mate, Good Luck
     
    • Thanks Thanks x 1
  16. dgruergerugerhiye

    dgruergerugerhiye BANNED BANNED Jr. VIP Premium Member

    Joined:
    Nov 4, 2010
    Messages:
    305
    Likes Received:
    450
    If you're completely non-technical, you could pirate a macro recorder and record yourself doing the task once manually. Provided the sites you're scraping stats from don't change, you could run your macro once per week with minimal additional work.

    This isn't an ideal solution for a serious IMer, but it's a start, and it works.
     
  17. Mr Salt

    Mr Salt Newbie

    Joined:
    Sep 17, 2013
    Messages:
    15
    Likes Received:
    0
    Occupation:
    self employed
    Location:
    Paradise
    That's not a bad idea. Can it be done with VB framework?

    I'm still triggering the alerts when posting... There should be a [.] and a [net] in there...
     
  18. Mr Salt

    Mr Salt Newbie

    Joined:
    Sep 17, 2013
    Messages:
    15
    Likes Received:
    0
    Occupation:
    self employed
    Location:
    Paradise
    Nevermind that last question. The "irobot" scraper that W130SN provided a link to, seems to be the ticket. It has a macro recorder, and it works pretty well for my simple task. I will use this as my starting point, and follow where my education leads.

    Thank you everyone for your input. Very much appreciated!