1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

data scraping advice please

Discussion in 'Black Hat SEO' started by darrensss, Apr 13, 2014.

  1. darrensss

    darrensss Power Member

    Joined:
    Jun 10, 2010
    Messages:
    697
    Likes Received:
    79
    Guys,

    I have 2 jobs that i need to do urgently.

    1) I need to scrape data from a web site - this one im okay with but i need to decide which tool to use?
    uBot or Zenno?
    Any advice on the best one to use?

    2) This job is the tricky one? similar to to the first job but its not a web based product i need to scrape but a piece of software?
    I have no idea how i can start to scrape the results of desktop software?
    Suggestions? if its even possible?
     
  2. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    8,834
    Likes Received:
    7,450
    Occupation:
    ZLinky2Buy SEO Services
    Location:
    ⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩
    Home Page:
    How does this desktop app work? You can't scrape a desktop app directly, you'd probably need to intercept its network data....
     
  3. darrensss

    darrensss Power Member

    Joined:
    Jun 10, 2010
    Messages:
    697
    Likes Received:
    79
    the desktop app is a little like excel .... you perform a search and the data is returned in columns/rows.
    I need to be able to perform the search and collect the data?
     
  4. jamesvick

    jamesvick Senior Member

    Joined:
    Jul 26, 2010
    Messages:
    968
    Likes Received:
    653
    Location:
    article directories
    Home Page:
    it can be done but you need some expertise with win32 commands. Basically you use sendkeys to send search and copy commands to the desktop software. The result returned is copied to clipboard and you get the data.
     
  5. darrensss

    darrensss Power Member

    Joined:
    Jun 10, 2010
    Messages:
    697
    Likes Received:
    79
    What software uses win32 commands?

     
  6. stugz

    stugz Junior Member

    Joined:
    Apr 14, 2013
    Messages:
    154
    Likes Received:
    33
    Windows automation software. Have a search on Google and pick one. No idea if there are any free ones available.

    If you can code I'd recommend Perl with Win32::OLE module.
     
  7. lancis

    lancis Elite Member

    Joined:
    Jul 31, 2010
    Messages:
    1,632
    Likes Received:
    2,384
    Occupation:
    Entrepreneur
    Location:
    Milky Way
    Home Page:
    It depends.

    If the database of the software is stored locally, best way would be to decode it.

    If the database is stored on remote server you'll need to figure how the program makes a request to the remote server. Then perform a bunch of requests to retrieve everything you need.
     
  8. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    8,834
    Likes Received:
    7,450
    Occupation:
    ZLinky2Buy SEO Services
    Location:
    ⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩
    Home Page:
    All windows programs do. How to intercept them is a different story ;)
     
  9. motog

    motog BANNED BANNED

    Joined:
    Apr 12, 2014
    Messages:
    33
    Likes Received:
    33
    first analyze your desktop app how it works :)
     
  10. stugz

    stugz Junior Member

    Joined:
    Apr 14, 2013
    Messages:
    154
    Likes Received:
    33
    As usual you post about things you have no idea about. There is no interception involved. It is a case of automating the GUI.
     
  11. Scritty

    Scritty Elite Member Premium Member

    Joined:
    May 1, 2010
    Messages:
    2,807
    Likes Received:
    4,496
    Occupation:
    Affiliate Marketer
    Location:
    UK
    Home Page:
    http://www.demondemon.com/2014/04/11/meta-data-or-big-data-extraction-from-the-web/

    This and excel scrape just about anything. Images, CSS, tables, lists.
    Software like this can be programmed to find links, click to "Next page", follow links to a depth, follow links of a certain format - scrape whatever it finds down there, decent file control.. blah blah.

    Free trial so probably worth a look (I don't sell it - I just use it)

    Wrote something like this myself in Python, then realised this was a ton better - well this and a quick export to Excel or Openoffice Calc and ten minutes fiddling and filtering.

    Scritty