1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

data scraping advice please

Discussion in 'Black Hat SEO' started by darrensss, Apr 13, 2014.

  1. Apr 13, 2014
    #1
    darrensss

    darrensss Power Member

    Joined:
    Jun 10, 2010
    Messages:
    697
    Likes Received:
    79
    Guys,

    I have 2 jobs that i need to do urgently.

    1) I need to scrape data from a web site - this one im okay with but i need to decide which tool to use?
    uBot or Zenno?
    Any advice on the best one to use?

    2) This job is the tricky one? similar to to the first job but its not a web based product i need to scrape but a piece of software?
    I have no idea how i can start to scrape the results of desktop software?
    Suggestions? if its even possible?
     
  2. Apr 13, 2014
    #2
    bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    11,839
    Likes Received:
    10,686
    Occupation:
    WHEREZ MA
    Location:
    BITCOINS AT?
    Home Page:
    How does this desktop app work? You can't scrape a desktop app directly, you'd probably need to intercept its network data....
     
  3. Apr 14, 2014
    #3
    darrensss

    darrensss Power Member

    Joined:
    Jun 10, 2010
    Messages:
    697
    Likes Received:
    79
    the desktop app is a little like excel .... you perform a search and the data is returned in columns/rows.
    I need to be able to perform the search and collect the data?
     
  4. Apr 14, 2014
    #4
    jamesvick

    jamesvick Senior Member

    Joined:
    Jul 26, 2010
    Messages:
    968
    Likes Received:
    660
    Location:
    article directories
    Home Page:
    it can be done but you need some expertise with win32 commands. Basically you use sendkeys to send search and copy commands to the desktop software. The result returned is copied to clipboard and you get the data.
     
  5. Apr 14, 2014
    #5
    darrensss

    darrensss Power Member

    Joined:
    Jun 10, 2010
    Messages:
    697
    Likes Received:
    79
    What software uses win32 commands?

     
  6. Apr 14, 2014
    #6
    stugz

    stugz Junior Member

    Joined:
    Apr 14, 2013
    Messages:
    154
    Likes Received:
    34
    Windows automation software. Have a search on Google and pick one. No idea if there are any free ones available.

    If you can code I'd recommend Perl with Win32::OLE module.
     
  7. Apr 14, 2014
    #7
    lancis

    lancis Elite Member

    Joined:
    Jul 31, 2010
    Messages:
    1,676
    Likes Received:
    2,407
    Occupation:
    Entrepreneur
    Location:
    Milky Way
    Home Page:
    It depends.

    If the database of the software is stored locally, best way would be to decode it.

    If the database is stored on remote server you'll need to figure how the program makes a request to the remote server. Then perform a bunch of requests to retrieve everything you need.
     
  8. Apr 14, 2014
    #8
    bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    11,839
    Likes Received:
    10,686
    Occupation:
    WHEREZ MA
    Location:
    BITCOINS AT?
    Home Page:
    All windows programs do. How to intercept them is a different story ;)
     
  9. Apr 14, 2014
    #9
    motog

    motog BANNED BANNED

    Joined:
    Apr 12, 2014
    Messages:
    33
    Likes Received:
    33
    first analyze your desktop app how it works :)
     
  10. Apr 14, 2014
    #10
    stugz

    stugz Junior Member

    Joined:
    Apr 14, 2013
    Messages:
    154
    Likes Received:
    34
    As usual you post about things you have no idea about. There is no interception involved. It is a case of automating the GUI.
     
  11. Apr 14, 2014
    #11
    Scritty

    Scritty BANNED BANNED Premium Member

    Joined:
    May 1, 2010
    Messages:
    2,833
    Likes Received:
    4,530
    Gender:
    Male
    http://www.demondemon.com/2014/04/11/meta-data-or-big-data-extraction-from-the-web/

    This and excel scrape just about anything. Images, CSS, tables, lists.
    Software like this can be programmed to find links, click to "Next page", follow links to a depth, follow links of a certain format - scrape whatever it finds down there, decent file control.. blah blah.

    Free trial so probably worth a look (I don't sell it - I just use it)

    Wrote something like this myself in Python, then realised this was a ton better - well this and a quick export to Excel or Openoffice Calc and ten minutes fiddling and filtering.

    Scritty