data scraping advice please

D

darrensss

Power Member

Jun 10, 2010

697

80

• Apr 13, 2014

• #1

Guys,

I have 2 jobs that i need to do urgently.

1) I need to scrape data from a web site - this one im okay with but i need to decide which tool to use?
uBot or Zenno?
Any advice on the best one to use?

2) This job is the tricky one? similar to to the first job but its not a web based product i need to scrape but a piece of software?
I have no idea how i can start to scrape the results of desktop software?
Suggestions? if its even possible?

 

bartosimpsonio

Elite Member

Executive VIP

Jr. VIP

Mar 21, 2013

24,202

30,884

• Apr 13, 2014

• #2

How does this desktop app work? You can't scrape a desktop app directly, you'd probably need to intercept its network data....

 

D

darrensss

Power Member

Jun 10, 2010

697

80

• Apr 14, 2014
• Thread Starter Thread Starter

• #3

the desktop app is a little like excel .... you perform a search and the data is returned in columns/rows.
I need to be able to perform the search and collect the data?

 

jamesvick

Senior Member

Jul 26, 2010

994

696

• Apr 14, 2014

• #4

darrensss said:

the desktop app is a little like excel .... you perform a search and the data is returned in columns/rows.
I need to be able to perform the search and collect the data?

Click to expand...

it can be done but you need some expertise with win32 commands. Basically you use sendkeys to send search and copy commands to the desktop software. The result returned is copied to clipboard and you get the data.

 

D

darrensss

Power Member

Jun 10, 2010

697

80

• Apr 14, 2014
• Thread Starter Thread Starter

• #5

What software uses win32 commands?

jamesvick said:

it can be done but you need some expertise with win32 commands. Basically you use sendkeys to send search and copy commands to the desktop software. The result returned is copied to clipboard and you get the data.

Click to expand...

 

S

stugz

Junior Member

Apr 14, 2013

154

35

• Apr 14, 2014

• #6

Windows automation software. Have a search on Google and pick one. No idea if there are any free ones available.

If you can code I'd recommend Perl with Win32::OLE module.

 

lancis

Elite Member

Jul 31, 2010

1,682

2,443

• Apr 14, 2014

• #7

It depends.

If the database of the software is stored locally, best way would be to decode it.

If the database is stored on remote server you'll need to figure how the program makes a request to the remote server. Then perform a bunch of requests to retrieve everything you need.

 

bartosimpsonio

Elite Member

Executive VIP

Jr. VIP

Mar 21, 2013

24,202

30,884

• Apr 14, 2014

• #8

darrensss said:

What software uses win32 commands?

Click to expand...

All windows programs do. How to intercept them is a different story

 

M

motog

BANNED

Apr 12, 2014

33

33

• Apr 14, 2014

• #9

first analyze your desktop app how it works

 

S

stugz

Junior Member

Apr 14, 2013

154

35

• Apr 14, 2014

• #10

bartosimpsonio said:

All windows programs do. How to intercept them is a different story

Click to expand...

As usual you post about things you have no idea about. There is no interception involved. It is a case of automating the GUI.

 

Scritty

Elite Member

May 1, 2010

3,019

4,707

• Apr 14, 2014

• #11

http://www.demondemon.com/2014/04/11/meta-data-or-big-data-extraction-from-the-web/

This and excel scrape just about anything. Images, CSS, tables, lists.
Software like this can be programmed to find links, click to "Next page", follow links to a depth, follow links of a certain format - scrape whatever it finds down there, decent file control.. blah blah.

Free trial so probably worth a look (I don't sell it - I just use it)

Wrote something like this myself in Python, then realised this was a ton better - well this and a quick export to Excel or Openoffice Calc and ten minutes fiddling and filtering.

Scritty