What programming language can be used to access other websites and gather information?

D

drake45

Newbie

Nov 24, 2012

21

1

• Oct 10, 2013

• #1

There are certain web apps that can access external websites and gather information.

One example is a serp checker:


User submits URL + Keywords to a website > The website uses this information to search google serp's and return the keywords positions > the website tells the user his rank position.


Can such actions be accomplished with Javascript/php? If not then what language can develop such web apps which allow a website to gather and return information from external websites?

 

Diplomat

Elite Member

Jr. VIP

Oct 25, 2011

1,654

937

• Oct 10, 2013

• #2

PHP will work, Javascript not

 

C

change

BANNED

Sep 27, 2013

54

23

• Oct 10, 2013

• #3

php cUrl will do that

 

neutralhatter

Regular Member

Jun 23, 2010

437

334

• Oct 10, 2013

• #4

Diplomat said:

PHP will work, Javascript not

Click to expand...

This is wrong. Javascript will work fine. Many languages will, but the real question is what is the fastest way...

 

SlimProxies

Supreme Member

Oct 4, 2011

1,423

1,324

• Oct 10, 2013

• #5

Python, Java, PHP, Javascript, Ajax, etc can all be used.

In other words, almost every language will do, you just have to choose if you want the data received on the server or client side.
I would suggest you start with PHP, since it's pretty straightforward (google PHP curl), and the online PHP community is huge.

 

Gophering

Junior Member

Mar 21, 2013

115

281

• Oct 11, 2013

• #6

Any Turing complete language will work here.

 

sm754

Registered Member

Mar 21, 2012

93

38

• Oct 11, 2013

• #7

Whatever languages have good HTML parsing libraries (regex doesn't count).
C/C++, C#, Java, PHP seem to all have decent options.

 

jazzc

Elite Member

Jan 27, 2009

2,964

14,717

• Oct 15, 2013

• #8

Gophering said:

Any Turing complete language will work here.

Click to expand...

Scraping in Postscript sounds fun

 

Gophering

Junior Member

Mar 21, 2013

115

281

• Oct 15, 2013

• #9

jazzc said:

Scraping in Postscript sounds fun

Click to expand...

Pfff, imagine scraping in brainfuck.

 

I

IceHD

Power Member

May 8, 2013

533

221

• Oct 15, 2013

• #10

I'm using SimplePie, it's PHP library that read xml feeds and uses cache/mysql to show the data. I also have a Pyton script that work greate, but I don't have the skills to enhace it.
I suggest using Python or C#, because of the seed.

 

apoorv

Regular Member

Aug 31, 2011

301

62

• Oct 15, 2013

• #11

Gophering said:

Pfff, imagine scraping in brainfuck.

Click to expand...

Haha!

--

To the OP, for that specific example, pretty much anything would work: Ruby or Python using Mechanize, PHP using cURL, or CasperJS if you are going to use JS (though the speeds are slow).