How to grab text from a website with cURL?

rnc505

Regular Member

Oct 28, 2008

229

112

• Feb 3, 2009

• #1

How do I use cURL to harvest the text of a website, say the blog entry titles of a Blogspot blog....

I know the text (in the source) before a title is:

Code:

<h3 class="post-title">

and after the title it is:

Code:

</h3>

How would I go about doing this? Thanks a lot!

 

195471

Regular Member

Oct 11, 2008

415

264

• Feb 3, 2009

• #2

These tutorials that may help you:

Code:

http://stackoverflow.com/questions/26947/how-to-implement-a-web-scraper-in-php
http://www.oooff.com/php-scripts/basic-curl-scraping-php/basic-scraping-with-curl.php

 

genie1

Junior Member

May 30, 2008

151

52

• Feb 3, 2009

• #3

You can always use wget then grep, or if wget is blocked, lynx --source works too. I know curl can do more, and I molest it daily, but sometimes I prefer simpler too

 

tattoo

Regular Member

May 10, 2008

405

37

• Feb 3, 2009

• #4

you can either turn it into well-formed xml and then traverse the tree or use a regex. in either case, it's beyond the scope of this forum. you will have better luck on a PHP or other programming forum.

 

fatboy

Elite Member

Aug 13, 2008

1,615

3,262

• Feb 6, 2009

• #5

Using Perl and Mechanize would do the trick as well
Been using Mechanize a lot lately and it rocks!

 

8

80degreez

Newbie

May 23, 2008

14

1

• Feb 6, 2009

• #6

I'm a heavy Mechanize user as well and highly recommend it if the page doesn't have JavaScript

 

M

mpruben

Registered Member

Jan 10, 2008

51

19

• Feb 6, 2009

• #7

Unless I'm missing something here all you need is 'substr' between the position of the start and end of the title, no?

 

sikx

Registered Member

Jan 4, 2009

65

169

• Feb 6, 2009

• #8

Here's the proper and easy way of doing it

Code:

http://nytemarez.com/scraping-with-php-and-dom/

 

D

demoniox

Registered Member

Mar 5, 2007

98

84

• Feb 23, 2009

• #9

use the php preg_match fuction

 

C

confined

Regular Member

Jan 4, 2009

216

91

• Feb 23, 2009

• #10

PHP:

<?php
//get the page with curl; look it up. php.net/curl_setopt

preg_match_all('@<h3 class="post-title">(.*)<\/h3>@is',$page,$matches);

$titles = $matches[1];

?>

untested. may need more tweaking