Content / Site Scraping

M

mystic

Junior Member

Jun 9, 2008

166

38

• Jan 2, 2009

• #1

I need to scrape a bunch of articles off a website. Whats the best way of scraping these 1500 pages?

Sorry if its been posted before but I searched for 20 minutes and haven't found anything

 

E

eranium

Newbie

Nov 23, 2008

41

15

• Jan 2, 2009

• #2

use HTTrack, you can scrape the whole site with this.

 

M

mystic

Junior Member

Jun 9, 2008

166

38

• Jan 2, 2009
• Thread Starter Thread Starter

• #3

eranium said:

you can scrape the whole site with this.

Click to expand...

sure does

Thanks a lot.. wish it had an option just to scrape the text off the page and not the whole .html page but this will work.

Thanks

 

fatboy

Elite Member

Aug 13, 2008

1,615

3,262

• Jan 2, 2009

• #4

You could probably wrote a script to rip what you are after - are there tags around the article that are easy to pick out at all?

I have been scraping content from pages using a couple of PHP classes and a bit of imaginative scripting!

 

U

ubermann

Registered Member

Jul 11, 2008

51

45

• Jan 2, 2009

• #5

This can just scrape the article content off the website. There is a free trial. And I think on this forum there is a trial reset if you do a search for it.

 

M

mystic

Junior Member

Jun 9, 2008

166

38

• Jan 2, 2009
• Thread Starter Thread Starter

• #6

I tried that link uberman and it didnt quite work like I needed.. thanks though

 

M

mystic

Junior Member

Jun 9, 2008

166

38

• Jan 2, 2009
• Thread Starter Thread Starter

• #7

anyone have any other recommendations? I would like to find something that rips the text (article) and not the whole html file.. anyone have any other recommendations?

 

M

miker99

Registered Member

Oct 15, 2008

61

9

• Jan 2, 2009

• #8

I've used this to scrape jobs from job sites in the past. I think they have a trial.

http://www.newprosoft.com/web-content-extractor.htm

 

M

mystic

Junior Member

Jun 9, 2008

166

38

• Jan 4, 2009
• Thread Starter Thread Starter

• #9

any more suggestions before I just pay someone to rip the site for me? Every scraper ends up ripping the whole site that is over 10gb with a shitload of useless files.. i just need the 800 or so articles ;D

 

B

bhw123

Registered Member

Jul 27, 2007

56

97

• Jan 8, 2009

• #10

scraped the content for mystic, 3k+ pages
time-consuming work

 

virus_1720

Elite Member

May 9, 2008

1,752

1,289

• Jan 8, 2009

• #11

scraping can be a tedious task but make sure you don't give up.