[HELP] Finding content to
- Thread starter woofoo
- Start date
As a result I'll get menus, footer, headers and a lot of different stuff. Also, I'll loose images and mark up for an article. That's a problem
infTee
Junior Member
- Mar 2, 2010
- 111
- 145
You could pick a few large article directories and scrape according to their sites design. Or examine how articles are contained within popular themes for WP article directories. Then scrape google using your keywords plus footprints from the themes, that way you have a better chance of cleaner content.
sockpuppet
Junior Member
- Nov 7, 2011
- 183
- 175
look for a div element with "content" in it's class attribute, from there extract all child p tags (tag = paragraph). if you don't find such a div just look for p tags with lots of text.
how to do it? the best way is to use an html parser. if you are using python, beautiful soup is the way to go. with java i would go with htmlcleaner and xpath. and for nodejs exists a complete dom implementation, you can use the jquery CSS selector engine sizzle on it. if you use another language google...
how to do it? the best way is to use an html parser. if you are using python, beautiful soup is the way to go. with java i would go with htmlcleaner and xpath. and for nodejs exists a complete dom implementation, you can use the jquery CSS selector engine sizzle on it. if you use another language google...
weedsmoker
Junior Member
- May 2, 2011
- 188
- 75
You must customize extraction process for each site you scrape. I use 2 methods, first is xpath like previous poster wrote.
title = xpathFind(html,'//h1[@id="title]')
article = xpathFind(html,'//div[@id="article"]')
Second method is finding content by string:
title = stringFind(html,'<h1 id="title">','</h1>')
article = stringFind(html,'<div id="article">','</div>')
Use xpath, when it fails then find content by string.
johndea
Regular Member
- Jun 23, 2011
- 308
- 35
Use this app: http://boilerpipe-web.appspot.com/
Or download the code: http://code.google.com/p/boilerpipe/
It strips boilerplate text and gives you just the content.