News website scraper

[Montella]

Does anyone know bots/services that can scrape full articles from news websites such as CNN, Huffington Post etc

Not snippets which redirect to the original article (webhose.io), but where you can imput a website & topic you'd like and it scrapes it fully.

 

[andy1]

try this one..httrack website copier ,..hopefully works yr need

 

[Deleted member 657006]

wont work classed as duplicate content website get ban.

rss only way .

unless you really want it but will get website ban

copy and paste quicker

 

[patriotnews]

wont work classed as duplicate content website get ban.

rss only way .

unless you really want it but will get website ban

copy and paste quicker

Who will "ban" the website?
And as to it being classified as duplicate content, so?

OP: what cms are you using? Lots of good RSS feed autopost plugins for WordPress.

 

[Montella]

Who will "ban" the website?
And as to it being classified as duplicate content, so?

OP: what cms are you using? Lots of good RSS feed autopost plugins for WordPress.

I'm on Wordpress. Any particular plugins you would recommend?

 

[greatops]

Does anyone know bots/services that can scrape full articles from news websites such as CNN, Huffington Post etc

Not snippets which redirect to the original article (webhose.io), but where you can imput a website & topic you'd like and it scrapes it fully.

Can code this for you if you want.


Cheers

 

[patriotnews]

I'm on Wordpress. Any particular plugins you would recommend?

WPeMatico free works good enough to get you started. You can schedule many channels, many schedules.
With just the free version, I was running 47 different RSS news feeds, over 102 different posting schedules.

 

[bobojonathan]

WPeMatico free works good enough to get you started. You can schedule many channels, many schedules.
With just the free version, I was running 47 different RSS news feeds, over 102 different posting schedules.

Do you pull full feeds from the news sites and how do you get rid of duplicate post.

 

[patriotnews]

Do you pull full feeds from the news sites and how do you get rid of duplicate post.

I pull full feeds from news sides and cut out the ones that block full posts.
I set it up so it only pulls the most 1-2 most recent posts, and there is a check box on WPeMatico that allows you to not pull duplicate content from the sites, so I avoid pulling the same articles.

 

[itz_styx]

you can do that with argo content's article scraper, you can also filter out urls and unwanted garbage with it.

 

[wangbu]

you can do that with argo content's article scraper, you can also filter out urls and unwanted garbage with it.

Thank you, I'd check it as well.

 

[yellowcat]

I can code this if you'd like
pm here or on skype @ yellowcat1771