News website scraper

M

Montella

Newbie
Joined
Jun 26, 2012
Messages
21
Reaction score
3
Does anyone know bots/services that can scrape full articles from news websites such as CNN, Huffington Post etc

Not snippets which redirect to the original article (webhose.io), but where you can imput a website & topic you'd like and it scrapes it fully.
 
Advertise on BHW
try this one..httrack website copier ,..hopefully works yr need
 
wont work classed as duplicate content website get ban.

rss only way .

unless you really want it but will get website ban

copy and paste quicker
 
redarrow said:
wont work classed as duplicate content website get ban.

rss only way .

unless you really want it but will get website ban

copy and paste quicker
Click to expand...
Who will "ban" the website?
And as to it being classified as duplicate content, so?

OP: what cms are you using? Lots of good RSS feed autopost plugins for WordPress.
 
patriotnews said:
Who will "ban" the website?
And as to it being classified as duplicate content, so?

OP: what cms are you using? Lots of good RSS feed autopost plugins for WordPress.
Click to expand...

I'm on Wordpress. Any particular plugins you would recommend?
 
Montella said:
Does anyone know bots/services that can scrape full articles from news websites such as CNN, Huffington Post etc

Not snippets which redirect to the original article (webhose.io), but where you can imput a website & topic you'd like and it scrapes it fully.
Click to expand...


Can code this for you if you want.


Cheers
 
Montella said:
I'm on Wordpress. Any particular plugins you would recommend?
Click to expand...
WPeMatico free works good enough to get you started. You can schedule many channels, many schedules.
With just the free version, I was running 47 different RSS news feeds, over 102 different posting schedules.
 
patriotnews said:
WPeMatico free works good enough to get you started. You can schedule many channels, many schedules.
With just the free version, I was running 47 different RSS news feeds, over 102 different posting schedules.
Click to expand...
Do you pull full feeds from the news sites and how do you get rid of duplicate post.
 
bobojonathan said:
Do you pull full feeds from the news sites and how do you get rid of duplicate post.
Click to expand...
I pull full feeds from news sides and cut out the ones that block full posts.
I set it up so it only pulls the most 1-2 most recent posts, and there is a check box on WPeMatico that allows you to not pull duplicate content from the sites, so I avoid pulling the same articles.
 
you can do that with argo content's article scraper, you can also filter out urls and unwanted garbage with it.
 
I can code this if you'd like
pm here or on skype @ yellowcat1771
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top