Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed

sean815 · #1 (**permalink**) 01-06-2010, 05:01 PM

I noticed there are many people who are clueless about Yahoo Pipes or can't figure it out. First off, let me explain what Pipes is. "Pipes is a powerful composition tool to aggregate, manipulate, and mash-up content from around the web." Some people will use Pipes to takes several RSS feeds and combine them into 1 feed for use with their website. Others might use Pipes to take a RSS feed that only displays a summary and convert the feed to display the entire content of the article or news story. Or you can use both techniques to aggregate or combine multiple summary feeds into a "content super feed" (I'm trademarking that LOL). For this tutorial, I will explain the second technique.

Step1
Sign up if you haven't already: http://pipes.yahoo.com

Step 2
Click the "Create a pipe" button on the main page.

Step 3
On the left menu, drag the "Fetch Feed" module into the grid. At this point you need to add your RSS feed that you want to convert into the "URL" text-box of this module. Use the XML version of your feed. For this example, I will use a health feed from the Washington Post:

Code:

feed://feeds.washingtonpost.com/wp-dyn/rss/health/index_xml

The purpose of this module is to tell Pipes what feed we are going to be working with. Simple!

Step 4
Drag the "Loop" module into the grid. The "Loop" module is located in the Operators submenu on the left menu. Click the little arrow to the left of "Operators" to display the submenu. After you have done this, you will notice there is another grid located inside of this "Loop" module. We will drag another module here in the next step.

The purpose of this module is to loop through each RSS item from the feed we specified in the previous module. For example, our feed has 100 stories in it. This module is going to loop through each story, 1 at a time, and do what we tell it to do.

Step 5
Drag the "Fetch Page" module into the "Loop" grid from the previous step. You should see a red box outlining the grid of the "Loop" module when you are hovering correctly. Now select the first dropdown next to "URL" and select item.link or type that in exactly.

The purpose of this module will basically look at the source page with in the RSS feed and strip out the full story. Since we threw this module into a "Loop" module, this will be looped for each RSS item(story) and grab the full story from the source page.

Step 6
Drag the "Regex" module onto the main grid. This module is located in the "Operators" submenu.

The purpose of this module is to manipulate the story, link, or even title. This module is only optional. Some examples I've used are to strip all links from a story. Or to remove the season and episode number from the title on a Hulu feed. Or to change every instance of the word Blackhat and make it output the word Whitehat instead. There are many regex examples out there on the web.

Step 7
Connect these all up. Currently these modules are all separate and there is no data flow from each module to the next. Data comes in the top of the module, filters through the module, then exits the module on the bottom. So click and hold the little circle at the bottom of the first module("Fetch Feed") and drag it to the top circle of the "Loop" module and release the click. You should see a connection or Pipe between the two modules. Now connect the bottom of the "Loop" module to the top of the "Regex" module. Finally, connect the bottom of the "Regex" module to the top of the "Pipe Output"

Now that the design is setup, you should test your connections. If you click on the "Pipe Output" module on the grid, it should turn orange. And at the bottom of the webpage, it should be generating the feed. After completion, you should see a list of your RSS stories in the bottom pane. If not, or there is an error, click the Refresh button in the bottom panes a few times. If you do not see any items then you either forgot to put the URL to your feed from Step 3 or you messed up your connections in Step 7. Try again or try another feed to test. Once you see your items in the bottom pane, you may continue.

Step 8
We need to find markers or characteristics on the source pages to pull out the full story. Open up your original RSS feed in a new browser or tab. Click on the first story so we are now on the source website reading the original story. Now on your browser, you need to view the source of the webpage. We need to find the story in this source. We also need to find a unique marker just before and just after the story. Here's a snippet of an article from the feed:

Code:

<!-- End New Comments Box: Common -->
 
<div class="sidebarhack"><b></b></div>
<div class="sidebar">
<div class="seo-header"><div style="float:left;padding-left:7px;">Who's Blogging</div><div style="float:right;padding-right:5px;"><a href="http://www.sphere.com/" style="padding:0;"><img src="http://media3.washingtonpost.com/wp-srv/images/logo_sphere_powered101x13.gif" border="0" width="101" height="13"/></a></div><div style="clear:both;"></div></div>
<div class="sidebarcontent">
&raquo; <a class="iconsphere" title="Related Blogs & Articles" onclick="return Sphere.Widget.search();" href="http://www.sphere.com/search?q=sphereit:http://www.washingtonpost.com/wp-dyn/content/article/2010/01/04/AR2010010402752.html" rel="nofollow">Links to this article</a>
</div>
</div>
</div>
<div id="ad_links_inner" style="display:none"><script type="text/javascript" src="http://media.washingtonpost.com/wp-srv/ad/quigo/article_inner.js"></script></div>

</td></tr></table>
<FONT SIZE="2">
<div id="byline">By <a href="http://projects.washingtonpost.com/staff/articles/rachel+saslow/" title="Send an e-mail to Rachel Saslow">Rachel Saslow</a></div>
Washington Post Staff Writer
<br/>
Tuesday, January 5, 2010
</FONT><P>
</div>
<div id="article_body" style="padding-left:10px;">
<span id="aptureStartContent"></span>
<p>
Scientists may have created a vaccine against cocaine addiction: a series of shots that changes the body's chemistry so that the drug can't enter the brain and provide a high.
</p>
<div id="body_after_content_column">
<p>
The vaccine, called TA-CD, shows promise but could also be dangerous; some of the addicts participating in a study of the vaccine started doing massive amounts of cocaine in hopes of overcoming its effects, according to Thomas R. Kosten, the lead researcher on the study, which was published in the Archives of General Psychiatry in October.
</p>
<p>
&quot;After the vaccine, doing cocaine was a very disappointing experience for them,&quot; said Kosten, a professor of psychiatry and neuroscience at Baylor College of Medicine in Houston.
</p>
<p>
Nobody overdosed, but some of them had 10 times more cocaine coursing through their systems than researchers had encountered before, according to Kosten. He said some of the addicts reported to researchers that they had gone broke buying cocaine from multiple drug dealers, hoping to find a variety that would get them high.
</p>
<p>
Of the 115 addicts in the study, 58 were given the vaccine, administered in a series of five shots over 12 weeks, while 57 received placebo injections. Six people dropped out before the end of the study. The researchers recruited the participants from a methadone-treatment program in West Haven, Conn., which made it possible to track them for the full 24 weeks of the study. The patients were addicted to cocaine and heroin; TA-CD is designed to work only on cocaine, including the crack form of the drug.
</p>
<p>
Like disease vaccines, TA-CD stimulates a person's immune system to produce antibodies. Of those who received all five vaccine injections, 38 percent reached antibody levels that were high enough to dull the effects of the drug. The antibodies stayed active for eight to 10 weeks after the last shot.
</p>
<p>
In the high-antibodies group, 53 percent stayed off cocaine more than half the time once they had built up immunity. That compares with 23 percent of those who produced fewer antibodies. The researchers monitored cocaine use through regular urinalysis.
</p>
<p>
&quot;In this study, immunization did not achieve complete abstinence from cocaine use,&quot; Kosten said. &quot;Previous research has shown, however, that a reduction in use is associated with a significant improvement in cocaine abusers' social functioning and thus is therapeutically meaningful.&quot;
</p>
<p>
About a quarter of those who received the vaccine did not make sufficient antibodies at all; Kosten isn't sure why.
</p>
<p>
&quot;That's the million-dollar question,&quot; said Margaret Haney, a professor of clinical neuroscience at Columbia University Medical Center, who is also researching the cocaine vaccine though she was not involved in Kosten's study.
</p>
<p>
In October, the journal Biological Psychiatry published online an article by Haney that also tested the effects of TA-CD.
</p>
<p>
Through newspaper ads, Haney had recruited 15 cocaine-dependent men to participate in her study. (Only 10 stayed to the end.)
</p>
</div>
<span id="aptureEndContent"></span>

In the beginning of the story, you should see the text "<span id="aptureStartContent"></span>". This is unique to the page meaning there is only 1 instance of it on the page source AND it is on every news story on this feed. This will be our beginning marker. Hey look at the end, Washington Post is handing us their shit on a silver platter "<span id="aptureEndContent"></span>". This will be our end marker. Now back to Yahoo Pipes.

Step 9
Within the "Fetch Page" module, you will see an area that says "Cut content from:" and this first box will be the beginning marker (<span id="aptureStartContent"></span>) and the box to the right of that will be your end marker(<span id="aptureEndContent"></span>)

Step 10
Within the "Fetch Page" module, ensure that "assign" is selected and NOT "emit". Ensure the dropdown says "first" and NOT "all". To the right of that, change the dropdown for "results to" to item.description. This is where the full content is swapped with the summary on your original RSS feed.

Step 11
Almost there

This is an optional step. If you are happy with your out put then skip this step. But you MAY want to strip links out of your story that may be inserted such as adds or reference links. You don't want these on your blog. Do you? Within the "Regex" module, add a rule by clicking the plus sign in the module. Select item.description.content in the first box. This is the item that we are editing. Paste into the "replace" box the following

Code:

<[/\]?[a]\s+[^>]*>

Don't ask how to read regex because thats a whole tutorial on its own.

Step 12
Save your Pipe by clicking the Save botton at the top right of your window. And name it.

Step 13
Lets get the NEW and IMPROVED feed url. Click "Run Pipe..." at the top of the page. Now click the "Get as RSS" link and you should see your new RSS feed. Copy that url into your favorite autoblogging plugin and you will now be ripping full news story's instead of excerpts

Feel free to ask questions and GOOD LUCK !!! If you like this then don't forget the Thanks

**StinkyBooBoo** · #2 (**permalink**) 01-06-2010, 11:59 PM

Hey sean very detailed tutorial thanks... I tried everything like you said and I keep getting this Preview failed. I'm very familiar with the basics of pipes.

I followed your instructions to a " T " but I keep getting "preview failed". WTF!?!

Very detailed post though I appreciate it. Another question, in order to maximize my Pipes knowledge what language is this? REGEX, C# ect. Thanks again

AgentOrange_MkUltra · #3 (**permalink**) 01-07-2010, 12:11 AM

Pipes are useful. BTW, if you plan to access a pipe too frequently from 1 ip (eg content scraping for autoblogs every x mins) you will be banned with error 600 or error 999 for several hours in a row. Random proxies are a must

deign23 · #4 (**permalink**) 01-07-2010, 12:25 AM

Thanks for this Tutorial. I'll be tweaking with this when i get time. I know this is a cool added way to implement an autoblogging site.

xumarix · #5 (**permalink**) 01-07-2010, 02:37 AM

Well constructed tutorials.Bits and pieces from what i learn on my own basically can be well documented in all of ur point,What im wondering is how to get full feed from any source feed which have different html template.You cant define where should u start to cut the content

sean815 · #6 (**permalink**) 01-07-2010, 03:14 AM

Quote:

Originally Posted by StinkyBooBoo

Hey sean very detailed tutorial thanks... I tried everything like you said and I keep getting this Preview failed. I'm very familiar with the basics of pipes.

I followed your instructions to a " T " but I keep getting "preview failed". WTF!?!

Very detailed post though I appreciate it. Another question, in order to maximize my Pipes knowledge what language is this? REGEX, C# ect. Thanks again

PM me the address to your pipe and I will look at it and report back.

sean815 · #7 (**permalink**) 01-07-2010, 06:46 AM

Heres an example of my yahoo pipe for this example.

Code:

http://pipes.yahoo.com/pipes/pipe.edit?_id=d69ccca47873d5caf25c952a1f36e1ba

Take a look and make sure yours looks the same.

macpaulos · #8 (**permalink**) 01-07-2010, 07:15 AM

sean815 - Awesome man. I've had many mixed results with YP. Alot of feeds are different and need to be tailored. This tutorial helps out immensely. Thanks for helping us all out.

Note to all - There is no magic pipe that works for every rss, as I said alot of feeds need to be tailored.

deign23 · #9 (**permalink**) 01-07-2010, 08:31 AM

hi sean815. I've been following your instructions. as you've mentioned that some markers or characteristics differ from every feed. In your example you used the <sp9n id="aptureStartContent"></sp9n>. Could you give post the most common characteristics here?

As far as i've tested this tutorial with other feeds, i don't see in the source this marker. And it didn't even work with divs.

It will be a good help for everyone here. Thanks and Good tainted!

macpaulos · #10 (**permalink**) 01-07-2010, 08:34 AM

I think maybe.....just guessing here to look for something common (in each story) above the H1 Tag and at the bottom of the article....each site is gonna try and stuff different things into each article (such as links, ads ect)

angrish10 · #11 (**permalink**) 01-07-2010, 09:03 AM

Query: Is there any way to extract the images from the source RSS feed.

I have followed this tutorial and got the results(excluding images).
Every site uses its own way to represent the data in background. So there is some work to be done there. Great tutorial ...BHW rocks

sean815 · #12 (**permalink**) 01-07-2010, 03:36 PM

Quote:

Originally Posted by deign23

hi sean815. I've been following your instructions. as you've mentioned that some markers or characteristics differ from every feed. In your example you used the <sp9n id="aptureStartContent"></sp9n>. Could you give post the most common characteristics here?

As far as i've tested this tutorial with other feeds, i don't see in the source this marker. And it didn't even work with divs.

It will be a good help for everyone here. Thanks and Good tainted!

macpaulos is completely correct. That Span tag was an example for this particular feed(Washington Post Health). For your feed, you need to locate a unique identifier. Id say that 60 - 70% of the feeds should have one.

Post your feed url and I can locate one for you if it exists.

wowzzren · #13 (**permalink**) 01-08-2010, 04:41 AM

Good Tutorials,I 'll follow it now,thanks.

tables · #14 (**permalink**) 01-08-2010, 06:35 PM

Thanks for the great tutorial! I'm working on twisting it a little bit, but I'm having trouble getting the "assign first results" to work.
When I do this, I can see the content I'm going for (along with a few lines I'll need to regex out) but when I view the feed, it's just the title of the article and it links back to the original posting of it. What should I do?

Thanks!

tee1 · #15 (**permalink**) 01-08-2010, 07:33 PM

nice very detailed tutorial, I have had some success in the past with pipes. If you haven't used them before I would recommend giving them a try. following this guide should get you where you to go with pipes.

cheers

wowzzren · #16 (**permalink**) 01-09-2010, 01:42 AM

May be too many people test this pipes these days,here are the result:

This Pipe ran successfully but encountered some problems:
warning Can't fetch pages that robots.txt disallow

But this is a good tutorial !thanks again.

seated · #17 (**permalink**) 01-09-2010, 05:01 AM

Thank you, this is by far the best information I've seen on the YA in BHW.

boiler · #18 (**permalink**) 01-11-2010, 01:57 AM

i'm using autoblogged, every time i want to take feed from yahoo pipe rss don't know why the article was not posted

monkeymeat · #19 (**permalink**) 01-11-2010, 07:56 AM

Killer post, and thanks a bunch for sharing this. I'm going to print out the steps until I get more used to doing it, but the amount of content this has potential for is off the charts.

deign23 · #20 (**permalink**) 01-11-2010, 08:25 AM

Quote:

Originally Posted by sean815

macpaulos is completely correct. That Span tag was an example for this particular feed(Washington Post Health). For your feed, you need to locate a unique identifier. Id say that 60 - 70% of the feeds should have one.

Post your feed url and I can locate one for you if it exists.

Thanks for the prompt reply. here's an example feed i'm trying to fetch

HTML Code:

hxxp://www.goarticles.com/feeds/Insurance/popular.rss

and here's the full text

HTML Code:

hxxp://www.goarticles.com/cgi-bin/showa.cgi?C=2418911

Hope you could help me identify the unique characteristic of this type of feed. Thanks in advance.

macpaulos · #21 (**permalink**) 01-11-2010, 08:49 AM

Not 100% sure on this, but I think these content sites run on different scripts and also have some kind of robot protection to stop people taking masses of content via automation. I think there are some scripts around BHW to bypass this but not sure if they work and how they integrate to autoblogs

sean815 · #22 (**permalink**) 01-11-2010, 02:28 PM

Quote:

Originally Posted by deign23

Thanks for the prompt reply. here's an example feed i'm trying to fetch

HTML Code:

hxxp://www.goarticles.com/feeds/Insurance/popular.rss

and here's the full text

HTML Code:

hxxp://www.goarticles.com/cgi-bin/showa.cgi?C=2418911

Hope you could help me identify the unique characteristic of this type of feed. Thanks in advance.

Cut content from: <div class="article_text cm_filter"> to: </div> and this is your feed url:

Code:

feed://www.goarticles.com/feeds/Insurance/popular.rss

sean815 · #23 (**permalink**) 01-11-2010, 02:31 PM

Quote:

Originally Posted by boiler

i'm using autoblogged, every time i want to take feed from yahoo pipe rss don't know why the article was not posted

It should work because I use Autoblogged with Pipes RSS and have little or no issues. One thing I noticed is if you change a pipe then it takes up to an hour sometimes to replicate and show the updated info in the Autoblogged feed viewer

bucksgen · #24 (**permalink**) 01-12-2010, 11:57 AM

Damn!! Your tutorial really working, thx sean.

**brittany46** · #25 (**permalink**) 01-12-2010, 12:38 PM

Thank You for an excellent tutorial! I don't see much ever mentioned here about Y pipes, but I have been using it for almost a year on my auto blogs. Good stuff to know!

bucksgen · #26 (**permalink**) 01-12-2010, 12:53 PM

hey.i have a prob..why the post different from RSS feed:
e.g:

Straight to the Source Read the Report here . 1.A LANDMARK RULING: Animals fed on GM components ARE different 2.SUMMARY: Report on animals exposed to GM ingredients in animal feed NOTE: Even in the European Union where there is mandatory labelling of GM ingredients in food and feed, there is no mandatory labeling as yet of meat, dairy products or eggs from animals fed on GM feed (GM corn or soya, for example). Even though such labeling is favoured by consumers, and is justifiable purely in terms of transparency, the food industry has tried to make out that GM feed fed to animals makes no difference either to the animal or the final product.

Feed:

* Straight to the Source

Read the Report here.

1.A LANDMARK RULING: Animals fed on GM components ARE different

2.SUMMARY: Report on animals exposed to GM ingredients in animal feed

NOTE: Even in the European Union where there is mandatory labelling of GM ingredients in food and feed, there is no mandatory labeling as yet of meat, dairy products or eggs from animals fed on GM feed (GM corn or soya, for example).

Even though such labeling is favoured by consumers, and is justifiable purely in terms of transparency, the food industry has tried to make out that GM feed fed to animals makes no difference either to the animal or the final product.

sean815 · #27 (**permalink**) 01-12-2010, 03:32 PM

Bucksgen, you gotta give me more to go on

What is the feed url?

bucksgen · #28 (**permalink**) 01-12-2010, 10:23 PM

Quote:

Originally Posted by sean815

Bucksgen, you gotta give me more to go on

What is the feed url?

here:
http://pipes.yahoo.com/pipes/pipe.run?_id=4250fc6e6d0cb1847d93688f74807c9d&_ren der=rss

using autoblogged 2.5.74

jolt3500 · #29 (**permalink**) 01-12-2010, 11:31 PM

Wow awesome post! For whatever reason I just am confused as hell when It came to yahoo pipes. Great thread!

artsurd8 · #30 (**permalink**) 01-13-2010, 03:15 AM

Hi Sean, thanks for great this tutorial!

I'd like to know how to use this method with autoblog via blogger, I mean using a direct email to post in blogger..

So the result of the yahoo pipes send to an email that directly will post it to blog (blogger)

bucksgen · #31 (**permalink**) 01-13-2010, 06:06 AM

^yeah i want to know too.

deign23 · #32 (**permalink**) 01-13-2010, 11:00 AM

Quote:

Originally Posted by sean815

Cut content from: <div class="article_text cm_filter"> to: </div> and this is your feed url:

Code:

feed://www.goarticles.com/feeds/Insurance/popular.rss

thanks men. i'll give it a try on the next day. i'm a bit dizzy right now.

Good tainted!

**rabenus** · #33 (**permalink**) 01-13-2010, 11:41 AM

Can you add another function wherein you can translate the feed three times? Like an English feed will be translated Japanese --> French --> English.. Is this possible?

boiler · #34 (**permalink**) 01-13-2010, 12:10 PM

Quote:

Originally Posted by bucksgen

here:
http://pipes.yahoo.com/pipes/pipe.run?_id=4250fc6e6d0cb1847d93688f74807c9d&_ren der=rss

using autoblogged 2.5.74

you can make it work with autoblogged ?

it posting to your site ?

mine only posting the meta post and not the content

**Abstroose** · #35 (**permalink**) 01-19-2010, 04:57 PM

Thanks for sharing the tutorial. It's a shame you have to define the markers for the content though, since my RSS feeds are scraped from hundreds of different sites.

sean815 · #36 (**permalink**) 01-19-2010, 10:38 PM

Quote:

Originally Posted by boiler

you can make it work with autoblogged ?

it posting to your site ?

mine only posting the meta post and not the content

Your pipe looks fine but make sure your using the RSS URL for your autoblog plugin. Make sure this is the url in Autoblogged:

Code:

feed://pipes.yahoo.com/pipes/pipe.run?_id=4250fc6e6d0cb1847d93688f74807c9d&_render=rss

sean815 · #37 (**permalink**) 01-19-2010, 10:40 PM

Quote:

Originally Posted by boiler

you can make it work with autoblogged ?

it posting to your site ?

mine only posting the meta post and not the content

I use Pipes feeds in 5 of my blogs that use the Autoblogged plugin. And some sites use multiple feeds and no problem.

angelic_devil · #38 (**permalink**) 01-21-2010, 12:41 AM

i cant seem to find the spanid thing in the ezinearticle plz help

here is the feed url

http://ezinearticles.com/rss/Health-...tness-Acne.xml

plz help me locate hte unique identifier

angelic_devil · #39 (**permalink**) 01-21-2010, 01:02 AM

ok i got it to work but somehow it works only without regex value...it says preview failed when i enter the regex value and click on pipe output
i entered the following in th regex

<[/\]?[a]\s+[^>]*> in regex...plz help

sean815 · #40 (**permalink**) 01-22-2010, 04:38 PM

Here is a link to an AP News feed I just created. This Pipe will strip any script tags, a href links, and div tags. So the output will be pure text with no ads. If you are having trouble with your own feeds, use this as an example.

Code:

http://pipes.yahoo.com/pipes/pipe.info?_id=9080e5040c92ecd1589e881fe77770eb

gurumyk · #41 (**permalink**) 01-24-2010, 12:44 AM

totally awesome tutorial sean! i finally made my own full rss feed.
do you have any suggestions how i can find rss feed specific to my niche?
thanks again man!

blackhaze · #42 (**permalink**) 01-24-2010, 12:49 AM

>>
We need to find markers or characteristics on the source pages to pull out the full story.
>>

thats the problem and this is why such a full-text scraper can never be universal, only for the sources where you know those footprints in the HTML and how to extract the text, because you have to hardcode that stuff.

gurumyk · #43 (**permalink**) 01-24-2010, 03:35 AM

hey sean,
just a quick question, what option do i have incase the feed doesnt have an xml extension?
something like this.

Code:

http://www.guardian.co.uk/football/fifa/rss

what do i use for step 3?

Quote:

Step 3
On the left menu, drag the "Fetch Feed" module into the grid. At this point you need to add your RSS feed that you want to convert into the "URL" text-box of this module. Use the XML version of your feed. For this example, I will use a health feed from the Washington Post:
Code:
feed://feeds.washingtonpost.com/wp-dyn/rss/health/index_xml
The purpose of this module is to tell Pipes what feed we are going to be working with. Simple!

tbkdkc · #44 (**permalink**) 01-24-2010, 04:57 PM

I seriously cannot get the fetch page module to go into the loop operator, am i that dumb?

brutang · #45 (**permalink**) 01-28-2010, 08:01 AM

Quote:

Originally Posted by tbkdkc

I seriously cannot get the fetch page module to go into the loop operator, am i that dumb?

drag out the loop module to the grid

then drag the fetch page module directly from the sidebar into the loop module- you can't stick the fetch page module into the loop module if it's already touched the grid... it's gotta go straight in

one question though, I can't seem to get it to strip the links; the regex <[/\]?[a]\s+[^>]*> works fine against a regex tester, but it will not take out the tags that make it a link. the closest i've gotten is
<a.*?>(.*?)</a>
but that strips the anchor text as well. Any tips? Is this perhaps on yahoo's end? Thanks!