|
|
|
 |
|

01-06-2010, 05:01 PM
|
 |
Registered Member
|
|
Join Date: Nov 2009
Posts: 56
Thanks: 2
Thanked 127 Times in 11 Posts
Reputation: 22
|
|
Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
I noticed there are many people who are clueless about Yahoo Pipes or can't figure it out. First off, let me explain what Pipes is. "Pipes is a powerful composition tool to aggregate, manipulate, and mash-up content from around the web." Some people will use Pipes to takes several RSS feeds and combine them into 1 feed for use with their website. Others might use Pipes to take a RSS feed that only displays a summary and convert the feed to display the entire content of the article or news story. Or you can use both techniques to aggregate or combine multiple summary feeds into a "content super feed" (I'm trademarking that LOL). For this tutorial, I will explain the second technique.
Step1
Sign up if you haven't already: http://pipes.yahoo.com
Step 2
Click the "Create a pipe" button on the main page.
Step 3
On the left menu, drag the "Fetch Feed" module into the grid. At this point you need to add your RSS feed that you want to convert into the "URL" text-box of this module. Use the XML version of your feed. For this example, I will use a health feed from the Washington Post:
Code:
feed://feeds.washingtonpost.com/wp-dyn/rss/health/index_xml
The purpose of this module is to tell Pipes what feed we are going to be working with. Simple!
Step 4
Drag the "Loop" module into the grid. The "Loop" module is located in the Operators submenu on the left menu. Click the little arrow to the left of "Operators" to display the submenu. After you have done this, you will notice there is another grid located inside of this "Loop" module. We will drag another module here in the next step.
The purpose of this module is to loop through each RSS item from the feed we specified in the previous module. For example, our feed has 100 stories in it. This module is going to loop through each story, 1 at a time, and do what we tell it to do.
Step 5
Drag the "Fetch Page" module into the "Loop" grid from the previous step. You should see a red box outlining the grid of the "Loop" module when you are hovering correctly. Now select the first dropdown next to "URL" and select item.link or type that in exactly.
The purpose of this module will basically look at the source page with in the RSS feed and strip out the full story. Since we threw this module into a "Loop" module, this will be looped for each RSS item(story) and grab the full story from the source page.
Step 6
Drag the "Regex" module onto the main grid. This module is located in the "Operators" submenu.
The purpose of this module is to manipulate the story, link, or even title. This module is only optional. Some examples I've used are to strip all links from a story. Or to remove the season and episode number from the title on a Hulu feed. Or to change every instance of the word Blackhat and make it output the word Whitehat instead. There are many regex examples out there on the web.
Step 7
Connect these all up. Currently these modules are all separate and there is no data flow from each module to the next. Data comes in the top of the module, filters through the module, then exits the module on the bottom. So click and hold the little circle at the bottom of the first module("Fetch Feed") and drag it to the top circle of the "Loop" module and release the click. You should see a connection or Pipe between the two modules. Now connect the bottom of the "Loop" module to the top of the "Regex" module. Finally, connect the bottom of the "Regex" module to the top of the "Pipe Output"
Now that the design is setup, you should test your connections. If you click on the "Pipe Output" module on the grid, it should turn orange. And at the bottom of the webpage, it should be generating the feed. After completion, you should see a list of your RSS stories in the bottom pane. If not, or there is an error, click the Refresh button in the bottom panes a few times. If you do not see any items then you either forgot to put the URL to your feed from Step 3 or you messed up your connections in Step 7. Try again or try another feed to test. Once you see your items in the bottom pane, you may continue.
Step 8
We need to find markers or characteristics on the source pages to pull out the full story. Open up your original RSS feed in a new browser or tab. Click on the first story so we are now on the source website reading the original story. Now on your browser, you need to view the source of the webpage. We need to find the story in this source. We also need to find a unique marker just before and just after the story. Here's a snippet of an article from the feed:
Code:
<!-- End New Comments Box: Common -->
<div class="sidebarhack"><b></b></div>
<div class="sidebar">
<div class="seo-header"><div style="float:left;padding-left:7px;">Who's Blogging</div><div style="float:right;padding-right:5px;"><a href="http://www.sphere.com/" style="padding:0;"><img src="http://media3.washingtonpost.com/wp-srv/images/logo_sphere_powered101x13.gif" border="0" width="101" height="13"/></a></div><div style="clear:both;"></div></div>
<div class="sidebarcontent">
» <a class="iconsphere" title="Related Blogs & Articles" onclick="return Sphere.Widget.search();" href="http://www.sphere.com/search?q=sphereit:http://www.washingtonpost.com/wp-dyn/content/article/2010/01/04/AR2010010402752.html" rel="nofollow">Links to this article</a>
</div>
</div>
</div>
<div id="ad_links_inner" style="display:none"><script type="text/javascript" src="http://media.washingtonpost.com/wp-srv/ad/quigo/article_inner.js"></script></div>
</td></tr></table>
<FONT SIZE="2">
<div id="byline">By <a href="http://projects.washingtonpost.com/staff/articles/rachel+saslow/" title="Send an e-mail to Rachel Saslow">Rachel Saslow</a></div>
Washington Post Staff Writer
<br/>
Tuesday, January 5, 2010
</FONT><P>
</div>
<div id="article_body" style="padding-left:10px;">
<span id="aptureStartContent"></span>
<p>
Scientists may have created a vaccine against cocaine addiction: a series of shots that changes the body's chemistry so that the drug can't enter the brain and provide a high.
</p>
<div id="body_after_content_column">
<p>
The vaccine, called TA-CD, shows promise but could also be dangerous; some of the addicts participating in a study of the vaccine started doing massive amounts of cocaine in hopes of overcoming its effects, according to Thomas R. Kosten, the lead researcher on the study, which was published in the Archives of General Psychiatry in October.
</p>
<p>
"After the vaccine, doing cocaine was a very disappointing experience for them," said Kosten, a professor of psychiatry and neuroscience at Baylor College of Medicine in Houston.
</p>
<p>
Nobody overdosed, but some of them had 10 times more cocaine coursing through their systems than researchers had encountered before, according to Kosten. He said some of the addicts reported to researchers that they had gone broke buying cocaine from multiple drug dealers, hoping to find a variety that would get them high.
</p>
<p>
Of the 115 addicts in the study, 58 were given the vaccine, administered in a series of five shots over 12 weeks, while 57 received placebo injections. Six people dropped out before the end of the study. The researchers recruited the participants from a methadone-treatment program in West Haven, Conn., which made it possible to track them for the full 24 weeks of the study. The patients were addicted to cocaine and heroin; TA-CD is designed to work only on cocaine, including the crack form of the drug.
</p>
<p>
Like disease vaccines, TA-CD stimulates a person's immune system to produce antibodies. Of those who received all five vaccine injections, 38 percent reached antibody levels that were high enough to dull the effects of the drug. The antibodies stayed active for eight to 10 weeks after the last shot.
</p>
<p>
In the high-antibodies group, 53 percent stayed off cocaine more than half the time once they had built up immunity. That compares with 23 percent of those who produced fewer antibodies. The researchers monitored cocaine use through regular urinalysis.
</p>
<p>
"In this study, immunization did not achieve complete abstinence from cocaine use," Kosten said. "Previous research has shown, however, that a reduction in use is associated with a significant improvement in cocaine abusers' social functioning and thus is therapeutically meaningful."
</p>
<p>
About a quarter of those who received the vaccine did not make sufficient antibodies at all; Kosten isn't sure why.
</p>
<p>
"That's the million-dollar question," said Margaret Haney, a professor of clinical neuroscience at Columbia University Medical Center, who is also researching the cocaine vaccine though she was not involved in Kosten's study.
</p>
<p>
In October, the journal Biological Psychiatry published online an article by Haney that also tested the effects of TA-CD.
</p>
<p>
Through newspaper ads, Haney had recruited 15 cocaine-dependent men to participate in her study. (Only 10 stayed to the end.)
</p>
</div>
<span id="aptureEndContent"></span>
In the beginning of the story, you should see the text "<span id="aptureStartContent"></span>". This is unique to the page meaning there is only 1 instance of it on the page source AND it is on every news story on this feed. This will be our beginning marker. Hey look at the end, Washington Post is handing us their shit on a silver platter "<span id="aptureEndContent"></span>". This will be our end marker. Now back to Yahoo Pipes.
Step 9
Within the "Fetch Page" module, you will see an area that says "Cut content from:" and this first box will be the beginning marker (<span id="aptureStartContent"></span>) and the box to the right of that will be your end marker(<span id="aptureEndContent"></span>)
Step 10
Within the "Fetch Page" module, ensure that "assign" is selected and NOT "emit". Ensure the dropdown says "first" and NOT "all". To the right of that, change the dropdown for "results to" to item.description. This is where the full content is swapped with the summary on your original RSS feed.
Step 11
Almost there  This is an optional step. If you are happy with your out put then skip this step. But you MAY want to strip links out of your story that may be inserted such as adds or reference links. You don't want these on your blog. Do you? Within the "Regex" module, add a rule by clicking the plus sign in the module. Select item.description.content in the first box. This is the item that we are editing. Paste into the "replace" box the following
Don't ask how to read regex because thats a whole tutorial on its own.
Step 12
Save your Pipe by clicking the Save botton at the top right of your window. And name it.
Step 13
Lets get the NEW and IMPROVED feed url. Click "Run Pipe..." at the top of the page. Now click the "Get as RSS" link and you should see your new RSS feed. Copy that url into your favorite autoblogging plugin and you will now be ripping full news story's instead of excerpts
Feel free to ask questions and GOOD LUCK !!! If you like this then don't forget the Thanks
|
|
The Following 114 Users Say Thank You to sean815 For This Useful Post:
|
3xrs.com (09-24-2010),
4i4i78 (03-11-2010),
abhi987 (06-08-2011),
aishahriar (01-12-2010),
allfreetodo (09-24-2010),
angelas111 (10-28-2010),
angrish10 (01-07-2010),
ataga12 (10-06-2010),
Barnaby (02-15-2010),
blackmagicmaster (02-05-2010),
BlueFire333 (07-09-2010),
boogie45 (12-15-2011),
bopkins (07-06-2010),
bosshog2 (06-30-2010),
brittany46 (01-12-2010),
brutang (01-29-2010),
bucksgen (01-12-2010),
Bugalugs (01-11-2010),
bustina (02-01-2010),
carima (04-29-2010),
chaos_id (10-06-2010),
chavdar74 (02-14-2010),
cjw10 (01-15-2010),
computermonkey (06-08-2010),
dailytrip (05-13-2011),
deign23 (01-07-2010),
denydean (11-04-2010),
djmonster (05-24-2011),
dudeshane01 (09-18-2010),
Dutch Harry (05-14-2010),
egomOnia (02-14-2010),
fencersbeatngu (07-22-2010),
Flinx (01-12-2010),
Galanoid (09-23-2010),
garne041 (03-09-2011),
geeknerd (01-24-2010),
giveitlegs (10-07-2010),
gordongekko (02-20-2010),
grafxextreme (01-27-2010),
gurumyk (01-24-2010),
hanakunkun (02-17-2011),
happyhatter (01-06-2010),
hardinflash (02-09-2010),
hazard_mkd (06-30-2010),
hillux (01-13-2010),
Iluminatul (03-15-2010),
jaroslav (02-05-2010),
johnsmith21 (01-13-2010),
JPXeon (03-10-2010),
jstratford (07-24-2010),
jsurya (03-25-2010),
jtliewim (01-07-2010),
kapsi29 (06-27-2010),
kimkils (03-18-2010),
kingbrend (01-06-2010),
laserblast (01-21-2010),
le-bouzin (02-26-2010),
lineguy (03-02-2011),
luppy (09-22-2010),
macpaulos (01-07-2010),
Malimanje (06-10-2011),
marketingwise (09-26-2010),
Meddy (11-01-2011),
Mezzeric (09-22-2011),
mike333 (03-14-2010),
mmichael (05-05-2010),
mojo_hanzo (10-30-2010),
MonoPuff (01-11-2010),
moshecogan (11-30-2010),
mutatedllama (03-02-2011),
nambooooo (06-03-2010),
nhkduy90 (01-23-2011),
parlenk (08-24-2010),
pintonbd (01-30-2010),
ProCryptor (02-02-2012),
Quang Vinh (06-20-2011),
redstone.1337 (10-28-2010),
reinie (11-22-2010),
revekozu (08-02-2010),
SaveManny (08-09-2010),
Sharkattack (02-08-2010),
shezboy (03-18-2010),
shines (04-24-2010),
SpectroMX5 (05-19-2010),
spokoiton (12-26-2011),
staycoolmen2006 (10-28-2010),
steveo70 (02-16-2010),
StinkyBooBoo (01-06-2010),
suave (03-12-2010),
sukses2010 (03-06-2010),
sumax (01-13-2010),
Swipular (04-06-2010),
sycthos (02-14-2010),
tahersaid (01-30-2010),
tee1 (01-08-2010),
thachcaodotcom (08-10-2010),
Tirade (11-07-2010),
trafficone (04-18-2010),
turoc (07-12-2010),
vegemite (06-23-2010),
vokzzi (01-08-2010),
wanna_earn (09-30-2010),
webnet (01-08-2010),
WinBoot (09-11-2010),
wolvax (02-11-2010),
WuMan (03-10-2010),
xbon869x (01-30-2010),
xpleet (11-19-2010),
z3zim4 (09-27-2011),
zattara (01-07-2010),
zebrahat (01-24-2010),
ZeroMeth (09-23-2011),
zhxlay (01-11-2010),
zniber (01-17-2010)
|

01-06-2010, 11:59 PM
|
 |
Junior Member
|
|
Join Date: Apr 2009
Location: South Side
Posts: 149
Thanks: 224
Thanked 430 Times in 38 Posts
Reputation: 83
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Hey sean very detailed tutorial thanks... I tried everything like you said and I keep getting this Preview failed. I'm very familiar with the basics of pipes.
I followed your instructions to a " T " but I keep getting "preview failed". WTF!?!
Very detailed post though I appreciate it. Another question, in order to maximize my Pipes knowledge what language is this? REGEX, C# ect. Thanks again
|

01-07-2010, 12:11 AM
|
 |
Junior Member
|
|
Join Date: May 2009
Location: Many steps aheads & Behind 4 Proxies
Posts: 170
Thanks: 7
Thanked 31 Times in 24 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Pipes are useful. BTW, if you plan to access a pipe too frequently from 1 ip (eg content scraping for autoblogs every x mins) you will be banned with error 600 or error 999 for several hours in a row. Random proxies are a must
|

01-07-2010, 12:25 AM
|
 |
Regular Member
|
|
Join Date: Apr 2009
Location: Philippines
Posts: 288
Thanks: 81
Thanked 78 Times in 46 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Thanks for this Tutorial. I'll be tweaking with this when i get time. I know this is a cool added way to implement an autoblogging site.
|

01-07-2010, 02:37 AM
|
|
Newbies
|
|
Join Date: Nov 2008
Posts: 43
Thanks: 12
Thanked 3 Times in 2 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Well constructed tutorials.Bits and pieces from what i learn on my own basically can be well documented in all of ur point,What im wondering is how to get full feed from any source feed which have different html template.You cant define where should u start to cut the content
|

01-07-2010, 03:14 AM
|
 |
Registered Member
|
|
Join Date: Nov 2009
Posts: 56
Thanks: 2
Thanked 127 Times in 11 Posts
Reputation: 22
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Quote:
Originally Posted by StinkyBooBoo
Hey sean very detailed tutorial thanks... I tried everything like you said and I keep getting this Preview failed. I'm very familiar with the basics of pipes.
I followed your instructions to a " T " but I keep getting "preview failed". WTF!?!
Very detailed post though I appreciate it. Another question, in order to maximize my Pipes knowledge what language is this? REGEX, C# ect. Thanks again
|
PM me the address to your pipe and I will look at it and report back.
|

01-07-2010, 06:46 AM
|
 |
Registered Member
|
|
Join Date: Nov 2009
Posts: 56
Thanks: 2
Thanked 127 Times in 11 Posts
Reputation: 22
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Heres an example of my yahoo pipe for this example.
Code:
http://pipes.yahoo.com/pipes/pipe.edit?_id=d69ccca47873d5caf25c952a1f36e1ba
Take a look and make sure yours looks the same.
|
|
The Following 2 Users Say Thank You to sean815 For This Useful Post:
|
|

01-07-2010, 07:15 AM
|
 |
Regular Member
|
|
Join Date: Oct 2009
Posts: 295
Thanks: 129
Thanked 51 Times in 41 Posts
Reputation: 17
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
sean815 - Awesome man. I've had many mixed results with YP. Alot of feeds are different and need to be tailored. This tutorial helps out immensely. Thanks for helping us all out.
Note to all - There is no magic pipe that works for every rss, as I said alot of feeds need to be tailored.
|

01-07-2010, 08:31 AM
|
 |
Regular Member
|
|
Join Date: Apr 2009
Location: Philippines
Posts: 288
Thanks: 81
Thanked 78 Times in 46 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
hi sean815. I've been following your instructions. as you've mentioned that some markers or characteristics differ from every feed. In your example you used the <sp9n id="aptureStartContent"></sp9n>. Could you give post the most common characteristics here?
As far as i've tested this tutorial with other feeds, i don't see in the source this marker. And it didn't even work with divs.
It will be a good help for everyone here. Thanks and Good tainted!
|

01-07-2010, 08:34 AM
|
 |
Regular Member
|
|
Join Date: Oct 2009
Posts: 295
Thanks: 129
Thanked 51 Times in 41 Posts
Reputation: 17
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
I think maybe.....just guessing here to look for something common (in each story) above the H1 Tag and at the bottom of the article....each site is gonna try and stuff different things into each article (such as links, ads ect)
|

01-07-2010, 09:03 AM
|
 |
Junior Member
|
|
Join Date: Aug 2009
Location: La Isla Bonita
Posts: 135
Thanks: 33
Thanked 151 Times in 34 Posts
Reputation: 15
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Query: Is there any way to extract the images from the source RSS feed.
I have followed this tutorial and got the results(excluding images).
Every site uses its own way to represent the data in background. So there is some work to be done there. Great tutorial ...BHW rocks
|

01-07-2010, 03:36 PM
|
 |
Registered Member
|
|
Join Date: Nov 2009
Posts: 56
Thanks: 2
Thanked 127 Times in 11 Posts
Reputation: 22
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Quote:
Originally Posted by deign23
hi sean815. I've been following your instructions. as you've mentioned that some markers or characteristics differ from every feed. In your example you used the <sp9n id="aptureStartContent"></sp9n>. Could you give post the most common characteristics here?
As far as i've tested this tutorial with other feeds, i don't see in the source this marker. And it didn't even work with divs.
It will be a good help for everyone here. Thanks and Good tainted!
|
macpaulos is completely correct. That Span tag was an example for this particular feed(Washington Post Health). For your feed, you need to locate a unique identifier. Id say that 60 - 70% of the feeds should have one.
Post your feed url and I can locate one for you if it exists.
|

01-08-2010, 04:41 AM
|
 |
Newbies
|
|
Join Date: Dec 2009
Posts: 28
Thanks: 18
Thanked 0 Times in 0 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Good Tutorials,I 'll follow it now,thanks.
|

01-08-2010, 06:35 PM
|
|
Newbies
|
|
Join Date: Aug 2009
Posts: 17
Thanks: 4
Thanked 0 Times in 0 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Thanks for the great tutorial! I'm working on twisting it a little bit, but I'm having trouble getting the "assign first results" to work.
When I do this, I can see the content I'm going for (along with a few lines I'll need to regex out) but when I view the feed, it's just the title of the article and it links back to the original posting of it. What should I do?
Thanks!
|

01-08-2010, 07:33 PM
|
|
Junior Member
|
|
Join Date: Dec 2008
Posts: 110
Thanks: 94
Thanked 162 Times in 23 Posts
Reputation: 13
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
nice very detailed tutorial, I have had some success in the past with pipes. If you haven't used them before I would recommend giving them a try. following this guide should get you where you to go with pipes.
cheers
|

01-09-2010, 01:42 AM
|
 |
Newbies
|
|
Join Date: Dec 2009
Posts: 28
Thanks: 18
Thanked 0 Times in 0 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
May be too many people test this pipes these days,here are the result:
This Pipe ran successfully but encountered some problems:
warning Can't fetch pages that robots.txt disallow
But this is a good tutorial !thanks again.
|

01-09-2010, 05:01 AM
|
 |
Regular Member
|
|
Join Date: Jan 2009
Location: Sunny South Florida
Posts: 207
Thanks: 51
Thanked 135 Times in 35 Posts
Reputation: 16
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Thank you, this is by far the best information I've seen on the YA in BHW.
|

01-11-2010, 01:57 AM
|
|
Power Member
|
|
Join Date: Apr 2009
Posts: 672
Thanks: 8
Thanked 123 Times in 54 Posts
Reputation: 16
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
i'm using autoblogged, every time i want to take feed from yahoo pipe rss don't know why the article was not posted
|

01-11-2010, 07:56 AM
|
|
Newbies
|
|
Join Date: Apr 2008
Posts: 4
Thanks: 5
Thanked 0 Times in 0 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Killer post, and thanks a bunch for sharing this. I'm going to print out the steps until I get more used to doing it, but the amount of content this has potential for is off the charts.
|

01-11-2010, 08:25 AM
|
 |
Regular Member
|
|
Join Date: Apr 2009
Location: Philippines
Posts: 288
Thanks: 81
Thanked 78 Times in 46 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Quote:
Originally Posted by sean815
macpaulos is completely correct. That Span tag was an example for this particular feed(Washington Post Health). For your feed, you need to locate a unique identifier. Id say that 60 - 70% of the feeds should have one.
Post your feed url and I can locate one for you if it exists.
|
Thanks for the prompt reply. here's an example feed i'm trying to fetch
HTML Code:
hxxp://www.goarticles.com/feeds/Insurance/popular.rss
and here's the full text
HTML Code:
hxxp://www.goarticles.com/cgi-bin/showa.cgi?C=2418911
Hope you could help me identify the unique characteristic of this type of feed. Thanks in advance.
|

01-11-2010, 08:49 AM
|
 |
Regular Member
|
|
Join Date: Oct 2009
Posts: 295
Thanks: 129
Thanked 51 Times in 41 Posts
Reputation: 17
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Not 100% sure on this, but I think these content sites run on different scripts and also have some kind of robot protection to stop people taking masses of content via automation. I think there are some scripts around BHW to bypass this but not sure if they work and how they integrate to autoblogs
|

01-11-2010, 02:28 PM
|
 |
Registered Member
|
|
Join Date: Nov 2009
Posts: 56
Thanks: 2
Thanked 127 Times in 11 Posts
Reputation: 22
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Quote:
Originally Posted by deign23
Thanks for the prompt reply. here's an example feed i'm trying to fetch
HTML Code:
hxxp://www.goarticles.com/feeds/Insurance/popular.rss
and here's the full text
HTML Code:
hxxp://www.goarticles.com/cgi-bin/showa.cgi?C=2418911
Hope you could help me identify the unique characteristic of this type of feed. Thanks in advance.
|
Cut content from: <div class="article_text cm_filter"> to: </div> and this is your feed url:
Code:
feed://www.goarticles.com/feeds/Insurance/popular.rss
|
|
The Following User Says Thank You to sean815 For This Useful Post:
|
|

01-11-2010, 02:31 PM
|
 |
Registered Member
|
|
Join Date: Nov 2009
Posts: 56
Thanks: 2
Thanked 127 Times in 11 Posts
Reputation: 22
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Quote:
Originally Posted by boiler
i'm using autoblogged, every time i want to take feed from yahoo pipe rss don't know why the article was not posted
|
It should work because I use Autoblogged with Pipes RSS and have little or no issues. One thing I noticed is if you change a pipe then it takes up to an hour sometimes to replicate and show the updated info in the Autoblogged feed viewer
|

01-12-2010, 11:57 AM
|
|
Junior Member
|
|
Join Date: Dec 2009
Posts: 127
Thanks: 29
Thanked 105 Times in 15 Posts
Reputation: 13
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Damn!! Your tutorial really working, thx sean.
|

01-12-2010, 12:38 PM
|
|
Jr. VIP
|
|
Join Date: Oct 2006
Posts: 137
Thanks: 40
Thanked 18 Times in 11 Posts
Reputation: 11
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Thank You for an excellent tutorial! I don't see much ever mentioned here about Y pipes, but I have been using it for almost a year on my auto blogs. Good stuff to know!
|

01-12-2010, 12:53 PM
|
|
Junior Member
|
|
Join Date: Dec 2009
Posts: 127
Thanks: 29
Thanked 105 Times in 15 Posts
Reputation: 13
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
hey.i have a prob..why the post different from RSS feed:
e.g:
Straight to the Source Read the Report here . 1.A LANDMARK RULING: Animals fed on GM components ARE different 2.SUMMARY: Report on animals exposed to GM ingredients in animal feed NOTE: Even in the European Union where there is mandatory labelling of GM ingredients in food and feed, there is no mandatory labeling as yet of meat, dairy products or eggs from animals fed on GM feed (GM corn or soya, for example). Even though such labeling is favoured by consumers, and is justifiable purely in terms of transparency, the food industry has tried to make out that GM feed fed to animals makes no difference either to the animal or the final product.
Feed:
* Straight to the Source
Read the Report here.
1.A LANDMARK RULING: Animals fed on GM components ARE different
2.SUMMARY: Report on animals exposed to GM ingredients in animal feed
NOTE: Even in the European Union where there is mandatory labelling of GM ingredients in food and feed, there is no mandatory labeling as yet of meat, dairy products or eggs from animals fed on GM feed (GM corn or soya, for example).
Even though such labeling is favoured by consumers, and is justifiable purely in terms of transparency, the food industry has tried to make out that GM feed fed to animals makes no difference either to the animal or the final product.
|

01-12-2010, 03:32 PM
|
 |
Registered Member
|
|
Join Date: Nov 2009
Posts: 56
Thanks: 2
Thanked 127 Times in 11 Posts
Reputation: 22
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Bucksgen, you gotta give me more to go on  What is the feed url?
|

01-12-2010, 10:23 PM
|
|
Junior Member
|
|
Join Date: Dec 2009
Posts: 127
Thanks: 29
Thanked 105 Times in 15 Posts
Reputation: 13
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Quote:
Originally Posted by sean815
Bucksgen, you gotta give me more to go on  What is the feed url?
|
here:
http://pipes.yahoo.com/pipes/pipe.run?_id=4250fc6e6d0cb1847d93688f74807c9d&_ren der=rss
using autoblogged 2.5.74
Last edited by bucksgen; 01-12-2010 at 10:29 PM.
|

01-12-2010, 11:31 PM
|
 |
Junior Member
|
|
Join Date: Jun 2009
Posts: 102
Thanks: 26
Thanked 2 Times in 2 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Wow awesome post! For whatever reason I just am confused as hell when It came to yahoo pipes. Great thread!
|

01-13-2010, 03:15 AM
|
 |
Newbies
|
|
Join Date: May 2009
Location: vectorland
Posts: 28
Thanks: 4
Thanked 2 Times in 2 Posts
Reputation: 9
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Hi Sean, thanks for great this tutorial!
I'd like to know how to use this method with autoblog via blogger, I mean using a direct email to post in blogger..
So the result of the yahoo pipes send to an email that directly will post it to blog (blogger)
|

01-13-2010, 06:06 AM
|
|
Junior Member
|
|
Join Date: Dec 2009
Posts: 127
Thanks: 29
Thanked 105 Times in 15 Posts
Reputation: 13
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
^yeah i want to know too.
|

01-13-2010, 11:00 AM
|
 |
Regular Member
|
|
Join Date: Apr 2009
Location: Philippines
Posts: 288
Thanks: 81
Thanked 78 Times in 46 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Quote:
Originally Posted by sean815
Cut content from: <div class="article_text cm_filter"> to: </div> and this is your feed url:
Code:
feed://www.goarticles.com/feeds/Insurance/popular.rss
|
thanks men. i'll give it a try on the next day. i'm a bit dizzy right now.
Good tainted!
|

01-13-2010, 11:41 AM
|
 |
Jr. VIP
|
|
Join Date: Apr 2009
Posts: 122
Thanks: 25
Thanked 21 Times in 13 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Can you add another function wherein you can translate the feed three times? Like an English feed will be translated Japanese --> French --> English.. Is this possible?
|

01-13-2010, 12:10 PM
|
|
Power Member
|
|
Join Date: Apr 2009
Posts: 672
Thanks: 8
Thanked 123 Times in 54 Posts
Reputation: 16
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Quote:
Originally Posted by bucksgen
|
you can make it work with autoblogged ?
it posting to your site ?
mine only posting the meta post and not the content
|

01-19-2010, 04:57 PM
|
 |
Metaphysical
|
|
Join Date: Nov 2008
Location: Astral Plane
Posts: 1,384
Thanks: 720
Thanked 1,948 Times in 571 Posts
Reputation: 1920
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Thanks for sharing the tutorial. It's a shame you have to define the markers for the content though, since my RSS feeds are scraped from hundreds of different sites.
|

01-19-2010, 10:38 PM
|
 |
Registered Member
|
|
Join Date: Nov 2009
Posts: 56
Thanks: 2
Thanked 127 Times in 11 Posts
Reputation: 22
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Quote:
Originally Posted by boiler
you can make it work with autoblogged ?
it posting to your site ?
mine only posting the meta post and not the content 
|
Your pipe looks fine but make sure your using the RSS URL for your autoblog plugin. Make sure this is the url in Autoblogged:
Code:
feed://pipes.yahoo.com/pipes/pipe.run?_id=4250fc6e6d0cb1847d93688f74807c9d&_render=rss
|

01-19-2010, 10:40 PM
|
 |
Registered Member
|
|
Join Date: Nov 2009
Posts: 56
Thanks: 2
Thanked 127 Times in 11 Posts
Reputation: 22
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Quote:
Originally Posted by boiler
you can make it work with autoblogged ?
it posting to your site ?
mine only posting the meta post and not the content 
|
I use Pipes feeds in 5 of my blogs that use the Autoblogged plugin. And some sites use multiple feeds and no problem.
|

01-21-2010, 12:41 AM
|
|
Registered Member
|
|
Join Date: Jan 2010
Posts: 77
Thanks: 3
Thanked 0 Times in 0 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
i cant seem to find the spanid thing in the ezinearticle plz help
here is the feed url
http://ezinearticles.com/rss/Health-...tness-Acne.xml
plz help me locate hte unique identifier
|

01-21-2010, 01:02 AM
|
|
Registered Member
|
|
Join Date: Jan 2010
Posts: 77
Thanks: 3
Thanked 0 Times in 0 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
ok i got it to work but somehow it works only without regex value...it says preview failed when i enter the regex value and click on pipe output
i entered the following in th regex
<[/\]?[a]\s+[^>]*> in regex...plz help
|

01-22-2010, 04:38 PM
|
 |
Registered Member
|
|
Join Date: Nov 2009
Posts: 56
Thanks: 2
Thanked 127 Times in 11 Posts
Reputation: 22
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Here is a link to an AP News feed I just created. This Pipe will strip any script tags, a href links, and div tags. So the output will be pure text with no ads. If you are having trouble with your own feeds, use this as an example.
Code:
http://pipes.yahoo.com/pipes/pipe.info?_id=9080e5040c92ecd1589e881fe77770eb
|

01-24-2010, 12:44 AM
|
|
Newbies
|
|
Join Date: Dec 2009
Posts: 21
Thanks: 37
Thanked 0 Times in 0 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
totally awesome tutorial sean! i finally made my own full rss feed.
do you have any suggestions how i can find rss feed specific to my niche?
thanks again man!
|

01-24-2010, 12:49 AM
|
 |
Power Member
|
|
Join Date: Jan 2008
Location: in the matrix
Posts: 601
Thanks: 10
Thanked 148 Times in 78 Posts
Reputation: 12
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
>>
We need to find markers or characteristics on the source pages to pull out the full story.
>>
thats the problem and this is why such a full-text scraper can never be universal, only for the sources where you know those footprints in the HTML and how to extract the text, because you have to hardcode that stuff.
|

01-24-2010, 03:35 AM
|
|
Newbies
|
|
Join Date: Dec 2009
Posts: 21
Thanks: 37
Thanked 0 Times in 0 Posts
Reputation: 10
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
hey sean,
just a quick question, what option do i have incase the feed doesnt have an xml extension?
something like this.
Code:
http://www.guardian.co.uk/football/fifa/rss
what do i use for step 3?
Quote:
Step 3
On the left menu, drag the "Fetch Feed" module into the grid. At this point you need to add your RSS feed that you want to convert into the "URL" text-box of this module. Use the XML version of your feed. For this example, I will use a health feed from the Washington Post:
Code:
feed://feeds.washingtonpost.com/wp-dyn/rss/health/index_xml
The purpose of this module is to tell Pipes what feed we are going to be working with. Simple!
|
|

01-24-2010, 04:57 PM
|
 |
Junior Member
|
|
Join Date: Oct 2008
Location: far far away
Posts: 154
Thanks: 110
Thanked 53 Times in 31 Posts
Reputation: 12
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
I seriously cannot get the fetch page module to go into the loop operator, am i that dumb?
__________________
Make it Rain
|

01-28-2010, 08:01 AM
|
|
Junior Member
|
|
Join Date: Nov 2009
Posts: 176
Thanks: 58
Thanked 61 Times in 43 Posts
Reputation: 11
|
|
Re: Yahoo Pipe Tutorial- Convert RSS summary to RSS full feed
Quote:
Originally Posted by tbkdkc
I seriously cannot get the fetch page module to go into the loop operator, am i that dumb?
|
drag out the loop module to the grid
then drag the fetch page module directly from the sidebar into the loop module- you can't stick the fetch page module into the loop module if it's already touched the grid... it's gotta go straight in
one question though, I can't seem to get it to strip the links; the regex <[/\]?[a]\s+[^>]*> works fine against a regex tester, but it will not take out the tags that make it a link. the closest i've gotten is
<a.*?>(.*?)</a>
but that strips the anchor text as well. Any tips? Is this perhaps on yahoo's end? Thanks!
Last edited by brutang; 01-28-2010 at 09:18 AM.
|
| Thread Tools |
Search this Thread |
|
|
|
| Display Modes |
Linear Mode
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is Off
|
|
|
|