tutorial - how to extract CL emails with Y4h00 p1pes

B

blazed

Junior Member
Joined
Aug 15, 2008
Messages
178
Reaction score
122
I just shared this on another forum I post on, because I was getting lots of questions about it and I decided I can't forget about BHW so I'm posting this for you guys as well. Hope someone finds it useful!

Sample Output:
80854583ic3.jpg


Source Code:
67i7ep.jpg


Explanation:
1) Fetch the RSS feed of the section you want to scrape.
2) Follow each link in the feed and extract the mailto: tag
3) Copy the tag as the item.title & item.description so it's readable in the pipe output
4) Output the pipe.

Ideas for improvement & better usage:
-decode the actual encoded link to make the data more workable
-output the pipe as a different format other than rss to make the data more workable (html, csv, json, xml, etc.)
-add support for multiple inputs
-add more regex-fu to support extracting titles & categories to their own fields
-subscribe to the feed with a feed reader and automatically build your list
 
Advertise on BHW
Hi Blazed

This looks great. Is there any way I can learn more about Yahoo Pipes?

So you suggest any book on this? I have few ideas. one like yours but taking an item & price from craigslist then getting the price from eBay for the same or similar item.

Any help would be appreciated very much.

Thanks
 
pipes.yahoo.com has everything you need to get started. I think they even have a video tutorial as an intro. Just dive right in and experiment.

Also there is a really good tutorial about scraping with yahoo pipes that you can find here:

http://www.daybarr.com/blog/2007/12...g-the-fetch-page-module-to-make-a-web-scraper

There is also a great service called dapper.net which lets you scrape things in a point and click way that is even easier, but because it's so easy you have less control and cant write your own regex's for example. However, you can always create a feed with dapper and use it as an input to yahoo pipes so that you can get up close and personal with the data.


blackjack said:
Hi Blazed

This looks great. Is there any way I can learn more about Yahoo Pipes?

So you suggest any book on this? I have few ideas. one like yours but taking an item & price from craigslist then getting the price from eBay for the same or similar item.

Any help would be appreciated very much.

Thanks
Click to expand...
 
Wow this is a hell of a share...
thanks mate... this will save alot of time...
any suggestions on how would i defeat CL's relay system? I keep getting (You have sent many email, we will stop relaying) email...
I used like 5 accounts today, any better suggestions?

thanks :)
 
check this out.. my upcoming tool: http://screencast.com/t/zJjJPXdx
 
hey blazed
could u tell me how i can extract full articles from ezinearticles?? i've tried a lot but cant seem to do things right :p
 
I will look into this and update this thread if I get anywhere :)


nikhil88 said:
hey blazed
could u tell me how i can extract full articles from ezinearticles?? i've tried a lot but cant seem to do things right :p
Click to expand...
 
thanks blazed, but i just managed to do it....cheers
 
Wow this seems like a great tool. I would love to know if it is possible or anyone has used this to scrape e-mails off of facebook profiles or from forums. If anyone knows I would love to hear the answer!
 
How do you get the code to make a yahoo pipe a rss feed. I can create the feed and get to the point of making it ready, but don't know how to go about getting it to be an rss feed? Maybe I am missing something? Thanks
 
ashenjr said:
How do you get the code to make a yahoo pipe a rss feed. I can create the feed and get to the point of making it ready, but don't know how to go about getting it to be an rss feed? Maybe I am missing something? Thanks
Click to expand...

after you save the feed, you can go to "my pipes", then chose the one you just saved and click on "run pipe" - there will be an rss icon that you can click on to get the RSS url.
 
bourder said:
Wow this seems like a great tool. I would love to know if it is possible or anyone has used this to scrape e-mails off of facebook profiles or from forums. If anyone knows I would love to hear the answer!
Click to expand...

You can definitely use y*hoo p*pes for this... I havent tried to scrape any sites yet that require authentication, but I can't imagine it to be much harder. You can always go to dapper.net that easily handles authentication and then just fetch the output of that dap with a pipe and do what you have to do.
 
Thank you very much for the tutorial. It is the first one I have tried to get going and actually got it to work. Only if i knew a way to decode the email address from the results.

Thanks again for the great tutorial...
 
So why wouldn't you use the On demand craigslist scrapper? It automates everything.. The program will scrape craigslist emails from whatever city/category you want. Unlimited uses, just a matter of how you figure out how to beat craigslist system on their crappy email addresses.

(link removed by admin - software belongs to another member)
 
spikeanut said:
So why wouldn't you use the On demand craigslist scrapper? It automates everything.. The program will scrape craigslist emails from whatever city/category you want. Unlimited uses, just a matter of how you figure out how to beat craigslist system on their crappy email addresses.
Click to expand...

I got error when I run your program.
 
Is there any easy tuturial for a noobs about this? thanks in advance.
 
scratbandit said:
Thank you very much for the tutorial. It is the first one I have tried to get going and actually got it to work. Only if i knew a way to decode the email address from the results.

Thanks again for the great tutorial...
Click to expand...

Well the best part is that you don't really need to decode them, because you are pulling the html tag that actually displays the email on the CL page. If you wanted to you can only include the item.title field and render the feed in html... then you just have a list of links that you can copy and there is no need to decode.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top