Newb ?: Tool That Will Extract All Pages/Posts Text From Blog/Website

[madman]

Hi, I am looking for a tool that will crawl a website like B00k Maarkiing Deemmon but will return all the Pages/Posts Text not just a snippet of the text. Sorry if this is such a basic question.

Thanks

 

[Rendias]

You want to scrap the whole website/blog or just getting the latest blog posts?

 

[madman]

You want to scrap the whole website/blog or just getting the latest blog posts?

Hi Rendias,

I'm interested in extracting all the website/blog pages/posts.

Thanks

 

[madman]

Any chance that one experienced member could offer a quick answer to my question? I would truly appreciate any input.

Thanks

 

[markm455]

Have you tried something like "offline explorer pro"? They have a 30 day free trial of the software if you want to give it a shot.

 

[madman]

Have you tried something like "offline explorer pro"? They have a 30 day free trial of the software if you want to give it a shot.

Thanks markm455. I did not know that there was commercial software that would do what I wanted.

Thanks again

 

[nipester]

Have you heard about wget? It's free and can run from VPSes.

 

[madman]

Have you heard about wget? It's free and can run from VPSes.

nipester, I have not heard of wget. Is VPSes just a server like hosgator? I don't know much about running scripts if I have to set them up. If you could explain a litle more in detail what this involves I may understand better.

Thanks

 

[zone69]

You can use wget to download the whole site.

See this post I posted a while back for how to do it.
http://www.blackhatworld.com/blackh...-server-based-website-crawler.html#post745372

Hope that helps!

 

[madman]

You can use wget to download the whole site.

See this post I posted a while back for how to do it.

Hope that helps!

zone69,

Thanks for the info. Does wget have a GUI interface or do you have to use command line instructions to use the program?

Thanks

 

[ru ru]

I'd rather use httrack.com than wget.

 

[niggles]

Are you wanting to grab a whole page or just the articles?

If it's just the articles, then even if you've grabbed the whole page, to automatically strip the article, you need to figure out the tags the article is between to strip out the actual article e.g <div id="article">some text...</div>

Cheers,
Niggles

 

[madman]

Are you wanting to grab a whole page or just the articles?

If it's just the articles, then even if you've grabbed the whole page, to automatically strip the article, you need to figure out the tags the article is between to strip out the actual article e.g <div id="article">some text...</div>

Cheers,
Niggles

Niggles,

Just the articles. B00kMarkkingDeemon has a function where it crawls a website but it only returns approx.100-200 charcters of the article in txt format, not the whole article. So I'm looking for something like BMD except that it will return the whole article text.

Thanks

 

[matapples01]

There's a Jr VIP Neta10 who has a paid scraper that may do what you're looking to do. I haven't used it to pull content but you enter the tags to search between and it can pull content from between the tags. It's here..

http://neta1o.com/index.php?/programs.html

 

[cemdev]

x2 httrack - it'll copy an entire website.

 

[twobol2002]

you can also use TelePort

 

[madman]

There's a Jr VIP Neta10 who has a paid scraper that may do what you're looking to do. I haven't used it to pull content but you enter the tags to search between and it can pull content from between the tags.

bobfrapples,

You have finally described what I am looking for. Essentially, I want a content scraper that will pull content from niche blogs so that I can rewrite the content to be unique. Just like rewriting an EEzineeAArticle. Are there other scrapers out ther too?

Thanks

 

[nirose]

I too use httrack. its very usefull and you can download with the filetype that you want.

 

[madman]

I too use httrack. its very usefull and you can download with the filetype that you want.

Hi nirose, I downloaded and installed httrack. I cannot figure out how to just download the blog posts text, during a test using one of my own blogs.

Could you explain how to configure the program to accomplish this task.

Thanks

 

[cemdev]

madman - httrack doesn't work that way. it'll make an offline copy of an entire website.

if you want to extract just blog posts you're going to have to write a scraper - and it's most likely going to have to be modified to the site you're scraping. you don't have to start from scratch - there are hundreds of free scraper scripts out there - pick one that comes close to what you want and modify it for the site you want to scrape. it shouldn't take long or be very hard, but you'll need some basic programming knowledge (this is a good thing to acquire anyways)