[GET] Article Scraper Created By Myself

Hello everyone!

This is a basic article scraper. I am sick of using SENuke's basic EZineArticles downloader I would prefer if the content came from more than one directory. This is nothing fancy - it's not multi-threaded and there is no proxy support yet (Although you can just set IE proxy to use one).

The tool currently supports:

Instructions:
1. Select whatever platforms you want to scrape from
2. Enter the search query
3. Tweak the delay [in milliseconds (1000ms=1s)]
4. Select the maximum number of articles from each directory.
5. Press start
6. Articles will be saved in the same directory as the .exe under "query.txt".


Article Base is touchy and it seems to be quick to captcha block. Increase the delay if you want to scrape more than a couple articles from that directory.
This only works on windows and requires internet explorer.

I am looking for feature suggestions - I was thinking of having it scrape for images and/or videos and inserting them in the articles. I may add this feature soon.
I am also looking for more quality article platforms I should add.

It has been a while since I shared one of my useful scripts - I have seen many people re-sharing without giving credit and even selling them in the past. Hopefully if someone is going to share this they could please at least give me credit.

Screenshot:


Download (Updated Version):

Virus Total:
Hmm... It is showing 1/44 detection ratio. Very weird it is just a basic AutoIT script - you can decompile it if you feel so inclined.

So it will download the full article(s) and then store them in .txt? Will it seperate the articles so they are not run into each other? Also i take it it does not do any spinning (meaning taking 1 paragrapgh from one article and then the second paragraph from another)

what is the output ? just a file with 8 articles in txt ?

SEORasta said:

So it will download the full article(s) and then store them in .txt? Will it seperate the articles so they are not run into each other? Also i take it it does not do any spinning (meaning taking 1 paragrapgh from one article and then the second paragraph from another)

Click to expand...

It will download the full articles and put them in a text file.

The formatted text file looks like this:

EZINE ARTICLES:
blah blah blah...
blah blah...
blah blah...

HELIUM ARTICLES:
more blah...
even more blahs....
etc...

I didn't bother doing paragraph spinning because SeNuke already takes care of that for me. This was a tool I created for myself that I figured would be useful for everyone else too . I may be able to add that kind of functionality later on if there is a big request!

Djalminha said:

what is the output ? just a file with 8 articles in txt ?

Click to expand...

If you choose maximum 2 articles and select all four directories then you will get 8 articles. You can crank up maximum number of articles and have it scrape a lot more though!

Thanks very much.

In which language have you wrote it?

edit: Got it, auto it. It is not as fast as vb.net.

Very nice Share OP will help with creating wiki's and web 2.0s so thanks

moonlighsunligh said:

Thanks very much.

In which language have you wrote it?

edit: Got it, auto it. It is not as fast as vb.net.

Click to expand...

Yeah it's just a quick script I wrote up and compiled in AutoIT. Not the fastest but I just wanted to get it done quick and dirty. I was considering writing it in C# but time is money .

od265 said:

Very nice Share OP will help with creating wiki's and web 2.0s so thanks

Click to expand...

No problem! That's the plan - I wanted to expedite making my SENuke campaigns. Let me know if you have any feature suggestions I'm all ears!

Ok I get it now....Proxy support would be GREAT so we can scrape like 10 + articles or so per directory. Right now though if I keep it at 5 per diretory I should be ok under my main IP.....will try later but thank you!

This seems awesome, I'm definitely going to try it out...

SEORasta said:

Ok I get it now....Proxy support would be GREAT so we can scrape like 10 + articles or so per directory. Right now though if I keep it at 5 per diretory I should be ok under my main IP.....will try later but thank you!

Click to expand...

I can probably add this in the next couple days. I have my hands full at the moment. I have all of the proxy rotation code written up in other projects its just a matter of integration.

Bestbuyfoam said:

This seems awesome, I'm definitely going to try it out...

Click to expand...

Let me know if you have any feature suggestions!

This is awasome. I also use other software to collect articles. But this is the best I have ever seen. Many thanks.

I've updated the software. I've added a fifth platform it can scrape - Yahoo Voices.
I've also cleaned up the output a little bit.

I will probably release a little update soon that incorporates proxies and whatever other requests I receive.

Here is the new download link:

Damn all the places I go to does not have the actual link.....got a better link to DL ?

SEORasta said:

Damn all the places I go to does not have the actual link.....got a better link to DL ?

Click to expand...

Here is a mirror on dropbox. Hopefully this helps. All the links on multiupload work fine for me.

No need to try it then

Thanks very much, seems to work well

chris4004 said:

No need to try it then

Click to expand...

Your post is very confusing - why is there no need to try it? Elaborate and maybe I can improve whatever your looking for.

jeromespitfire said:

Thanks very much, seems to work well

Click to expand...

Glad you like it! Please give me whatever suggestions you think would make it better!

thanks for the good tip, you just saved me lots of time

Hey Everyone,

Just added proxy support to the script (proxies that don't require authentication). This will allow you to scrape articles much faster by allowing a lower delay time and avoid getting blocked by the directory spam controls.
I also added a checkbox for the option to show IE or not.

To use proxies add your list of proxies to "tProxies.txt" in the application. Each proxy should be on a new line in format "host":"port" - I've included a few proxies as a sample.

Screenshot of the updated GUI:


Download it here:

Let me know if you have any problems or suggestions!