Anyone successfully scraped EzineArticles?

Advertise on BHW
curl and php working just put a sleep command into the code sleep(5) it's stopped they query 5 sec

and you can use custom random useragents and random proxies with curl..

this is the solution.. some site don't like if you doing querys without useragent.. like blackhatword:)

and another trick setup the refererr to last page what your bot visited.. if you do this all it will work perfectly
 
If your script for scraping is working, you may try to make a few seconds delay like Tanksapka says or you can improve your script to make each query through different proxy server (elite, anonymous - so it doesn't show your IP to EZA). You can make a loop in php that get's random proxy (from each line of text file where you would store your anonymous proxies) for each query.
 
Hey i have done one article they approved that also , bit lucky so i got payment.
 
I've got EzineArticles, GoArticles etc. support for the WordPress AutoBlogging-Plugin I'm working on.

Here's how I've implemented it:
EzineArticles use Google Custom Search to enable users to search their site. By opening up the Google Search iframe in its own window you can see what params it has, just recreate this in your script and you'll be able to get the urls you need for the articles. Usually you do not have to use proxies etc. to get these urls from Google.

For scraping the articles I currently use PHP and CURL (will switch to Ruby soon, but that's another story...) and I generate a random user agent for every request (I've got a yml-file with a few thousand various UAs I scraped from some UA-site). I also accept and store cookies from EzineArticles with CURL (occasionally they'll check the cookies to see if you're a bot). If my ip gets banned, I alternate between a few thousand HTTP/SOCKS5-proxies I got somewhere. An alternate route is to scrape the articles from Google's/Yahoo's cache (as someone else mentioned), but I've done that before and I've run into some captcha stuff etc, this solution works better for me.
 
SebastianJ said:
I've got EzineArticles, GoArticles etc. support for the WordPress AutoBlogging-Plugin I'm working on.

Here's how I've implemented it:
EzineArticles use Google Custom Search to enable users to search their site. By opening up the Google Search iframe in its own window you can see what params it has, just recreate this in your script and you'll be able to get the urls you need for the articles. Usually you do not have to use proxies etc. to get these urls from Google.

For scraping the articles I currently use PHP and CURL (will switch to Ruby soon, but that's another story...) and I generate a random user agent for every request (I've got a yml-file with a few thousand various UAs I scraped from some UA-site). I also accept and store cookies from EzineArticles with CURL (occasionally they'll check the cookies to see if you're a bot). If my ip gets banned, I alternate between a few thousand HTTP/SOCKS5-proxies I got somewhere. An alternate route is to scrape the articles from Google's/Yahoo's cache (as someone else mentioned), but I've done that before and I've run into some captcha stuff etc, this solution works better for me.
Click to expand...

How'd you find a couple thousand SOCKS5 proxies? With only 1k, my list is tiny in comparison.
 
TheDaemon said:
How'd you find a couple thousand SOCKS5 proxies? With only 1k, my list is tiny in comparison.
Click to expand...

elite-proxies - blogspot
proxy-heaven - blogspot
socks24 [org]

(Couldn't post the urls straight up as I haven't been a member too long, so the forum software is a bit whiny.)


I get all my proxies from those sites (well, I've also got a few private ones, but mainly from there). I've also written a custom script that automatically test the proxies and removes the unresponsive ones so that I'm only left with the ones that actually work.
 
SebastianJ said:
elite-proxies - blogspot
proxy-heaven - blogspot
socks24 [org]

(Couldn't post the urls straight up as I haven't been a member too long, so the forum software is a bit whiny.)


I get all my proxies from those sites (well, I've also got a few private ones, but mainly from there). I've also written a custom script that automatically test the proxies and removes the unresponsive ones so that I'm only left with the ones that actually work.
Click to expand...

Good deal. I have a script that does the exact same thing, but I was just pulling from socks24 and the *.*.*.*:27977 trick on Google.
 
Code: 
I was going to do a scraper in php when I thought it might be easier  to just use yahoo pipes.

Theres nothing wrong with using PHP for Scraping, its actually quite efficient. But you can scrape in Python or Perl almost as easy.



Code: 
Well I tried using a pipe and EzineArticles  basically blocked the scrape :(   lol.. Here is what I got back after I tried to scrape:

[B]You are browsing  faster than a normal human would.[/B]

  You may be seeing this page for one of the following reasons:
 
[LIST]
[*]You have performed too many searches in a short period of time.
[*]You have requested too many pages in a short period of time.
[*]You have exceeded the daily allowable limit of page views.
[*]You have used a script or program to scrape content or keywords.
[/LIST]
I didnt even know there was a daily limit on how many articles you could view.  anyone else scrape from here?

first step:

1) Create a PHP script to scrape the website
2) Read about PHP XPath and Content scraping (google it)
3) Ensure you are setting a good Useragent when the script makes a connection to the website
4) on your dedi you can run proxychains and cycle through proxys in /etc/proxychains.conf, and just prefix ./proxychains php <yourapp>.

I can build you the solution if you want in under an hour lol
 
I am working on scrapt ezine now. I have vps and unlimited sock5. So i can do it. But the problem is Ezine has 5 millions article while I can pull 25k 30k articles daily. It will take me some months to get all ezine articles
 
I am working on scrapt ezine now. I have vps and unlimited sock5. So i can do it. But the problem is Ezine has 5 millions article while I can pull 25k 30k articles daily. It will take me some months to get all ezine articles
Click to expand...
What's the point of scraping all their articles? To create an EzineArticles-replica? :)
 
Well... duh. But what's the actual point of scraping ALL their articles compared to only scraping the ones you actually really need (based on keyword, category etc.)? If you are going to scrape 5 million articles, you definitely got to have really decent indexes on your database otherwise all queries against your articles table will be super slow. And yeah, your database will be huge as well.
 
Sebastian, can you really only scrape 30k/daily? This possibly means that you have not a clean list of proxies or your VPS is just slow. Or the insertion in the databases takes its time?

On my home computer with 16MBit/s at least w/o proxy, 10k/hourly is perfectly possible.
 
Speaking of scraping, anyone notice Tor access of EA now? Looks like they're putting a captcha on Tor hits now.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top