[GET] Free content for any niche. (get it before it's gone)
- Thread starter prab1996
- Start date
ranjim
Regular Member
- Apr 5, 2012
- 399
- 47
Asked Loopline regarding this. He mentioned that it is not capable of doing so. You're better off with what other members suggested above (proxyswitcher (not sure if this is the one) + download the mall addon on firefox.). Or download free trial of Seo Content Machine. DL the lists given above and run them on SCM's URL content scraper. A proxy would last around 500 articles so you'll need a lot if you plan on scraping a lot.
fatboy
Elite Member
- Aug 13, 2008
- 1,615
- 3,262
And here I was thinking I heard the last of this shitty thread.
No one is singling me out, kvmcable mentioned me as they thought you were throwing your toys out the pram about me. If you were trying to slate someone else then I have no doubt whatsoever kvm would of done the same for them.
If you think about it everyone is owed respect UNTIL they make it obvious they do not deserve it, in my opinion living by that rule will be a lot better, however you appear to go from the other direction that everyone means nothing to you until they live up to your standards.
Do you know every single addon in all the browsers? If you read back I pointed out that all you really need is a bit of bash script to do it all. Not everyone likes running scripts, not everyone likes browser plugins, not everyone likes running exe files - horses for courses.
Can you point out anywhere where I say that I am a skilled programmer? I use Ubot FFS, thats drag and drop. Do I have a programmers mindset, yes, do I have the skills to use multiple languages - depends how you look at it. I put stuff together using perl, php, ubot and little bits of .net as I try to move away from Ubot as the company behind it is dragging it to the bottom if a murky sea. If you want a skilled coder, you wouldn't want to pay pennies for a bit I write, you want skilled look at the likes of TomPots or DarkPixel, they are in my eyes the skilled programmers.......I am a hobby hacker, I have a 9-5 (no matter how shit that is) to pay the bills, coding, scripting, hosting and sysadmin is my hobby for beer money. Never made a secret about that.
Then lets do it, the thread has served its purpose, bitching all day isn't going to pay your bills.......
No one is singling me out, kvmcable mentioned me as they thought you were throwing your toys out the pram about me. If you were trying to slate someone else then I have no doubt whatsoever kvm would of done the same for them.
If you think about it everyone is owed respect UNTIL they make it obvious they do not deserve it, in my opinion living by that rule will be a lot better, however you appear to go from the other direction that everyone means nothing to you until they live up to your standards.
Do you know every single addon in all the browsers? If you read back I pointed out that all you really need is a bit of bash script to do it all. Not everyone likes running scripts, not everyone likes browser plugins, not everyone likes running exe files - horses for courses.
Can you point out anywhere where I say that I am a skilled programmer? I use Ubot FFS, thats drag and drop. Do I have a programmers mindset, yes, do I have the skills to use multiple languages - depends how you look at it. I put stuff together using perl, php, ubot and little bits of .net as I try to move away from Ubot as the company behind it is dragging it to the bottom if a murky sea. If you want a skilled coder, you wouldn't want to pay pennies for a bit I write, you want skilled look at the likes of TomPots or DarkPixel, they are in my eyes the skilled programmers.......I am a hobby hacker, I have a 9-5 (no matter how shit that is) to pay the bills, coding, scripting, hosting and sysadmin is my hobby for beer money. Never made a secret about that.
Then lets do it, the thread has served its purpose, bitching all day isn't going to pay your bills.......
BreaknBrix
Power Member
- Mar 25, 2014
- 778
- 4,523
Speak for yourself.
If the bot really "worked" like a real scraper you wouldn't see members still making posts asking how to mass download the articles. And I wouldn't have gotten 6 pm's in the last 12 hours from people saying "I used fatboys bot but it won't do what I need, can you please share your method?" So it works for small, limited scrapes, but not for mass downloading.
Some of you guys are really close though. Just use download them all by firefox, a proxy switcher, proxies, and if you need anything more than 25k articles you will need some macros to automate the process. Just be sure to write "cache" in the filter part of the DTA so it only downloads cached versions. Right click, download, wait till the addon stops working (I was averaging 300 articles / proxy), switch proxies, repeat over and over till you got what you need. It's very simple and works better than that bot (imo). And again, I appreciate him taking a whole 20 minutes to build the bot but some people here wasted DAYS on end just figuring out this stupid simple method.
I agree. I'm done.
-BB
Last edited:
fatboy
Elite Member
- Aug 13, 2008
- 1,615
- 3,262
Okay - last reply before I unsubscribe from this thread.......
The bot worked fine when Yahoo Voices was still up and going, I scraped a load and others did to. Yes, now you have to scrape cached links its as much use as tits to a bullfrog. I can put proxy switching in it quite easily, but to be honest your way of doing it will probably be just fine.
At a guess you could probably use a hundred or so proxies, select a random one, download 50 articles (Google seems to captcha at 70 - 75), switch to another proxy, repeat.....by the time you get to the end of your proxy list Google would of probably chilled its beans enough to use the first one again.
Loop that bitch up and you will be happy.
Like I said, thats my last reply on the thread / bitchfest.
Enjoy the weekend!
BreaknBrix
Power Member
- Mar 25, 2014
- 778
- 4,523
Ah, I did not realize you built the bot when voices was still up. For some reason I thought the database went down July 1st but I had to check and realized it was Aug 1st. Makes perfect sense now.
Anyway, enjoy your weekend too. And congrats on winning Jr VIP.
- Jun 20, 2013
- 11,510
- 8,916
I have posted a article on my blog but it didnt get index, i tried everything. That article is already deindex from Google 3 days ago. Any solution ?
prab1996
Elite Member
- Jan 8, 2013
- 3,492
- 2,044
rominholima
Newbie
- Jul 26, 2014
- 12
- 1
Some of them are already deindexed. On my niche, 2 of the 100's articles that I (manually
) got are posted on my site!Check regularly!
Dan Free
Regular Member
- Nov 22, 2007
- 221
- 879
Well, I got to this thread a couple of days late, but it is a interesting read.
The basic technique is a great find. I am not planning on using the raw articles because who knows who else will be using the raw articles and therefore you would have duplicate content.
This technique reminds me of scraping Amazon's Cloud:
site:.s3.amazonaws.com "keyword"
site:.cloudfront.net "keyword"
What you get here are mostly PDF files, but you might find an unlocked bucket.
The basic technique is a great find. I am not planning on using the raw articles because who knows who else will be using the raw articles and therefore you would have duplicate content.
This technique reminds me of scraping Amazon's Cloud:
site:.s3.amazonaws.com "keyword"
site:.cloudfront.net "keyword"
What you get here are mostly PDF files, but you might find an unlocked bucket.
InsanelySane
Power Member
- Nov 23, 2013
- 574
- 116
Glad the bitching fest is over
Fatboys bot works flawlessy and is essentially idiot proof. Its designed to scrape target urls text body. It doesnt apply the google.webcache to links automatically, you need to manually do it.
In regards to the content
I understand the concept here that once content clears the cache its essentially "unique". Though you would think (you would think) google would have some kind of safeguards or backup-backup cache here for being aware of duplicate content. If not for everyday sites then simply due to the size of the de-indexing, and the fact its yahoo. Google must have known that scraping would be a concern of a site of this magnitude.
Would be interested to see if people have these articles not just ranking but ranking well.
my 2 cents
Fatboys bot works flawlessy and is essentially idiot proof. Its designed to scrape target urls text body. It doesnt apply the google.webcache to links automatically, you need to manually do it.
In regards to the content
I understand the concept here that once content clears the cache its essentially "unique". Though you would think (you would think) google would have some kind of safeguards or backup-backup cache here for being aware of duplicate content. If not for everyday sites then simply due to the size of the de-indexing, and the fact its yahoo. Google must have known that scraping would be a concern of a site of this magnitude.
Would be interested to see if people have these articles not just ranking but ranking well.
my 2 cents
cheekykent
Regular Member
- Sep 3, 2013
- 364
- 74
PM me if you can scrape niche related articles. Will pay for your time obviously.
Retraction
Junior Member
- Mar 3, 2014
- 170
- 183
Okay, I'm tired of PMs. I'm not going to release my private method, but this also works.
Scrape the URLs. There are a couple of good posts that give you a tutorial. I think someone dropped a 200k+ yahoo voices URL txt file here. Use that if you're too lazy.
Download Notepad++ and open the txt file.
Press ctrl+f and press the "Replace" tab
Find what: http://
Replace with: http://webcache.googleusercontent.com/search?q=cache:
There you go. You got 200k+ links of google cached yahoo voice articles.
http://www.metaproducts.com/mp/mass_downloader.htm
Download that. Import your URL list and press download. Don't make thread count too high or its going to skip over a lot of articles.
Make sure you have lots of proxies or a couple of good VPNs
Don't PM again about this. FFS people, I have never coded in my life and I found two methods.
Scrape the URLs. There are a couple of good posts that give you a tutorial. I think someone dropped a 200k+ yahoo voices URL txt file here. Use that if you're too lazy.
Download Notepad++ and open the txt file.
Press ctrl+f and press the "Replace" tab
Find what: http://
Replace with: http://webcache.googleusercontent.com/search?q=cache:
There you go. You got 200k+ links of google cached yahoo voice articles.
http://www.metaproducts.com/mp/mass_downloader.htm
Download that. Import your URL list and press download. Don't make thread count too high or its going to skip over a lot of articles.
Make sure you have lots of proxies or a couple of good VPNs
Don't PM again about this. FFS people, I have never coded in my life and I found two methods.
xha44a
Power Member
- Dec 2, 2012
- 534
- 458
I've got the 12k niche related articles I wanted so I'll throw out some tips. For those who are pulling from Google cache if you add a &strip=1 operator to your URL, it removes pretty much all rich formatting and leaves you with just the article (minus a few menu's on the bottom)
ThopHayt
Elite Member
- Jul 25, 2011
- 7,012
- 2,864
Let me clarify a point, I believe that the OP is suggesting you use these for a base of SPUN content. So you splice, spin, and reuse the articles. Not for direct use. Still not THAT useful because spammy garbage content is barely useful today. Two years ago this would have been GREAT, but not anymore IMHO.
