Blocking/stopping scrapers

CredibleZephyre · Oct 31, 2013

I have a client who has had his content stolen quite a bit. I've done some research etc on most of the usual ways to attempt to prevent scraping (there's no foolproof plan but some are effective for the lazy scrapers... if that isn't redundant). But this is blackhat world and many of the users here are the kind that come up with creative solutions to problems, so does anyone have some fancy ideas or techniques to hinder scrapers efforts?

roadhamster · Oct 31, 2013

You can script some javascript, what most of the bots/ scrapers can't/ don't execute. If a bot visits your site, the javascript won't be executed, and no content is loaded.
Has a little drawback for visitors who have javascript disabled.
Good luck.

sags22 · Dec 1, 2013

IP filtering and scripting a non executable java script can help keep the scrapers at bay. This prevents the content from appearing to them. CAPTCHA texts are also highly used to hinder most of the scrapers.

ShifterIO · Dec 1, 2013

I suggest you think regarding the javascript. You might end up with google bot not be able to read your content correctly.

Schvamp · Dec 1, 2013

On top of my head I can think of.
-Block known low quality IPs and remove proxy access.

About javascript.
Google bot shouldnt have any problem understanding it. But you might run into problem with visitors having JS disabled.
Instead you can reverse the idea. Include a paragraph with backlinks to the original source and your brand name where it was posted.
And then have JS that paragraph removed.

Result?
The user wont have any problem with the content, and if JS is activated, they wont see that extra paragraph.
If it's not active, they will see it. But it's not that big of a deal is it? You can even set the font color to match the background.

You wont be able to stop scrapers. But you can improve what content you give them.

murachi · Dec 1, 2013

It's impossible to stop it altogether but there's a few techiniques you can use.

As others have said, adding extra javascript can help but it may stop search engines being able to crawl your content properly.

Something I've done in the past is, setup robots.txt, setup a page called jail.html, disallow access to the page through robots.txt then insert a link on one of your pages, hiding it with CSS and record IPs of visitors to jail.html.
This can help to quickly identify requests from scrapers that are disregarding robots.txt

Another technique is teergrubing, when you identify a scraper, keep his connections open for as long as physically possible without timing them out. Although this may alert them that you're on to them.
http://en.wikipedia.org/wiki/Teergrubing

TZ2011 · Dec 1, 2013

Schvamp said:
On top of my head I can think of.
-Block known low quality IPs and remove proxy access.

About javascript.
Google bot shouldnt have any problem understanding it. But you might run into problem with visitors having JS disabled.
Instead you can reverse the idea. Include a paragraph with backlinks to the original source and your brand name where it was posted.
And then have JS that paragraph removed.

Result?
The user wont have any problem with the content, and if JS is activated, they wont see that extra paragraph.
If it's not active, they will see it. But it's not that big of a deal is it? You can even set the font color to match the background.

You wont be able to stop scrapers. But you can improve what content you give them.

Both things that you mentioned would be considered a problematic with google. Presenting other text for users with/without javascript ? Font color to mach background color ? Looking for trouble. Google bots are not stupid, but also not smart enough to recognize your intentions and qualify methods (heavily used by spammers last X years) to be acceptable behavior and good visitor experience.

benarata · Dec 4, 2013

You will never stop your sight from ever being scraped 100% of the time. If somebody wants it they will get it. But what you can do to protect his content is File a "Real" copyright registration for his work through the US Copyright Office. I do this every month I take all of the changes and new content on my all of my sites web 2.0 everything and register them as a group that way its only 35.00 for all of the work added that month on all of the sites and blogs. You can do this every three months but it needs to be done monthly to keep any of your works from slipping through the cracks and missing a deadline. When you catch someone stealing your content file a DMCA take down with scroogle MS and yahoo the take down will happen fast with a copyright registration from the US Copyright Office. Its a slam dunk and the infringe party will have a hard time clamming it was a bad take down order. I usually will not file with their IP -it fucks with them harder when the are trying to figure out why there sites have disappeared on the major search providers.

validseo · Dec 4, 2013

I've written content scraping bots for fortune 500 companies for competitive intel purposes... The single best thing you can do is to put the data into images and then visibly watermark and copyright the images... most datascrapers are trying to populate fields in a database. Make it hard for them to do that cleanly.

CredibleZephyre · Dec 4, 2013

validseo said:
I've written content scraping bots for fortune 500 companies for competitive intel purposes... The single best thing you can do is to put the data into images and then visibly watermark and copyright the images... most datascrapers are trying to populate fields in a database. Make it hard for them to do that cleanly.

Yeah images makes it tough but that makes your content non-indexable doesn't it?

bartosimpsonio · Dec 4, 2013

Create ONE invisible link from a 1 pixel image, the robots will follow the link, the humans won't. Whoever follows that link is a scraper, block them.

Repulsor · Dec 4, 2013

Well, any ideas except using Javascript to load content can be scrapped even by a newbie programmer. Assured. Curl is really powerful, and easy as well to overcome those tiny little blocks you may put forward. IPs,Cookies, etc etc.

Go with the javascript idea.

validseo · Dec 4, 2013

CredibleZephyre said:
Yeah images makes it tough but that makes your content non-indexable doesn't it?

But if google can index it then anyone can scrape it... cant have it both ways. Even if you made it so only Googlebot sees the text version... people will just scrape the data from the google cache... it is either wide open or made inconvenient in images.

Javascript wont help you either... its easy to make bots that render javascript and perform mouseovers and clicks... it might stop a beginner, but nobody with any bot writing experience will be stopped by those measures... at the end of the day the data is either text or images and it is all communicated via HTTP... everything else is cheap parlor tricks that attempt to make people think it is more complicated than that.

DunDidIt2X · Dec 4, 2013

There are only ways to keep unexpierence scrapers at bay. Best way for you would be to actually use java to encrypt the content itself. But there is no stopping scraping pro's, unless you make the content exclusive like password protected. But then again this is only good till a scraper signs up and gets access to the sites data. But then you could build a protocol which only allows signed in users to access so many webrequests on the site by Username. IP blocking etc wont work.

validseo · Dec 4, 2013

If it is the entire collection that has the value (meaning 75% of the collection would be totally worthless) then make a third of the pages always display the data in an image.... That way most of your site is indexable and the scrapers can't have the whole dataset.

If the pages have integer ids in the database then id % 3 == 0 means the page data is displayed in image instead of text.

Nigel Farage · Dec 4, 2013

Have someone make you an ad-sense clicking autobot, get about 1000 legit proxies and get their ad-sense account banned.

Corydoras007 · Dec 5, 2013

validseo said:
If it is the entire collection that has the value (meaning 75% of the collection would be totally worthless) then make a third of the pages always display the data in an image.... That way most of your site is indexable and the scrapers can't have the whole dataset.

If the pages have integer ids in the database then id % 3 == 0 means the page data is displayed in image instead of text.

This is actually a cool idea and think this can actually work if you're worried about your content getting scraped. THere is no foolproof way of preventing scrapers so might as well give them incomplete data.

My only worry is how this will affect your SEO. But I like the idea!

Tensegrity · Dec 14, 2013

As a veteran screen scraper, I can tell you that there is absolutely no way to completely stop screen scrapers aside from making your site approved members only. I do all my scraping with javascript and it's so massively easy that I've made a living doing it.

What you need to do is tell your client to stop focusing on people scraping and focus more on selling. The internet is for sharing, that is how it works, how it runs, and how it will always be. If your client focuses more on advertising and finding customers, or using their skill for something that can't be scraped (like a service or custom job), then they will be more successful.

Blocking/stopping scrapers

Junior Member

Regular Member

BANNED

Elite Member

Power Member

Newbie

Senior Member

Junior Member

Senior Member

Junior Member

Elite Member

Senior Member

Senior Member

Elite Member

Senior Member

BANNED

Regular Member

Elite Member

Main Menu

Marketplace

Making Money

BlackHat World