An expert in stoping scrapping bots frrom webkit 534 on a website

[labishops]

I am experiencing negative seo for my website, each time i post a new post in my wordpress, i see it immediately in my competitors website....they copy, scrape it through webkit 534. How can one stop this scripts/softwares

 

[Jinyus]

I am experiencing negative seo for my website, each time i post a new post in my wordpress, i see it immediately in my competitors website....they copy, scrape it through webkit 534. How can one stop this scripts/softwares

The easiest way is to use cloudflare

 

[roadhamster]

Are you using PHP? Probably wordpress? Is the webkit 534 a user agent? Put something like

if(preg_match('/534/',$_SERVER['HTTP_USER_AGENT']))
{
header('HTTP/1.1 500 Internal Server Error');
exit();
}

in your header.php in wordpress

It gives an http 500 error code and exits wordpress for them.

 

[SpoonFeeder]

The first line of defence should be blocking their IP/IP Range via .htaccess and blocking the user agents as @roadhamster suggested.

I personally convert the data(numbers/charts) into image and watermark it with my URL so it takes a whole lot of work and time and resources for them if they were to pull the data from the pic.

 

[labishops]

Thanks. Let me try then il get back

 

[babypar]

If you detect that agent you should send them content which would really hurt their rankings

 

[Marcel Freiberg]

Hmm, couldn‘t you use it for some advertising for your page. A nice article about your blog why you made it what the address is, ... maybe they will learn that automatic copy can harm em.

Best luck
Mf

 

[labishops]

Thank you all for the good response....I run a custom writing site thus we post alot of materials, in 1000s thus its hard to customize each to hurt when they run the script to scrape... am hoping I can find a lasting solution. because they copy and still rank

 

[labishops]

@roadhamster cc. others ....As you can see, blocking the IP adress wont help...they seem to access with multiple IP addresses(hundreds) , none is similar to the next. Its like a scrapping script is running. Check images below. I integrated the php code on the head but yet am seeing no change

 

[SEOMadHatter]

I've built bespoke scrape and automation systems on a fairly large scale over the last 2 or 3 years. Some of the suggestions above will help if they stick with a simple scrape showing a webkit UA but it'll take you more time to implement CloudFlare or something than it will for them to get around it.

Some big sites spend a lot of money to prevent scraping and there's nothing which reliably works. If they want to scrape you - they're going to. Anyone who says otherwise is kidding themselves.

If you detect that agent you should send them content which would really hurt their rankings

I've had automated sites pick up adult content by mistake (it was closely related to the niche) and it hasn't dented normal rankings. If the scrape picks up links (unlikely) you could try linking out to some pharma sites but it's not likely to help.

You could *try* scaring them off legally. Depending on where they're hosted a scary sounding letter might be enough but again if it's worth them doing it there's very little to stop them posting your content to 2.0's or something. These can get indexed before original content as well.

In terms of negative SEO this isn't anything new. It was in a public case study... I want to say two years ago and honestly, I've used it myself. If you can't scare them off legally your best bet is to get Google to realize you're the original source. Get some good links and hope the spiders like you better. Using an indexing service *might* help but if I was in your shoes I'd probably setup an IFTTT to syndicate your content to social networks and a few 2.0's to try and convince Google you're the source.

 

[labishops]

Thanks for your input. Much appreciated.

 

[roadhamster]

@roadhamster cc. others ....As you can see, blocking the IP adress wont help...they seem to access with multiple IP addresses(hundreds) , none is similar to the next. Its like a scrapping script is running. Check images below. I integrated the php code on the head but yet am seeing no change

Check the IP's, they're from Microsoft, the useragent says it's BingPreview, so in combination with a whois on the IP's it's the bingbot visiting your website, and yes, they're crawling your website for the bing-search index, they are known to scrape websites. Nothing to worry about.

Another option is to do a ping to the website which is copying your content and block the IP the website is running on, most likely the script which copies your posts is automated and running on the same server.

 

[SEOMadHatter]

Oh actually. There is one thing.

You could submit a spam report to Google. It'll take them forever to act on it and if you're doing any kind of shady SEO yourself I wouldn't go down this route (be brutally honest with yourself here because an SEO knows what they're looking at and Google will have more info to go on).

Again they'll just swap to another domain or 2.0 but it's something if your own site is squeaky clean.

 

[Deleted member 657006]

i would do this .

use my own ip tracker , find the ip that are scrapeing the info add them to a databae and redirect all them ip back to google best way .

example only.

<?php $allowed = array( '123.45.67.890', '456.45.67.890', '789.45.67.890', ); if(!in_array($_SERVER['REMOTE_ADDR'], $allowed)) { header("Location: /block.php"); exit(); } // continue with code ... ?>

 

[Deleted member 657006]

you can make up somethink easy with mine above and this example .

http://www.analysespider.com/ip2country/redirect-visitors-country.html

 

[rickydzine]

As others have said, just block the UA, seems to be an amateur scraper, using a default scraper. If the user agent changes then don't bother wasting time. It's near damn impossible to stop scrapers these days.

 

[babypar]

One way you can detect bots is monitoring the time between requests, if they are too quick, just add the ip to a blacklist. It’s not perfect but it may work for you depending on how they scrape

 

[labishops]

Wow. Thanks alot.