Wp theme detector crawling

This answer was taken from Quora...
There are two major ways to block a crawler:

1. Block it's IP addresses. You'll have to collect all the IP's of the crawler (which may or may not be a big deal), and add them to a blacklist of your web-server, firewall, or any other software or service that you might be using to protect your website. With this kind of block the crawler cannot even start a connection to your website, which means minimal resources of yours are spent fighting the crawler. You could of course do the same at application level - by analyzing the IP address of the requester and serving an error, or empty reply, or drop a connection. But that would mean you spend too much resources (including your time to code the logic) instead of just using the facilities of your web-server.

2. You may block the crawler at a higher level - by analyzing the "User-Agent" HTTP-header, and serving some HTTP-error, like 503, instead of the content. You could also just drop the connection instead of spending resources on reply. This implies that crawler does not hide it's identity and not uses user-agent of some web browser. This also means that you spend quite a lot of system resources on accepting a connection, analyzing the request and serving reply.

I would not call robots.txt a method to block a crawler, as most crawlers you'd like to block do not honor robots.txt anyway. However, if your question was about how to instruct polite crawlers like Googlebot about what parts of your site are to be crawled, then robots.txt is your choice.
 
There are plugins that stop theme and plugin detectors.

I don't have any recommendations but you can find articles and information with a Google search.
 
You can block IPs, or other headers they could be sending. Try to debug by crawling your own wp site and see what you can catch. Sometimes it's a user agent or a fixed IP which can easily be blocked or even redirected
 
Oh and sometimes the IP and or user agent can be spoofed, so you can also try to check the IP hostname and other IP data to see if they are really the wp crawler
 
Back
Top