Should replica web servers of a load balanced server be blocked for public access?
- Thread starter xpro
- Start date
- Dec 3, 2015
- 5,337
- 6,024
Can you clarify whether your primary concern is preventing search engines from indexing the replica servers, or are you also worried about users directly accessing those servers and bypassing the load balancer?
xpro
Regular Member
- Jan 21, 2009
- 442
- 24
I'm not concerned about users accessing the replica servers. My main concern is that Google does not consider the content of replica servers as duplicate content. At this point I think it's safe to block public access to the replica servers right? Only the load balancer should be able to access it?
- Apr 23, 2012
- 2,876
- 5,898
Depends on your existing load balancing setup.
Can you tell me the exact setup? Reverse Proxy? DNS Round Robin? Or something else?
Apart from your main domain, how are these replicas accessible via the open web? Sub domains? IPs?
Can you tell me the exact setup? Reverse Proxy? DNS Round Robin? Or something else?
Apart from your main domain, how are these replicas accessible via the open web? Sub domains? IPs?
xpro
Regular Member
- Jan 21, 2009
- 442
- 24
DNS is pointing to a single HAProxy server. HAProxy to a replica IP (replica 1) and a subdomain (replica 2) on a round robin fashion.
- Nov 23, 2020
- 5,930
- 1,698
Blocking the IP from public access won't prevent Google from detecting duplicate content. Google mainly checks for duplicate content via the URL and not based on IP addresses. If the content is identical on all servers, Google might still treat it as duplicate.
- Apr 23, 2012
- 2,876
- 5,898
There are two ways you can approach and resolve this
1. Setup Canonical Tags to tell search engines the actual URL of your page.
HTML:
<link rel="canonical" href="https://site.com/your-url" />You can read up here - https://developers.google.com/search/docs/crawling-indexing/canonicalization
2. You can use
ACL to route traffic on your replicas through your main domain - https://www.haproxy.com/documentation/haproxy-configuration-tutorials/core-concepts/acls/Something like (This is not tested, and I don't know how your existing cfg file is setup, so please test it properly)
Code:
# Setup allowed hostname
acl host_main hdr(host) -i site.com
# Redirect any non-matching hosts to main domain
http-request redirect code 301 location https://site.com%[capture.req.uri] unless host_main//
The third option would be (depending on how comfortable you are with HAproxy, allowing access to the replicas only via HAProxy)