Should replica web servers of a load balanced server be blocked for public access?

[xpro]

I have a website that is load balanced across multiple web servers. Should the IP address/domain of the replica web servers be blocked to the public so Google does not treat them as duplicate content?

 

[Topiano]

I have a website that is load balanced across multiple web servers. Should the IP address/domain of the replica web servers be blocked to the public so Google does not treat them as duplicate content?

Can you clarify whether your primary concern is preventing search engines from indexing the replica servers, or are you also worried about users directly accessing those servers and bypassing the load balancer?

 

[xpro]

I'm not concerned about users accessing the replica servers. My main concern is that Google does not consider the content of replica servers as duplicate content. At this point I think it's safe to block public access to the replica servers right? Only the load balancer should be able to access it?

 

[BlogPro]

Depends on your existing load balancing setup.

Can you tell me the exact setup? Reverse Proxy? DNS Round Robin? Or something else?

Apart from your main domain, how are these replicas accessible via the open web? Sub domains? IPs?

 

[xpro]

Depends on your existing load balancing setup.

Can you tell me the exact setup? Reverse Proxy? DNS Round Robin? Or something else?

Apart from your main domain, how are these replicas accessible via the open web? Sub domains? IPs?

DNS is pointing to a single HAProxy server. HAProxy to a replica IP (replica 1) and a subdomain (replica 2) on a round robin fashion.

 

[NeedMyLink]

Blocking the IP from public access won't prevent Google from detecting duplicate content. Google mainly checks for duplicate content via the URL and not based on IP addresses. If the content is identical on all servers, Google might still treat it as duplicate.

 

[BlogPro]

DNS is pointing to a single HAProxy server. HAProxy to a replica IP (replica 1) and a subdomain (replica 2) on a round robin fashion.

There are two ways you can approach and resolve this

1. Setup Canonical Tags to tell search engines the actual URL of your page.

<link rel="canonical" href="https://site.com/your-url" />

You can read up here - https://developers.google.com/search/docs/crawling-indexing/canonicalization

2. You can use ACL to route traffic on your replicas through your main domain - https://www.haproxy.com/documentation/haproxy-configuration-tutorials/core-concepts/acls/

Something like (This is not tested, and I don't know how your existing cfg file is setup, so please test it properly)

# Setup allowed hostname
    acl host_main hdr(host) -i site.com
   
# Redirect any non-matching hosts to main domain
    http-request redirect code 301 location https://site.com%[capture.req.uri] unless host_main

//

The third option would be (depending on how comfortable you are with HAproxy, allowing access to the replicas only via HAProxy)