Proxy chaining

I have a setup that’s need some upgrade. At the moment each server (Cloud VPS ) is connected to proxy provider via whitelisted IPs. Problem is that my account has limited amount of slots for IPs I can whitelist which is way lower than amount of servers I require.

Possible solution is to put software based router/firewall (pfSense or similar) and route and dnat/snat entire traffic from/to VPSs or to setup intermediary proxy like Dante or Squid to circumvent this limitation.

I would like to hear from experienced individuals here what will be the best route to take in such case. I welcome all ideas.

Note: everything is public cloud based so solution must be virtualised

Alex

Bump!

The scenario you described is a common one faced by those needing to scale beyond IP whitelisting limits imposed by proxy providers. Here are the two solutions you proposed, elaborated upon, along with their advantages and potential issues:

1. Software-based Router/Firewall (e.g., pfSense):
   • Advantages:
     • Centralized Management: All of your traffic management, routing, and NATing would be in one place. This can simplify the configuration and make it easier to maintain.
     • Flexibility: pfSense, in particular, offers a broad range of features, from VPNs to intrusion detection systems, which might be helpful depending on your setup's complexity.
     • Transparent: For the originating servers, this is a transparent solution. They don't need to know about the proxy; they just send their traffic as usual.
   • Potential Issues:
     • Single Point of Failure: If your pfSense (or other software-based router) instance goes down, all of your servers might lose their connectivity.
     • Performance Overhead: Depending on your traffic volume, the software-based router might become a bottleneck. Make sure to provision enough resources (CPU, RAM) and choose a cloud provider that offers good network performance.
2. Intermediary Proxy Server (e.g., Squid, Dante):
   • Advantages:
     • Caching: If you're routing web traffic and the content doesn't change frequently, a proxy server like Squid can cache content, which can save bandwidth and improve response times.
     • Fine-Grained Control: You can have more nuanced control over which traffic gets proxied and which doesn't.
     • Authentication & Logging: Most proxy servers offer detailed logging, which can be beneficial for debugging or monitoring. They can also support various authentication methods if that's a requirement.
   • Potential Issues:
     • Complexity: Managing a proxy server can be a bit more complex, especially if you're dealing with SSL/TLS traffic. Encrypted traffic may need special handling or might not be cacheable.
     • Server Configuration Required: Unlike the transparent router solution, each server would need to be configured to use this proxy.

Recommendation:

Both solutions have their merits, and the best choice largely depends on your specific requirements:

• If you want a transparent solution that doesn't require reconfiguring each server, and you value the added flexibility of a full-featured firewall, then a software-based router might be your best bet.
• If you want to potentially benefit from caching, or you need fine-grained control over the traffic, then an intermediary proxy server would be more suitable.

For maximum availability, regardless of the chosen solution, consider setting up a high-availability cluster (e.g., a CARP cluster if you're using pfSense) to minimize the chances of a single point of failure.

Remember to continuously monitor the performance and health of whichever solution you choose, and periodically review your setup to adjust for changes in your traffic patterns or requirements.

turelink said:

The scenario you described is a common one faced by those needing to scale beyond IP whitelisting limits imposed by proxy providers. Here are the two solutions you proposed, elaborated upon, along with their advantages and potential issues:

1. Software-based Router/Firewall (e.g., pfSense):
   • Advantages:
     • Centralized Management: All of your traffic management, routing, and NATing would be in one place. This can simplify the configuration and make it easier to maintain.
     • Flexibility: pfSense, in particular, offers a broad range of features, from VPNs to intrusion detection systems, which might be helpful depending on your setup's complexity.
     • Transparent: For the originating servers, this is a transparent solution. They don't need to know about the proxy; they just send their traffic as usual.
   • Potential Issues:
     • Single Point of Failure: If your pfSense (or other software-based router) instance goes down, all of your servers might lose their connectivity.
     • Performance Overhead: Depending on your traffic volume, the software-based router might become a bottleneck. Make sure to provision enough resources (CPU, RAM) and choose a cloud provider that offers good network performance.
2. Intermediary Proxy Server (e.g., Squid, Dante):
   • Advantages:
     • Caching: If you're routing web traffic and the content doesn't change frequently, a proxy server like Squid can cache content, which can save bandwidth and improve response times.
     • Fine-Grained Control: You can have more nuanced control over which traffic gets proxied and which doesn't.
     • Authentication & Logging: Most proxy servers offer detailed logging, which can be beneficial for debugging or monitoring. They can also support various authentication methods if that's a requirement.
   • Potential Issues:
     • Complexity: Managing a proxy server can be a bit more complex, especially if you're dealing with SSL/TLS traffic. Encrypted traffic may need special handling or might not be cacheable.
     • Server Configuration Required: Unlike the transparent router solution, each server would need to be configured to use this proxy.

Recommendation:

Both solutions have their merits, and the best choice largely depends on your specific requirements:

• If you want a transparent solution that doesn't require reconfiguring each server, and you value the added flexibility of a full-featured firewall, then a software-based router might be your best bet.
• If you want to potentially benefit from caching, or you need fine-grained control over the traffic, then an intermediary proxy server would be more suitable.

For maximum availability, regardless of the chosen solution, consider setting up a high-availability cluster (e.g., a CARP cluster if you're using pfSense) to minimize the chances of a single point of failure.

Remember to continuously monitor the performance and health of whichever solution you choose, and periodically review your setup to adjust for changes in your traffic patterns or requirements.

Click to expand...

Thanks for explanation as this is going to be helpful to everybody.

Like i said. Sole purpose is to "mask" bunch of servers behind one whitelisted IP I've got from proxy provider. Mine main problem here is that i would like to set it and forget it because it won't be touched anytime soon and no person in my team is skilled enough to "manage" it not to mention setup/config. Traffic is HTTP/HTTPS.

pFsense is best option for this IMHO but it is too complicated for someone without routing experience (at least) and i would like to avoid any future problems if someone from my team misconfigures it sometime down the line, while Dante is simple enough to configure and maintain it /recover it in case of issues.


How would you go about with setup of Dante in this case? I am thinking of 2 LAN interfaces - one for incoming connections from other servers and one for outgoing connections to upstream proxy.

Small update if anyone else will be looking for this. After several experiments I decided to implement Tinyproxy mainly because of its simplicity (config wise). Clients can be added by simple "Allow" directives while "Upstream" directive covers chaining in most cases.

Use a software-based router/firewall, such as pfSense. pfSense can be installed on a virtual machine and configured to route all of the traffic from your VPS's to the proxy provider.

turelink said:

The scenario you described is a common one faced by those needing to scale beyond IP whitelisting limits imposed by proxy providers. Here are the two solutions you proposed, elaborated upon, along with their advantages and potential issues:

1. Software-based Router/Firewall (e.g., pfSense):
   • Advantages:
     • Centralized Management: All of your traffic management, routing, and NATing would be in one place. This can simplify the configuration and make it easier to maintain.
     • Flexibility: pfSense, in particular, offers a broad range of features, from VPNs to intrusion detection systems, which might be helpful depending on your setup's complexity.
     • Transparent: For the originating servers, this is a transparent solution. They don't need to know about the proxy; they just send their traffic as usual.
   • Potential Issues:
     • Single Point of Failure: If your pfSense (or other software-based router) instance goes down, all of your servers might lose their connectivity.
     • Performance Overhead: Depending on your traffic volume, the software-based router might become a bottleneck. Make sure to provision enough resources (CPU, RAM) and choose a cloud provider that offers good network performance.
2. Intermediary Proxy Server (e.g., Squid, Dante):
   • Advantages:
     • Caching: If you're routing web traffic and the content doesn't change frequently, a proxy server like Squid can cache content, which can save bandwidth and improve response times.
     • Fine-Grained Control: You can have more nuanced control over which traffic gets proxied and which doesn't.
     • Authentication & Logging: Most proxy servers offer detailed logging, which can be beneficial for debugging or monitoring. They can also support various authentication methods if that's a requirement.
   • Potential Issues:
     • Complexity: Managing a proxy server can be a bit more complex, especially if you're dealing with SSL/TLS traffic. Encrypted traffic may need special handling or might not be cacheable.
     • Server Configuration Required: Unlike the transparent router solution, each server would need to be configured to use this proxy.

Recommendation:

Both solutions have their merits, and the best choice largely depends on your specific requirements:

• If you want a transparent solution that doesn't require reconfiguring each server, and you value the added flexibility of a full-featured firewall, then a software-based router might be your best bet.
• If you want to potentially benefit from caching, or you need fine-grained control over the traffic, then an intermediary proxy server would be more suitable.

For maximum availability, regardless of the chosen solution, consider setting up a high-availability cluster (e.g., a CARP cluster if you're using pfSense) to minimize the chances of a single point of failure.

Remember to continuously monitor the performance and health of whichever solution you choose, and periodically review your setup to adjust for changes in your traffic patterns or requirements.

Click to expand...

Great Explanation