Looking for advice on DDoS mitigation for a small indie MMORPG

ll4wliet

Newbie
May 24, 2024
1
0
Hello everyone (sorry if this is the wrong category, but this seemed the only place to ask for advice),

My team runs a small indie MMORPG (around 1k players online at a time). We have been experiencing a barrage of DDOS attacks and network stability issues for the past 2 months. I would like to preface that my experience in networking is quite limited. I am looking for some advice to gain better insight into the overall traffic going through our server, ways to identify the type(s) of DDoS attacks leveraged against us, and possibly ways to mitigate them.

Let me outline our journey so far.

  1. OVH hosting
    We initially hosted our server at http://ovhcloud.com/en/, they claim to have great DDOS protection. However, their protection https://github.com/ovh/infrastructure-roadmap/issues/173.
  2. OVH + Cloudflare reverse proxy
    Our next idea was to use a https://www.cloudflare.com/en-gb/learning/cdn/glossary/reverse-proxy/. We got a new dedicated IP from OVH, and pointed it to our domain name in Cloudflare with proxying enabled. Players would now connect to our domain name and their traffic would be filtered by Cloudflare and then rerouted to our server. This seemed to stop the DDOS attacks but sporadically OVH's anti-DDOS protection would kick in and start https://lowendtalk.com/discussion/179935/ovh-ddos-mitigation-522-cloudflare. So that did not work either.
  3. OVH + HAProxy + Fly.io
    Next, we figured that maybe the issue with Cloudflare was that all of our traffic was now being tunneled through too few IPs (i.e. 1000 users worth of traffic coming from only 5 distinct IPs) and this might set off the OVH Edge firewall.
    So, we decided to implement our load-balancing solution using http://fly.io, which let us deploy VMs all over the world with easy scaling, and https://www.haproxy.org.
    However, this approach faced the same issue as the Cloudflare reverse proxy, with OVH's Edge firewall blocking the traffic.
  4. Tempest hosting (Path.net DDOS protection), the savior?
    OVH Customer support has been both slow to reply and overall unhelpful. So we decide to look at other hosting providers, specifically one with great DDOS protection. Here comes https://www.tempest.net, who own https://path.net (one of the largest L3-L7 DDOS mitigation platforms). We migrated over our services and all seemed good, the attackers were unable to attack us for some time.
  5. Tempest + Firewall (filter and ratelimiting)
    A week has passed since our migration and we are yet again under siege. We contacted Tempest customer support and they were very quick to reply and helped us configure our firewall, setting a filter and rate-limiting rules.
    This stopped our server from going down completely when under attack but network stability issues remain.
  6. Where are we at now?
    Sporadically (every 1-3 days, sometimes more frequently) a large chunk of our player base gets disconnected from the game (around 200-300), which we suspect is due to attacks.
    Furthermore, their network seems unstable in general, with individual players getting disconnected throughout the day. Sometimes the affected players would experience extremely high ping leading up to a disconnect, sometimes without notice their connection would just be dropped, and often once they got disconnected, the server would time out their future requests for the next 3-10 minutes.
    It has been a wild journey and both our team and player base are exhausted dealing with this.

This brings me to the main purpose of this post, a plea for help, any advice would be much appreciated. There are two main points of interest I am looking to get advice on:

Network monitoring solutions

We want to be able to gain more insight into the traffic going through our server. Both to improve our team's understanding and to provide our hosting provider with useful data to better assist us.
Since we cannot predict when exactly an attack will happen, and since the attacks themselves are very short-lived (< 1 minute), we want to maintain historical packet dumps for at least the past 12 hours of traffic.

We are looking into a few options:
  • tcpdump + cronjob
  • ntopng
    We also stumbled upon http://ntop.org which provides a very nice web interface for inspecting incoming traffic, but this seems mainly aimed at real-time monitoring, with historical data capture requiring additional licenses that we cannot currently afford. If there is a similar cheap/free service that provides an out-of-the-box monitoring and analysis solution, please do post a reply.

Additional mitigation solutions

We would like to do as much as we can on our end to reduce attack vectors and/or mitigate ongoing attacks. However, we are not sure what kind of DDOS attack is being employed against us (at what level it occurs, what method it uses, etc..), so we are unsure where to even start with this.

Currently, we have done the following:
  1. Configured rules: closing all ports except for the one our game service listens on.
  2. Configured a filter: max of 200 packets per second per connection allowed for the port mentioned above.
  3. Configured a ratelimiter: mac of 500 packets per second

We also looked into https://www.ntop.org/products/ddos-mitigation/nscrub/ as this seemed quite noob-friendly to implement as a bump in the wire (transparent bridge) DDoS mitigation system, though this seems more so aimed to be deployed at the level of a hosting provider. Since our hosting provider (tempest.net) already has their own mitigation platform (path.net), we are not sure this would provide us any benefit at all, i.e. once the traffic passes Path and enters our server, is it too late for us to filter it? Additionally, we cannot afford to spend money on license costs for nScrub unless we are sure it will provide us a benefit.

Are there other things we can do on our machine, or are we limited to tempest customer support to configure Path for our specific service?
 
Our next idea was to use a https://www.cloudflare.com/en-gb/learning/cdn/glossary/reverse-proxy/. We got a new dedicated IP from OVH, and pointed it to our domain name in Cloudflare with proxying enabled. Players would now connect to our domain name and their traffic would be filtered by Cloudflare and then rerouted to our server. This seemed to stop the DDOS attacks but sporadically OVH's anti-DDOS protection would kick in and start https://lowendtalk.com/discussion/179935/ovh-ddos-mitigation-522-cloudflare. So that did not work either.
If this works then move from OVH to some other host and simply apply it back
 
Any Path DDoS Protection provider will work for you.
You can also check Voxility protection, Voxility is not better than Path but does the job for Layer 3 and Layer 4.

For Voxility Protection you should use:
flokinet.is
alexhost.com
ginernet.net (Path + Voxility)
Javapipe.com
 
Hello,

I've been doing my own similar research and over the years have had tried similar things + same providers. I'm not sure if my advice will be of any help if the big providers path.net are going down.

What I can ask you is if you've got the IP's setup with Anycast? You may want to look into BGP setup where you have more control over the IP space you have. Also you've mentioned ntopng, this is new to me. I personally use PRTG for monitoring my network. There is other software on the market for ddos protection/monitoring BGP netflow/flowspec & stopping the attack.
 
Back
Top
AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features and essential functions on BlackHatWorld and other forums. These functions are unrelated to ads, such as internal links and images. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock