Help! Bing/msn bot using huge load of my server

kurosaki4d · Oct 28, 2019

I am facing a big problem with my server. I have a website that keeps getting massive page requests coming from "Bing/Msn" bot every second or two and the ip changes now and then. Which is putting a heavy load on my server.

My CPU is constantly over 90%

I tried to block the bot from htaccess and robots.txt but they don't seem to have any effect.

If anyone has an idea how to defeat this it would be much appreciated.

Thank you in advance

TomTheCat · Oct 28, 2019

If you have access to root, try iptables and block the entire class. But why the heck will you block Bing? If I were you I will fix the issues. First, I will get rid of the IDS you have in place, it sucks, and I think that overloads your server. If you have an expired domain, Bing looks up the old pages.

kurosaki4d · Oct 28, 2019

TomTheCat said:
If you have access to root, try iptables and block the entire class. But why the heck will you block Bing? If I were you I will fix the issues. First, I will get rid of the IDS you have in place, it sucks, and I think that overloads your server. If you have an expired domain, Bing looks up the old pages.

Thank you for your reply !

Regarding "iptables" i can't use it since the blocks keeps on changing, they are very random. In the picture it shows the same ip but after like a minute or so the ip changes the all blocks. So i'm not sure if iptables will work in this case.

If i can find the real issue instead of blocking Bing that would be even better, you are absolutely right.

But would you care to clarify more on what you meant by getting rid of my "Intrusion detection system". I only have Wordfence, what else can i work on ?

The domain that keeps getting hit doesnt expire until late 2020 if that's what you meant ! As for the old pages, i'm not sure those ever existed, if you look closely the names indicate that !

TomTheCat · Oct 28, 2019

kurosaki4d said:
Thank you for your reply !

Regarding "iptables" i can't use it since the blocks keeps on changing, they are very random. In the picture it shows the same ip but after like a minute or so the ip changes the all blocks. So i'm not sure if iptables will work in this case.

If i can find the real issue instead of blocking Bing that would be even better, you are absolutely right.

But would you care to clarify more on what you meant by getting rid of my "Intrusion detection system". I only have Wordfence, what else can i work on ?

The domain that keeps getting hit doesnt expire until late 2020 if that's what you meant ! As for the old pages, i'm not sure those ever existed, if you look closely the names indicate that !

Code:

NetRange:       40.74.0.0 - 40.125.127.255
CIDR:           40.112.0.0/13, 40.124.0.0/16, 40.125.0.0/17, 40.120.0.0/14, 40.76.0.0/14, 40.96.0.0/12, 40.80.0.0/12, 40.74.0.0/15
NetName:        MSFT
NetHandle:      NET-40-74-0-0-1
Parent:         NET40 (NET-40-0-0-0-0)
NetType:        Direct Assignment
OriginAS:
Organization:   Microsoft Corporation (MSFT)
RegDate:        2015-02-23
Updated:        2015-05-27
Ref:            https://rdap.arin.net/registry/ip/40.74.0.0

You have to block all IP classes from CIDR. Since you have a lot of traffic, Wordfence or what you have in place there checks each visit by user-agent as far as I can see (the iOS version wasn't blocked) and this overloads the server. You can try and disable it for a few hours and see if the problem persists. The old pages somehow existed. If the domain was yours in the past, you were hit by some virus that created the pages on the server.

kurosaki4d · Oct 28, 2019

Thank you for these valuable informations mate. What you say makes a lot of sence, the domain might have belonged to someone else before who did some bad things with it !

Anyway, i disabled Wordfence and i would like to know how can i block all IP classes from CIDR ? Because i did search and i couldn't find much on how to do so ?

Do i generate a list of all the ip addresses of that range you gave me and then blacklist them all ? Is that the right way ?

TomTheCat · Oct 28, 2019

kurosaki4d said:
Thank you for these valuable informations mate. What you say makes a lot of sence, the domain might have belonged to someone else before who did some bad things with it !

Anyway, i disabled Wordfence and i would like to know how can i block all IP classes from CIDR ? Because i did search and i couldn't find much on how to do so ?

Do i generate a list of all the ip addresses of that range you gave me and then blacklist them all ? Is that the right way ?

Code:

iptables -A INPUT -p tcp -s <IP CLASS TO BLOCK> -i ens2f0 --sport 80:65535 -m state --state NEW,ESTABLISHED -j DROP
iptables -A OUTPUT -p tcp -s <IP OF YOUR WEBSITE BEING HIT> -d <IP CLASS TO BLOCK> --sport 80:65535 --dport 513:65535 -m state --state ESTABLISHED -j ACCEPT

Replace <IP CLASS TO BLOCK> with 40.112.0.0/13, and <IP OF YOUR WEBSITE BEING HIT> with your website IP address. Also, replace ens2f0 with your network card name. Copy each line in SSH and hit enter. Do the same for the other IP classes listed on CIDR. Also, you have to save the iptables rules, if you plan to restart the server in the future, otherwise the rules will vanish.

kurosaki4d · Nov 5, 2019

Thank you for your help, that's good to know, i'm sure it might come up handy someday !

For now what i did is i reduced the crawl rate of Bing and Google to the absolute minimum from "Search console" and i added the crawl rate also in the robots.txt file, plus i added wordfence to block any malicious bots that pass on as google and to block it if it exceeds a certain amount of requests within 1 minute.

That gave me a great result and relieved the extra pressure from the server.

Thank you !

Help! Bing/msn bot using huge load of my server

kurosaki4d

Power Member

TomTheCat

Elite Member

kurosaki4d

Power Member

TomTheCat

Elite Member

kurosaki4d

Power Member

TomTheCat

Elite Member

kurosaki4d

Power Member

Main Menu

Marketplace

Making Money

BlackHat World