so you recommend 1 network?If they are different ad products like (first ad network display ads only and second only showing video ads etc.), you should not get much negative effect.
But if you use them for same/similar ad types, it is going to slow your website down while your earnings probably won't raise up because most of the ad networks's ads are coming from the same source; Google.
Try to choose the one with a better revshare and serving better to your niche with good ad types.