Server location doesn't matter (much).
Speed is important for SEO (at least Google), but that doesn't mean a few ms of difference would be much.
However, having a faster speed of loading is proven to be increasing converting rate. Thus, if you focus on that then server location should be put in...