These are some answers from Google Webmaster Forum Regarding the Topic.
I Hope It serves at least as a guide.
*
Answer by Google employee*Apart from SEO, I would look at it from a user's perspective. If your site is hosted in France and your users are mostly in France, then chances are they're going to have a great connection to your server (high speed, low latency). If your users are mostly in New Zealand and your server is hosted in France, chances are that their connection to your server isn't going to be the greatest. Google - and other search engines - assume that you thought about this and have located your server in the rough area where you expect your users to be, which is why we use the server location as an aid in determining the geotargeting of your website.
That said, we know this is not always possible and sometimes local hosting companies aren't up to the task of hosting a high-powered server with great internet connectivity. This is why we provide the option in Webmaster Tools for websites that use gTLD domain names (.com, .net, .org but also .eu and .asia).
So if you are using a ccTLD (like .fr) then you generally do not need to worry about the rest (outside of making sure that your users are happy. Because it's pretty clear where the users you are targeting are going to be. If you are using a gTLD, you can either use the server location or the setting in Webmaster Tools to give us this information.
*
Answered by a webmaster*Well, if you have a country specific domain name it is geo targeted anyway. If you have a TLD - set the geo target. The video says that if you are particulary concerned then you can move site to same country.
But I have never seen two sites in index and one beat the other solely due to server location as there are many factors that make a good website.
