That's EXACTLY when to buy gold. When it goes down, not the other way around. Would you rather buy cheaper gold, or more expensive gold?
I say try to get as much gold/silver/properties as you , even if they lose value, they lose value in proportion to the value of the money you used to buy it...