If you get your site ranked well without aff links on it and then place them on the site, IF having affiliate links on the site is an issue that Google doesn't like, then there is a good chance Google may dip your site in the rankings.
Is your site a WP site? If so let me know as I have something that can hide your affiliate links from Google so it won't affect your rankings.
Shez
Thanks for the replies.
Shez, yes it is a WP site.