Penalize probably not but user experience will be worse and your bounce rate will increase. On top of that, some websites that use Popups are flagged by Chrome with "This site shows intrusive or misleading ads.".
Unless you're running like a streaming/porn site, I'd avoid popups or at least just bind them to specific Elements on the site (eg. a video player).