How so? If you make a decent-looking site and your content is good, I don't see any plausible reason for a decline. I watch plenty of social media pages doing it all the time, and draw inspiration for my own marketing tactics by watching how they perform.
Also, moderation is key.
If you're not blasting clickbait all day, every day, I don't really see why people would unfollow, and that aside, there's always plenty of others to take their place. Do what you do normally, but sprinkle your website link in there, so followers that enjoy your current posting format can continue to do so, and those dumb/adventurous enough to check out your website will do so, making you a little more money.
Worth testing, IMO. If you look at any and all meme websites, they all started from somewhere, and there's a good chance they leveraged social media to get to where they are today. Test and twist, you're bound to find gold somewhere, somehow.
Food for thought, if nothing else.