I agree that Fiverr sucks sometimes, though, there is not much you can do. Cover your losses and move on. If you are a content provider, I am definitely sure that you shouldn't be selling anything on Fiverr for $5. Instead, offer real value and let people pay good money for it somewhere else. In that case, you are only going to get real buyers paying good money.
If, however, you are not the one who can go outside of Fiverr and provide good content, then I am definitely sure why people are complaining about your gig.
My point: Quit Fiverr! They suck, and you need to bend over to buyers for $4. It is ridiculous! So, pretty simple, don't do it!