This is all speculation as I've never been in the situation to sell advertising on a site.
I probably would look at other sites in my niche and see how much they're selling ads for. Then I would find out their traffic (if they sale ads they are probably publishing traffic statistics). finally, I'd use that as a ratio for how much to sell my ads for.
Example: If Competitor A is selling ad space at $10 per month and they get 1000 impressions, then I know to sell my ads at about $1 per 100 impressions.
You can probably get an average of all the prices from all your other competitors and at least you'll know where to start your price at. If there's no one in your niche selling ads directly then perhaps you can get some ideas from adsense rates.