I suppose the reason no one answered this yet is simply because it is much too vague a question. Advertising sales depend on oh so many things, with one of the most important, as you might suspect, being how in demand advertising on your site is.
When you say standard ad rotator, I am assuming you mean phpAdsNew. If so, you should enable their "zones" feature which will allow you to target specific forums on your site. You can definitely raise prices if you allow an advertiser to target just a specific section of your site.
Additionally, keep in mind that forums notoriously perform the worst of nearly every other content site on the web. This is because forum regulars nearly always ignore the advertisements. Therefore, you can expect to only receive the majority of the ad clicks from guests and first-time visitors. If you want to raise the overall click thru rate (CTR) of the ad campaigns you have running, you can opt to show them only to guests.
It is very important, especially on forums, to enable frequency capping for your visitors. In other words, don't show the same ad to the same person more than a couple times per day. 3 per 24 hrs or 4 per 24 hrs is usually industry standard. Doing so will raise the click thru rate (the performance) of ads for two reasons. The first is because advertisers won't constantly waste impressions on people who aren't interested. (If you haven't clicked on an ad the first 20 times you've seen it, odds are you are never going to.) The second reason is because advertisers won't constantly waste impressions on regular members who could spend hours on the site in a single session.
Now ... how much to charge. There is only one rule of thumb - as much as you can while still taking in orders :) It's completely experimentation. The average site charges anywhere from $0.50 CPM to $100.00 CPM. The best thing you can do is start at a price around $5 CPM and see if you get any takers. If you do, gradually raise the price until new sales start to slow down too much. If you don't, try running a couple of 25% off sales until you find a price that works best.
In doing so, here are some things to consider -
Do you have a prominent link on your site to an advertising Media Kit with clearly labeled contact information?
Are you getting as many leads as you would expect?
Is your audience very targeted?
Can you target campaigns to specific sections of your site?
Perhaps most importantly, what is the CTR like? How do the ads perform?
What are your demographics like?
The typical banner CTR can range between 0.1% and 5%. As mentioned before, forums will usually average on the lower end of that. Play around with ad placements. Try to keep them above the fold (visible without any window scrolling).
Demographics are also very important. Sure, it's great to target, but think about who you are targeting. Are you extremely targeted towards 17 year old gamers interested in a particular XBOX game? Or are you extremely targeted towards 35 year old corporate IT executives interested in purchasing expensive computer equipment for their companies?
Think about what an ROI (return on investment) is for the advertiser. Factor in your CPM rate (cost per thousand impressions) and CTR, and figure out how much money it averages out to be per click for the advertiser. How much money would an advertiser potentially make by advertising with you? If an advertiser sells a product for $50 and your CTR is so low that it ends up he is paying an average of $30 per click, then what are the chances he would advertise with you again?
Keep in mind that banner advertising is an important branding aspect to advertisers, so it has much more added value over pay per click advertising such as AdWords.