Firstly, don't sweat it. Google only shows you a sampling of the sites it knows about. Just because your backlink count dropped doesn't mean that Google doesn't recognize your link on those sites anymore.
Just keep tabs on your traffic and make sure that this isn't a trend that is affecting your rankings and traffic from the SERPS.
If a decrease in backlinks goes hand-in-hand with a decrease in traffic, you might want to shoot a letter over to Google via Google Webmaster Tools asking if you're banned or blacklisted for any reason and if there's anything you can do to correct it. Prove that you don't buy/sell links and are upfront and honest with them.
Speaking of selling links - you wrote you don't buy any, but do you sell any? Do you do link swaps on your site that might be confused with something that looks like or might look like links that were paid for? If so, that could be what's hurting you.
