The way that I understand it is, you have full control over who you link to. Therefore, if casino and other bad sites link to you, no harm done because you have no control over what sites link to you. But you do have control over who you choose to link to - and this can have a negative affect on your ranking.
Linking to one or two offtopic sites is usually just fine - it's possible you may just really like the sites and are supporting them. However, red flags get set off when you are caught linking to lots of off topic sites. When that happens, one of three things is going on. Either you are being paid to link to the sites, it's an off topic link swap, or you are running a free-for-all site. Either way, Google won't like your reason.
As far as linking to casino sites - I would avoid this. First of all, you don't want to be associated with what some consier the scum of the net. Second of all, casino sites, warez sites, and adult sites often use scammy SEO techniques. And you *will* get penalized for linking to a site that uses unethical SEO techniques such as page cloaking, etc.
very few SEO techniques are unethical... the ethics tag is marketing speak for the search engines.
search engine manager: "crap that stuf undermines our algorithm and make our ad business looked overpriced. what should we do?"
another search engine manager: "frame it as being assoicated with email spam and throw the ethics tag on it"
some stuff that is promoted is rubbish...but most seo techniques in an of themselves have nothing to do with ethics, but are better looked at from a risk vs reward perspective