The way that I understand it is, you have full control over who you link to. Therefore, if casino and other bad sites link to you, no harm done because you have no control over what sites link to you. But you do have control over who you choose to link to - and this can have a negative affect on your ranking.
Linking to one or two offtopic sites is usually just fine - it's possible you may just really like the sites and are supporting them. However, red flags get set off when you are caught linking to lots of off topic sites. When that happens, one of three things is going on. Either you are being paid to link to the sites, it's an off topic link swap, or you are running a free-for-all site. Either way, Google won't like your reason.
As far as linking to casino sites - I would avoid this. First of all, you don't want to be associated with what some consier the scum of the net. Second of all, casino sites, warez sites, and adult sites often use scammy SEO techniques. And you *will* get penalized for linking to a site that uses unethical SEO techniques such as page cloaking, etc.