My personal belief is that a site will not suffer if every so often a bad neighborhood links to them, especially if the number of "bad" links is far outweighed by the number of high quality backlinks. As a site becomes more well known and mainstream, it can only be expected that a wide array of quality and non-quality sites link to them.

However, if only bad neighborhoods link to a particular site, or the majority of backlinks a site has are from hundreds of free-for-all link pages, then it could be assumed by the SEs that this site travels in the wrong circles.