There are several of Blu-ray player for Mac OS X, which one are you using? Do you think the one you are using is the best?

You need to convert your Blu-ray ISO files to 1080P MOV or M4V, MKV for further playback on iMac with QuickTime Player or VLC media player. Also you can have a try Macgo Mac Blu-ray Player, it can play Blu-ray directly without convertion loss. But you have to install a BR drive on your Mac. I hope it will help you
http://www.macblurayplayer.com/how-to-play-a-bluray-disc-on-mac.htm

I am using macgo mac bluray player. I also have used VLC player. Both are good.