I can't figure this out!! I am using removeMouseListener right after a click is done (provided its the right person's turn)
I would say that this is a logical error. It might work if done properly, but in my opinion, the proper way to do this would be to keep both client and server aware of whose turn it is at all times. If the client clicks when it isn't their turn, don't tell the server about it. If the server clicks when it isn't their turn, don't tell the client about it. And on both client and server, if it is their turn and the other player sends them "click coordinates", then ignore it, but throw an Exception (or display an error message) because it is an error. (Example: if it's the server's turn but the client sends them click coordinates, then it's obviously an error). So what I'm saying is that you shouldn't remove the mouse listeners - instead, implement "talking about whose turn it is" between client and server, and do so such that they always know whose turn it is.
With that said, I'll try to help you debug the code you currently have, but I can't promise that I'll be able to fix whatever it is.