I was about to post this same message. It opens automatically when I load a page, and that's neat, but closing it only lasts for thirty seconds or so. I thought it was a new message when it opened the second time. I guess it just wanted to be open.
I've seen the same thing. Sometimes I have to close the chat twice before it stays closed.
The close button closes out of the individual chat room that it is in. Most likely, you have multiple chat room alerts for some reason (i.e. someone posted something new to your shoutbox) which is why you would need to close out of it more than once. Essentially you're closing out of each new alert individually.