I couldwrite a customize JButton in java
What I mean by a customize JButton is that I overridden some of it's methods, such as paintBorder, paintComponent.
I could write a round,triangle or any other shaped JButton.
I wonder how to write a different shaped JFrame, such as round or triangle JFrame.
THanks for helping me

I don't like to speculate on what I "would" do (since I've never tried anything like this, so the advice could be totally off - try at own risk of wasting your time :) ), but I would try to make a custom class that extends JButton and modify the paint method. The Graphics class has methods to draw shapes such as ovals, rectangles, etc.

Java only supports rectangular shapes for the parent window(ie the one that sits in the desktop). You can simulate other shapes by making part of the window transparent, leaving just (say) a circle visible. It's not that easy, but this reference will get you started...