Swing likes to give itself a lot of leeway to monkey with your layout. On the one hand, it tries to do a good job of making things look right, on the other hand, it takes your setting as "preferences" a lot of the time, even if you use the "setSize()" method instead of setPreferredSize()".
You might look at the layout manager you're using. Some - GridLayout, for example - are much more prone to resize components, while the more free-form (BorderLayout) will move components around rather than resize them. I only know this because I just spent a few hours on a layout, and finally realized that this was what was tripping me up.
If you really need a tight, rigid layout, the best way to get it might be to nest panels until it all works out for you, but that takes a lot of work.
This doesn't work on a JPanel, but you can set the enclosing window (JFrame or JDialog, etc.) to be not resizable. You work really hard getting the layout just right, and then you don't want the user messing it up by resizing the window. Just call setResizable(false); on the enclosing window.