If you're managing your own server, I wouldn't install a panel. However it is sometimes much easier to administer over a panel, in that case I would say setting up a good quality VPN would be your best bet and configuring your panel to only allow connections from localhost, that way you can use your system credentials to log in and work from the server itself.
Remember to set up SSH on a non standard port, and to use a long passphrase to access your account. If you have the ability, only allow those with absolute need to log in over SSH, root should never be allowed. Grant sudo privliges to those who need it and limit them where possible.
[edit] a panel may also not be where they are getting in, remember to firewall everything you don't need using iptables.