a quick way I can think off the top of my head is to make an onchange function for the check boxes, that will take the particular check box being checked as a parameter. Each time it runs, it checks to see first if any boxes are checked. If not, recheck the box being unchecked, pop up an alert or whatever type of warning you would like to show.
If at least one is checked, we then count how many are checked. If more than 4, remove the new check and put up an alert.
Some of the language constructs you will need to read up on are:
You will also need to know about how to check for greater than or less than, and the alert() function if you wish to use it.
Remember, you can reuse the same function for each onchange if you make it right ;)