a quick way I can think off the top of my head is to make an onchange function for the check boxes, that will take the particular check box being checked as a parameter. Each time it runs, it checks to see first if any boxes are checked. If not, recheck the box being unchecked, pop up an alert or whatever type of warning you would like to show.
If at least one is checked, we then count how many are checked. If more than 4, remove the new check and put up an alert.
Some of the language constructs you will need to read up on are: