You can't assess "efficiency" unless you have a reasonable list of the various ways in which the data needs to be accessed and a rough count of how often each of those will happen. Do you need to find the priority for a given operator? Do you need a list of alloperators with the same priority? How often? etc etc.
Write down the method signatures for the complete set of accessors that you would need to access your data. That's (a) the requirements statement for the data structure and (b) the starting point for your implementation.Discussions of efficiency in Java are almost always premature. The compiler and runtime are really good nowdays at optimising your code, and if you are using this data structure for processing stuff that you just read in from a file then it's the file i/o that swamps everything else. Unless you have some very good reason otherwize you should encapsulate all your data structures behind public accessor methods, then start with a data structure that is really obvious and clear as to its purpose and implementation. IF sometime later you determine there really is an efficiency problem you can change the data structure without changing the accessor methods and therefore without breaking anything else.
If you post the accessor signatures, and a quick note of which will be used the most, we can make sensible suggestions for an easy and clear way to implement them.