of course the very act of checking an object to see if it's in the garbage collection queue may well remove it from the queue, and is likely to itself generate objects that make it on to the queue the moment your check ends.
Therefore the check is useless. You have a Schroedinger's cat situation here, you can't check the queue without changing the queue.
How does visualvm get it's info (eg number of instances awaiting finalisation)? Maybe there's no way from within the language, but what kind of instrumentation interface does visualvm hook into?
That's a good observation.
First to clarify: the number of objects pending finalization isn't really the same as the number of objects eligible for garbage collection. The former means that the objects are already identified as part of garbage (or simply put have been GC'ed) and have been put in the finalizer queue by the GC. The latter asks the question "when the next GC cycle runs (minor or major), which objects would be picked up for GC".
Secondly, it's easy to get the number of objects pending finalization; we already have a MBean for it. There is a catch though; that count is only for objects which have a non-trivial finalize method i.e. is only applicable for classes which override the finalize method. Since the OP (I believe) has no control over the objects, this count would not reflect objects of custom classes which don't override finalize. So I would say that count is not a very good measure in this particular case.
and of course, one should never rely on finalize() being called at all, so in production code its very existence if often a clear sign of design flaws or at least a serious red flag as to the correctness of the code.
As the execution of finalize() is not guaranteed on any instance, any code in there is by definition unreliable...