Re MD5: I guess we are talking about two different things here: security and file uniqueness, hence the confusion. I would personally use MD5 hashing since it seems to be a widely used technique for testing file uniqueness and optimize if and only if required.

Also, coming up with a good hash solution [if that is what you were suggesting to the OP] is far from a walk in the park, though it seems to be a good exercise in learning more about hash functions.

If you are talking about kernel file caching which results from reliance on system calls to do I/O, memory mapping the file solves that issue.

Personally I would write this in C over Java. Only because Java is Object oriented and the problem you have posed isn't. Well sort of, but even more the fact that C will probably be much faster than Java for this project.