Personally I would write this in C over Java. Only because Java is Object oriented and the problem you have posed isn't. Well sort of, but even more the fact that C will probably be much faster than Java for this project.


Also, this topic has been posted on daniweb before, so you might want to try to find the thread. I'm not sure how you plan to implement it, but in any case, be sure to consider something: does the file have to match exactly in order to be considered a match? Or will it be based on what % of the file matches? In either case, be sure to discontinue scanning the file at any point if its determined it doesn't match. You might want to consider using Tries. See Trie based cheat checker.

If there is already a class available that can search through all of the files on your System, why not make a Decorator class or a class that Maps files (or file names) to a number, and if the file is encountered again, replace the value with an incremented form of the same value, then return the map?

-Alex

thanks for the replys guys


but...

i dont want to use C - I'm not bothered about speed, loading or scanning duration of the intended application

and i find the post before mine abit confusing

If anyone here has information or a link to information that will help me incorporate a Md5 checksum to a GUI please let me know

Nothing complication because I want to be able to understand how it and specific code works!

also the application is stand alone well operating in a Java Runtime Environment

I am not sure you will find a specific article relating md5 to a java GUI, but it isn't really different to just md5 in a command-line environment. If you can get your checksums generating, just attach it to a GUI, by events or whatever your requirement may be.

I can't see a problem with writing this in Java. For the checksums, you could look at the MessageDigest class, or just use some other fairly strong hash function. MD5 isn't necessarily the best choice: it's designed to be a secure hash function, and trades performance for that security. IMO, what you need for duplicate recognition is just a "fairly strong" hash function: a good-quality 64-bit hash function should be ample. The advantage of MD5 is simply that you get it "out of the box". You could also try using a large buffer for your I/O and doing it in a separate thread to the hash calculation-- it may give a gain on some systems (multiprocessor and uniprocessor).

A very sliiight downside to writing this program in Java is that Java provides no way to signal to the operating system to read files without going via the file cache-- so you get slightly slower one-off reads than are in principle possible on most OSs.

> it's designed to be a secure hash function, and trades
> performance for that security

Maybe you got that the other way around since a lot of vulnerabilities or staged attacks have been found with MD5 making it a unsafe choice for security applications.

> a good-quality 64-bit hash function should be ample

Doesn't seem like a wise choice given that the MD5 algorithm which used 128 bit has been compromised.

IMO since many still sites still use the MD5 checksum to check the integrity of downloaded files, the situation with MD5 isn't as bad as it seems.

> in Java is that Java provides no way to signal to the operating
> system to read files without going via the file cache

File cache?

Well, that's also true, but as a design principle, MD5 (and other "secure" hash functions) trades performance for security.

As a secure hash function, it would be totally useless. But so what? What the poster wants to do is find duplicate files on a machine. Unless they're trying to guard against the attack whereby somebody deliberately plants a file on the machine to fool the file scanner, they don't actually need a secure hash function-- they just need a reasonable quality one. If they are trying to protect against that attack, then they need to weigh up the (at the moment minimal) security risk of MD5 vs other more secure-- but potentially more expensive and memory-consuming-- hashes.

Yep-- the OS will generally automatically cache data read from file. This is generally useful-- on average there's a reasonable chance you'll read enough of that data again to make it worthwhile. But in a few minority cases such as the file scanner (other examples might include, say, processing a database transaction log), you know in advance that you only read the data once, so it will overall have a negative impact on performance for the OS to cache that data. Generally, OS's provide a means to say "please read this data without caching". But Java does not expose that facility.

Re MD5: I guess we are talking about two different things here: security and file uniqueness, hence the confusion. I would personally use MD5 hashing since it seems to be a widely used technique for testing file uniqueness and optimize if and only if required. Also, coming up with a good hash solution [if that is what you were suggesting to the OP] is far from a walk in the park, though it seems to be a good exercise in learning more about hash functions.

> Yep-- the OS will generally automatically cache data read from file.

If you are talking about kernel file caching which results from reliance on system calls to do I/O, memory mapping the file solves that issue.