Is Java the wrong language for a duplicate file scanner?

[caps_lock]

So i've downloaded netbeans, and had a play around.

But I'm thinking is Java the wrong language to design a duplicate file scanner.

The program if not obvious should hopefully atleast identify any files that exist on a computer more than once (copies).

Im also going to try and make it available online, a bit similar to online virus scanning websites.

Ive got the website up in HTML, but im willing to scrap that if the duplicate file scanner cant be added to the website code as a plugin / object etc.

Is java the wrong language, is netbeans not the best IDE? Shall I consider AJAX

[BestJewSinceJC]

Personally I would write this in C over Java. Only because Java is Object oriented and the problem you have posed isn't. Well sort of, but even more the fact that C will probably be much faster than Java for this project.

Also, this topic has been posted on daniweb before, so you might want to try to find the thread. I'm not sure how you plan to implement it, but in any case, be sure to consider something: does the file have to match exactly in order to be considered a match? Or will it be based on what % of the file matches? In either case, be sure to discontinue scanning the file at any point if its determined it doesn't match. You might want to consider using Tries. See Trie based cheat checker.

[Alex Edwards]

If there is already a class available that can search through all of the files on your System, why not make a Decorator class or a class that Maps files (or file names) to a number, and if the file is encountered again, replace the value with an incremented form of the same value, then return the map?

-Alex

[caps_lock]

thanks for the replys guys

but...

i dont want to use C - I'm not bothered about speed, loading or scanning duration of the intended application

and i find the post before mine abit confusing

If anyone here has information or a link to information that will help me incorporate a Md5 checksum to a GUI please let me know

Nothing complication because I want to be able to understand how it and specific code works!

[caps_lock]

also the application is stand alone well operating in a Java Runtime Environment

[sillyboy]

I am not sure you will find a specific article relating md5 to a java GUI, but it isn't really different to just md5 in a command-line environment. If you can get your checksums generating, just attach it to a GUI, by events or whatever your requirement may be.

[neilcoffey]

I can't see a problem with writing this in Java. For the checksums, you could look at the MessageDigest class, or just use some other fairly strong hash function. MD5 isn't necessarily the best choice: it's designed to be a secure hash function, and trades performance for that security. IMO, what you need for duplicate recognition is just a "fairly strong" hash function: a good-quality 64-bit hash function should be ample. The advantage of MD5 is simply that you get it "out of the box". You could also try using a large buffer for your I/O and doing it in a separate thread to the hash calculation-- it may give a gain on some systems (multiprocessor and uniprocessor).

A very sliiight downside to writing this program in Java is that Java provides no way to signal to the operating system to read files without going via the file cache-- so you get slightly slower one-off reads than are in principle possible on most OSs.

[~s.o.s~]

> it's designed to be a secure hash function, and trades
> performance for that security

Maybe you got that the other way around since a lot of vulnerabilities or staged attacks have been found with MD5 making it a unsafe choice for security applications.

> a good-quality 64-bit hash function should be ample

Doesn't seem like a wise choice given that the MD5 algorithm which used 128 bit has been compromised.

IMO since many still sites still use the MD5 checksum to check the integrity of downloaded files, the situation with MD5 isn't as bad as it seems.

> in Java is that Java provides no way to signal to the operating
> system to read files without going via the file cache

File cache?

[neilcoffey]

> it's designed to be a secure hash function, and trades
> performance for that security

Maybe you got that the other way around since a lot of vulnerabilities or staged attacks have been found with MD5 making it a unsafe choice for security applications.

Well, that's also true, but as a design principle, MD5 (and other "secure" hash functions) trades performance for security.

> a good-quality 64-bit hash function should be ample

Doesn't seem like a wise choice given that the MD5 algorithm which used 128 bit has been compromised.

As a secure hash function, it would be totally useless. But so what? What the poster wants to do is find duplicate files on a machine. Unless they're trying to guard against the attack whereby somebody deliberately plants a file on the machine to fool the file scanner, they don't actually need a secure hash function-- they just need a reasonable quality one. If they are trying to protect against that attack, then they need to weigh up the (at the moment minimal) security risk of MD5 vs other more secure-- but potentially more expensive and memory-consuming-- hashes.

> in Java is that Java provides no way to signal to the operating
> system to read files without going via the file cache

File cache?

Yep-- the OS will generally automatically cache data read from file. This is generally useful-- on average there's a reasonable chance you'll read enough of that data again to make it worthwhile. But in a few minority cases such as the file scanner (other examples might include, say, processing a database transaction log), you know in advance that you only read the data once, so it will overall have a negative impact on performance for the OS to cache that data. Generally, OS's provide a means to say "please read this data without caching". But Java does not expose that facility.

[~s.o.s~]

Re MD5: I guess we are talking about two different things here: security and file uniqueness, hence the confusion. I would personally use MD5 hashing since it seems to be a widely used technique for testing file uniqueness and optimize if and only if required. Also, coming up with a good hash solution [if that is what you were suggesting to the OP] is far from a walk in the park, though it seems to be a good exercise in learning more about hash functions.

> Yep-- the OS will generally automatically cache data read from file.

If you are talking about kernel file caching which results from reliance on system calls to do I/O, memory mapping the file solves that issue.

[neilcoffey]

Re MD5: I guess we are talking about two different things here: security and file uniqueness, hence the confusion. I would personally use MD5 hashing since it seems to be a widely used technique for testing file uniqueness and optimize if and only if required.

And I'd say that's a fair enough way forward. The other advantage of MD5 is that Java gives it you "out of the box".

Also, coming up with a good hash solution [if that is what you were suggesting to the OP] is far from a walk in the park, though it seems to be a good exercise in learning more about hash functions.

No, I was thinking the poster could just use a standard one. Numerical Recipes suggests one that I think would work reasonably well.

> Yep-- the OS will generally automatically cache data read from file.

If you are talking about kernel file caching which results from reliance on system calls to do I/O, memory mapping the file solves that issue.

In my head, I suppose I was actually discounting mapping the file because in practice, Java has (or has had) some severe problems with this, such as not providing a wap to unmap a section of file once it is mapped, and imposing a less-than-expected limit on the total size of mapped file sections. However, if the poster can get things working with file mapping, I'd be very interested to know.

[jbennet]

Personally I would write this in C over Java. Only because Java is Object oriented and the problem you have posed isn't. Well sort of, but even more the fact that C will probably be much faster than Java for this project.

He wants it to be able to be run online in a web browser. Java can do that. C cant. He says

Im also going to try and make it available online, a bit similar to online virus scanning websites. Ive got the website up in HTML, but im willing to scrap that if the duplicate file scanner cant be added to the website code as a plugin / object etc. Is java the wrong language, is netbeans not the best IDE? Shall I consider AJAX