Member Avatar for tomz6

Are there any other good/reliable hashs which are smaller than MD5 and faster? (for purposes of checking for file corruption - not for security purposes) (preferably 16bit?)

Edited by tomz6
  • 2 Contributors
  • 2 Replies
  • 358 Views
  • 32 Minutes Discussion Span
  • Latest Post Latest Post by rubberman

Dani AI

Generated

For non-security file-integrity checks there are much better tradeoffs than a 16-bit digest. A 16-bit hash has only 65,536 values, so collisions become likely once you hit a few hundred files — roughly a 50% chance with ~300 items (birthday-paradox behavior). (britannica.com)

Practical, very fast options used for accidental-corruption detection:

  • CRC-32: ubiquitous in ZIP/PNG/gzip toolchains and available as a streaming API (zlib). Extremely fast for accidental errors but not cryptographically secure. Use only when an attacker is not a concern; 32 bits is fine for small collections but will collide if the file count grows large (see below). (zlib.net)

Modern non-cryptographic hashes give far better speed/size tradeoffs:

  • xxHash (XXH32/XXH64/XXH3) — designed for RAM-speed hashing and excellent dispersion; XXH64 or XXH3 are the usual recommendations for low collision risk at high throughput.
  • If cryptographic assurance is also required, BLAKE3 is a modern, very fast cryptographic hash (parallelizable), but it’s overkill if only accidental-corruption detection is needed. (github.com)

Implementation notes and a template workflow:

  • Stream the file in fixed-size blocks and update the hash state so large files never need to be fully buffered.
  • Pick 32-bit only if storage is constrained and the total number of distinct files is small; otherwise prefer 64-bit (or larger) digests.
  • As observed, MD5 is heavier than these non-crypto options; as asked by , 16-bit is generally unsafe for file collections larger than a few hundred items.

Example pseudocode (replace Hash_* calls with the chosen library API):

#include <fstream>
std::ifstream in("file.bin", std::ios::binary);
HashState h = Hash_init();            // e.g., xxh state or crc init
char buf[64*1024];
while (in.read(buf, sizeof(buf)) || in.gcount()) {
  Hash_update(h, buf, (size_t)in.gcount());
}
auto digest = Hash_finalize(h);

Summary: for speed and practical collision resistance use XXH64/XXH3 (or CRC-32 for tiny deployments); avoid 16-bit except for toy cases. (github.com)

Recommended Answers

All 2 Replies

Member Avatar for rubberman

MD5 is popular because it can be implemented small and fast. Better ones are SHA256 and SHA512 hashes, which can be implemented pretty efficiently in todays' hardware. In any case, adding a salt value to permute the hash of a string is always a good idea from the security perspective.

So, for a simple file checksum, md5 is very good, and will very rarely generate a collision (if ever), but for speed, the old cksum tool is much faster for big files (like video files of 1+GB size).

FWIW, md5 check sums are 128 bits in size, sha256 are (duh) 256 bits, and sha512 are 512 bits.

Member Avatar for rubberman

In case you are enterested, my organization uses sha256 to encrypt user identifiable browsing information in our cell phone proxy browsers. These are one-way hashes, and make recovering data about a specific user very difficult. We are considering moving to a sha512+salt hash to make them pretty much un-encryptable, even with modern brute-force tools and tonnes of processors.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.