Member Avatar for johndoe444

Hello,

I wanted to see whether 2 directories are identical in terms of content. So what I did was tar the directories and then md5sum-ed them and then compare but the hashes were not identical. I tried to go inside each directory and md5sum-ed each file of each directory and compared individually and then each file in the different directory seemed to have the same hash. Then I checked with the tars and then tars of different directories didn't match (by using diff). So now that is causing md5sum to not match. But then why would that happen? So now which one is wrong - tar or md5sum?

Thanks.

  • 2 Contributors
  • 2 Replies
  • 138 Views
  • 2 Weeks Discussion Span
  • Latest Post Latest Post by johndoe444

Dani AI

Generated

Short answer: neither tool is wrong. A tar archive is a byte stream that contains file payloads plus headers and ordering; two archives can contain identical file data but differ in header fields or entry order, so their MD5s differ even though the per-file MD5s inside each directory match. was on the right track.

Typical causes for bitwise differences: directory traversal order when tar was created, file mtimes stored in headers, owner/group (UID/GID or names), permission bits, symlink targets, device node numbers, sparse-file handling, extended attributes/ACLs/SELinux contexts recorded in PAX headers, and differences in tar format or compression (gzip stores a timestamp unless you use -n). Any of those change the archive bytes without changing the file contents.

Practical checks and reproducible workflows to use right away:

# See what differs in archive headers / ordering
tar -tvf a.tar > a.list
tar -tvf b.tar > b.list
diff -u a.list b.list
# Verify file contents (handles spaces safely)
find dir1 -type f -print0 | xargs -0 md5sum | sort -k2 > /tmp/dir1.md5
find dir2 -type f -print0 | xargs -0 md5sum | sort -k2 > /tmp/dir2.md5
diff -u /tmp/dir1.md5 /tmp/dir2.md5

To produce deterministic archives (useful for comparing tar files themselves), normalize metadata and ordering when creating the tar. GNU tar provides options such as --sort, --mtime, --owner, --group and --numeric-owner; for example:

tar --sort=name --mtime='1970-01-01' --owner=0 --group=0 --numeric-owner -cf reproducible.tar dir/
gzip -n reproducible.tar

Notes: older tar versions may lack some options; compressed outputs need gzip -n (or equivalent) to avoid embedded timestamps. For content-only comparison, the per-file checksum approach above is usually the simplest and most reliable.

Recommended Answers

All 2 Replies

Member Avatar for shibblez

Neither, they're both correct. A tar file contains metadata as well as the original data, so if the metadata doesn't match then the checksums won't. If you look on wikipedia for tar(file format) you'll see that the metadata includes stuff like creation data fro both the files and the tarball

Member Avatar for johndoe444

sounds interesting. May be I should try to experiment with the generated tar file. Thanks for the reply.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.