Linux deduping is in store next

EddieC 0 Tallied Votes 481 Views Share

Plagued by duplicate files that clog up your system? Quantum, a company once known mainly for hard drives, yesterday unveiled a new version of its StorNext File System that it says optimizes storage efficiency by implementing automatic data deduplication. It's part of StorNext 4.0, Quantum's high-performance sharing and data management platform, which now can also automate data tiering and retrieve partial files based on timecode, for the video editor in you.

Storage analyst Tom Coughlin, of Coughlin Associates, said that for media types, the benefits of deduping would be less significant than, say, those of time-based features. Deduping eliminates repetition of data in storage blocks, thereby reducing demand for storage capacity and network bandwidth for moving data around. "Media professionals are dealing mostly with original content, so there's not much duplication in that," he said. But retrieving just the 15 seconds you need from a 30-minute video? "That lets me move just the pieces of data I need, so I can save time as well as operating and capital expenses by making due with existing resources."

The ability of a file system itself to perform deduplication constitutes a form of data tiering, according to Chris Duffy, NextStor product marketing manager at Quantum. He said that by enabling a "dedupe repository," all files dropped in a particular folder are thereby optimized and can be accessed faster than those not using that folder. "You can call it tiering because you're actually optimizing your primary storage environment."

StorNext installs on Linux, Mac OS X, Unix and/or Windows hosts that share disks, arrays or SANs, and facilitates high-speed file sharing and simultaneous file access. A metadata controller monitors sharing of Linux and Windows client machines. StorNext 4.0 is set to begin shipping in mid-February; average pricing starts at around US$2000.

Dani AI

Generated

Good call, — vendors adding dedupe to enterprise stacks matters, but Linux users and admins should treat dedupe as a feature to evaluate, not flip a switch on. The original news is useful; what follows fills the practical gap for Linux: what kinds of dedupe exist, where Linux offers it, and quick checks to decide whether to enable it.

Deduplication modes and trade-offs: inline dedupe removes duplicates as data is written (saves capacity and bandwidth but raises CPU/memory pressure); post-process dedupe stages data and removes duplicates later (lower write impact but needs extra staging space). There are also block-layer (below the filesystem) and filesystem-level approaches; each has different performance and recovery impacts. (techtarget.com)

Linux options worth knowing:

  • ZFS dataset-level dedupe is inline and can save large space for highly duplicated data, but it consumes substantial RAM and can degrade performance — always simulate first. (openzfs.github.io)
  • Filesystem tools such as duperemove find duplicate extents and request kernel dedupe (FIDEDUPERANGE) on supporting filesystems (Btrfs, XFS). That gives a safe, on-disk collapse without moving file paths. (mankier.com)
  • Block-level inline dedupe (example: Red Hat VDO) sits under LVM/filesystems and provides dedupe+compression for many workloads; it’s often the safest path for broad, transparent savings on RHEL systems. (docs.redhat.com)

Quick practical checklist (test everything on samples first):

  • Simulate ZFS savings: zdb -S <pool> to estimate dedupe/compress ratio.
  • Scan-and-dedupe candidates with duperemove: duperemove -dr --hashfile=/var/tmp/hashes /path/to/data (review results before -d).
  • For RHEL, try a VDO test volume with vdo create --name=vdo1 --device=/dev/sdX --vdoLogicalSize=10T and benchmark. (openzfs.github.io)

Final note: enable dedupe only after measuring likely savings vs. added CPU/RAM and after planning recovery/backups. For many workloads (media editing, encrypted content) compression or reflink/cloning may be a better first step. (openzfs.github.io)

EddieC 0 Posting Whiz in Training

Clarification on the pricing:
A data deduplication license costs $1,000 per TB (physical usable capacity of the repository). Replication for the file server is priced at $15,000, while StorNext Storage Manager – part of the core product used to provide automated data movement – is priced at $35,000. The Distributed Data Mover is priced at $15,000, and Partial File Retrieval costs $36,000.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.