Member Avatar for Violet_82

Hi there, I was wondering if compressing a file and work with it while it is still compressed is possible/will be possible in the future?
Say you have a huge file, I am talking about terabytes of files, obviously just in theory, would it be possilble to work with it without decompressing it?
thanks

  • 2 Contributors
  • 5 Replies
  • 105 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by JorgeM

Dani AI

Generated

Short answer: it is already possible to work with very large files while keeping them stored compressed — but only if the compression is arranged so the reader can fetch and decompress small pieces (chunks or blocks) instead of a single long stream. The typical approaches are either (a) store the data in a container or format that supports chunked/block compression and indexing, or (b) put the data on a storage layer that compresses at the block level.

Concrete examples and where to look:

  • Block/chunked container formats let applications read only the compressed blocks they need (good for random reads and partial updates). Examples: HDF5 chunked datasets (), BGZF (blocked gzip used in genomics to allow seeking) (), and columnar formats such as Apache Parquet that compress columns in chunks ().
  • Storage-level block compression (ZFS, many SANs) compresses data per logical block so reads typically touch only the blocks required ().

Practical next steps for a TB-scale dataset (grounded in ’s scenario and expanding on ’s points):

  • Identify access patterns (random single-record reads, ranges, full scans). Pick a format that matches those patterns (chunked HDF5 or Parquet for analytical workloads; BGZF-like blocked gzip if you need simple seekable compressed files).
  • Choose a block size: smaller blocks = finer-grained reads but higher metadata overhead; larger blocks = better compression ratio but more extra IO on partial reads.
  • Prototype and benchmark CPU cost, read latency, and compressed-size tradeoffs before converting all data.
  • For write-heavy workloads prefer designs that rewrite single blocks or use append/journaled containers to avoid expensive full-file rewrites.

Caveats: stream gzip/tar is poor for random access; in-place updates are harder on block-compressed files; backup and concurrency need design consideration. The references above give implementation details and proven patterns to follow.

Recommended Answers

All 5 Replies

Member Avatar for JorgeM

When you compress a file in Windows and you access it, Windows automatically uncompresses it for you during the access. This adds additional CPU power (not a big deal) and of course, you have to have enough space on disk during the decompression.

Member Avatar for Violet_82

thanks JorgeM, so I take it is not possible to access a compressed file and working on it while it is compressed? What is it that prevent this from happening, i mean, why a file needs to be decompressed (whether manually or automatically) before being accessed?
thanks

Member Avatar for JorgeM

This is my understanding of how Windows handles compressed files. I would assume even if you had an app that was able to access the same compressed file, it would need to be uncompressed to view the contents. I would tend to think this is so because of the way that compression works. Generally, compression looks at patterns in the file and removes those patterns and replaces these patterns with items that take up less space.

Member Avatar for Violet_82

thanks, I think I might need to look into how compression works then, just out of interest. Do you suggest any good link?

Member Avatar for JorgeM

I think you will find various techniques for compression. Start with a generic search to get basic information about the concept and go from there.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.