Data Deduplication and Compression: Saving Storage Space

Enterprise data volumes are doubling every two to three years, driven by cloud adoption, digitisation of business processes, remote work generating distributed data, and regulatory requirements that mandate longer retention periods. While the cost per terabyte of raw storage continues to decline, the total storage spend for most organisations is rising because data growth outpaces price drops. Data reduction technologies — primarily deduplication and compression — address this by reducing the amount of physical storage needed to hold a given logical data set. A 10:1 data reduction ratio means that 100 TB of logical data occupies only 10 TB of physical storage, translating directly into hardware savings, lower power consumption, and reduced replication bandwidth for disaster recovery.

For IT resellers, understanding data reduction is essential when sizing storage solutions and backup infrastructure. A customer who stores 50 TB of virtual machine images will see dramatically different deduplication ratios than one storing 50 TB of compressed video files. Accurately estimating the achievable reduction ratio determines whether you recommend a 24-bay NAS or a 12-bay unit, and whether the customer's backup repository needs 100 TB or 20 TB of raw capacity. Getting this wrong means either over-provisioning (wasting the customer's money) or under-provisioning (requiring an expensive expansion within months).

Data deduplication (often shortened to "dedup") identifies and eliminates redundant copies of data, storing only one unique instance of each data segment and replacing duplicates with pointers to the stored copy. The process works by dividing data into segments (chunks), computing a cryptographic hash (typically SHA-256) for each chunk, and comparing the hash against an index of previously stored chunks. If a matching hash exists, the chunk is a duplicate and only a pointer is stored; if no match exists, the chunk is new and is written to storage. This approach is remarkably effective for workloads with high redundancy — virtual machine backups, file server snapshots, and email archives routinely achieve deduplication ratios of 10:1 to 50:1.

The chunk size significantly affects deduplication effectiveness. Fixed-length chunking divides data into blocks of a constant size (e.g., 4 KB, 8 KB, 64 KB). This is simple and fast but can miss deduplication opportunities when data is inserted or shifted within a file, causing all subsequent chunk boundaries to change. Variable-length chunking (also called content-defined chunking) uses a sliding-window algorithm to determine chunk boundaries based on the data content itself, making it resilient to insertions and shifts. Most enterprise deduplication engines use variable-length chunking for better reduction ratios, at the cost of slightly higher CPU overhead during the chunking process.

Inline deduplication processes data as it is written, before it reaches the storage media. Every incoming block is hashed and checked against the dedup index in real time — if it is a duplicate, it is never written to disk, saving both capacity and write I/O. The advantage is immediate space savings and reduced wear on SSDs (since duplicate writes are avoided entirely). The trade-off is that inline dedup adds latency to write operations because the hash computation and index lookup must complete before the write is acknowledged. For backup workloads where write speed is critical (to meet backup windows), this latency can be a bottleneck on underpowered systems.

Post-process deduplication writes all incoming data to disk first, then runs a background dedup job that identifies and removes duplicates after the fact. This approach delivers the fastest possible write performance because no processing occurs during the write path. The downside is that you need enough raw storage capacity to hold the full un-deduped data set temporarily, and the post-process job consumes CPU, memory, and I/O bandwidth when it runs. Post-process dedup is becoming less common as hardware has become powerful enough to handle inline processing without significant performance impact, but it remains useful in scenarios where backup performance is the absolute priority.

Source-side deduplication runs on the machine generating the data — the backup agent on a server, for example. Before sending data across the network to the backup repository, the agent computes hashes and checks them against the repository's index. Only unique (non-duplicate) data is transmitted, dramatically reducing network bandwidth consumption. This is particularly valuable for remote office and branch office (ROBO) backups where the WAN link is limited. Veeam, Commvault, and Veritas NetBackup all support source-side dedup in their agent-based backup products.

Target-side deduplication runs on the backup repository or storage appliance. The source sends all data across the network, and the target performs dedup before (inline) or after (post-process) storing it. This approach is simpler to manage because the dedup logic is centralised, and it does not consume CPU resources on the production servers. Dedicated backup appliances from vendors like Dell Data Domain (now PowerProtect DD), HPE StoreOnce, and ExaGrid are essentially purpose-built target-side dedup engines optimised for handling high-throughput backup streams.

While deduplication eliminates duplicate chunks across a data set, compression reduces the size of individual data blocks by encoding redundant patterns more efficiently. Compression algorithms replace repeated byte sequences with shorter representations — for example, a run of 100 identical bytes can be encoded as "repeat byte X, 100 times" in just a few bytes. Compression and deduplication are complementary: dedup removes copies of identical chunks, while compression makes each remaining unique chunk smaller. Used together, they deliver compound data reduction ratios that neither technology achieves alone.

Several compression algorithms are commonly used in storage and backup products. LZ4 is a fast, lightweight algorithm that prioritises speed over compression ratio — it is the default in many storage arrays and backup products (including Veeam) because it adds minimal latency to the write path. Zstandard (zstd), developed by Facebook, offers a tuneable trade-off between speed and ratio, making it increasingly popular in modern storage systems. Gzip/deflate provides better compression ratios than LZ4 at the cost of higher CPU usage, and is widely used for archival. LZMA/XZ achieves the highest compression ratios but is CPU-intensive, making it suitable only for offline archival scenarios where speed is not a concern.

For backup administrators, the critical question is how deduplication and compression affect the backup window — the time available to complete all backup jobs before production workloads resume. Source-side dedup has a positive impact because it reduces the volume of data transferred, often cutting network transfer time by 80 to 95 per cent for incremental backups. A daily incremental backup of a 10 TB file server that has only 200 GB of changed data might further dedup down to 40 GB of unique data, transferring in minutes rather than hours. Inline dedup on the target can slow ingest rates if the dedup engine cannot keep up with the data stream, but modern appliances with dedicated ASIC or FPGA hardware (such as Dell PowerProtect DD) can ingest at multi-gigabyte-per-second rates with inline dedup enabled.

Compression adds CPU overhead proportional to the algorithm's complexity and the data volume. LZ4 compression on a modern multi-core server adds negligible time — perhaps 1 to 3 per cent to the backup window — while providing 1.5:1 to 3:1 compression on typical business data. Zstandard at medium compression levels adds slightly more overhead but achieves 2:1 to 5:1 ratios. The net effect is almost always positive: the time saved by writing less data to disk and consuming less network bandwidth outweighs the time spent on compression. The exception is workloads dominated by incompressible data, where compression adds CPU overhead without reducing data volume.

Veeam Backup & Replication is the most widely deployed backup solution in the Australian mid-market. Veeam implements both inline deduplication and compression at the backup proxy level. The dedup block size is configurable (typically 256 KB or 1 MB for LAN-based jobs, and 256 KB for WAN-accelerated jobs), and compression uses LZ4 by default with options for zlib and zstd. Veeam also supports integration with dedicated dedup appliances — when writing to a Dell PowerProtect DD or HPE StoreOnce, Veeam can offload dedup to the appliance and disable its own dedup to avoid double processing. For resellers, configuring Veeam's dedup and compression settings correctly for each customer's environment is one of the highest-value optimisations you can make.

Commvault offers a comprehensive data management platform with built-in source-side and target-side deduplication. Commvault's dedup engine uses variable-length chunking and maintains a global dedup database (DDB) that can span multiple storage targets, maximising dedup efficiency across the entire backup estate. Commvault also supports hardware-accelerated dedup on appliance platforms and has strong integration with cloud object storage for tiering deduplicated data to S3 or Azure Blob. For large enterprise customers with complex multi-site environments, Commvault's global dedup is a significant differentiator.

QNAP and Synology NAS devices bring dedup and compression to the SMB market. QNAP's QuTS hero operating system supports inline deduplication and compression on ZFS-based storage pools, delivering enterprise-grade data reduction on affordable hardware. Synology's Active Backup for Business suite includes target-side dedup for VM, file server, and PC backups. For smaller customers who cannot justify a dedicated dedup appliance, these NAS platforms provide significant space savings at a fraction of the cost. Resellers should be aware that dedup on NAS devices is memory-intensive — the dedup index must be held in RAM for acceptable performance, so the NAS should be configured with maximum supported memory.

Data reduction ratios vary enormously depending on the data type and retention policy. Virtual machine backups with multiple retention points are the best case — because each daily backup shares most of its data with the previous day, dedup ratios of 20:1 to 50:1 are common when retaining 30 or more days of backups. File server data with versioned documents typically achieves 5:1 to 15:1. Database backups vary — SQL databases with many similar tables dedup well, while encrypted or heavily compressed database exports dedup poorly. Media files (video, audio, images) are the worst case — they are already compressed and unique, yielding ratios of 1.1:1 to 1.5:1 at best. When sizing storage for a customer, use conservative estimates (the lower end of the range) and confirm with a proof-of-concept if the purchase decision hinges on achieving a specific ratio.

When designing storage and backup solutions with data reduction, follow these guidelines. First, always enable both dedup and compression — they are complementary and the combined ratio is almost always significantly better than either alone. Second, choose inline dedup for primary storage and backup targets with sufficient CPU and RAM, and reserve post-process dedup only for environments where write performance is absolutely critical and cannot tolerate any additional latency. Third, use source-side dedup for WAN-based backups (remote offices backing up to a central repository) to minimise bandwidth consumption. Fourth, size the dedup index memory appropriately — as a rule of thumb, allow 1 to 2 GB of RAM per terabyte of deduplicated logical data. Fifth, monitor dedup ratios over time and investigate sudden drops, which may indicate changes in the data profile (such as a new application generating incompressible data) or dedup index issues.