The Hardlink Inode Flyweight: Inode-Level Storage Deduplication
Eliminate millions of redundant file duplicates across massive build caches by collapsing identical file bodies into shared physical inode flyweights.
find /var/cache -type f ! -empty -links 1 -exec sha256sum {} + | sort | awk 'curr==$1 {print "ln -f", prev_f, $2} {curr=$1; prev_f=$2}' | bash - The Lineage to the Gang of Four
In the Gang of Four catalog, the Flyweight pattern conserves resources through fine-grained object sharing:
“Use sharing to support large numbers of fine-grained objects efficiently.” — Gang of Four, Structural Patterns
Flyweight dictates that when an application holds thousands of identical objects, their intrinsic state should be stored once in memory, while extrinsic attributes (like position or context) remain independent.
The Transmutation to POSIX Inodes and Directory Entries
In the Linux Virtual Filesystem (VFS), a file is composed of two distinct parts:
- The Inode (Intrinsic Flyweight State): The physical data blocks on disk, file size, permissions, and cryptographic content.
- The Directory Entry / Dentry (Extrinsic State): The human-readable filename and its path location in the folder hierarchy.
In build servers, CI pipelines, and artifact stores (node_modules, Maven caches, container image layers), millions of files are identical byte-for-byte. Storing each duplicate with a separate physical inode exhausts disk space and thrashes the kernel page cache.
The Hardlink Inode Flyweight scans for duplicate file contents via SHA-256 checksums, replacing redundant file copies with Hardlinks (ln -f). Different paths (extrinsic state) point to a single shared physical inode (intrinsic state), freeing gigabytes of disk in a single pass.
The Spell Formula
Cast this invocation to locate identical files and collapse them into shared inode flyweights:
find /var/cache -type f ! -empty -links 1 -exec sha256sum {} + \
| sort \
| awk 'curr==$1 {print "ln -f", prev_f, $2} {curr=$1; prev_f=$2}' \
| bash -
Anatomy of the Spell
1. find /var/cache -type f ! -empty -links 1 -exec sha256sum {} +
! -empty: Skips zero-byte files (linking empty files yields no disk savings).-links 1: Only evaluates files that are not already hardlinked, avoiding redundant hash computations.sha256sum {} +: Batches hundreds of files per process invocation, computing cryptographic fingerprints to guarantee zero byte collisions.
2. sort | awk '...'
sort: Groups files with identical SHA-256 hashes adjacent to each other.awk 'curr==$1 {print "ln -f", prev_f, $2}': When two adjacent records share the identical checksum hash (curr==$1),awkemits a command linking the duplicate path$2to the canonical targetprev_f.
3. | bash -
- Executes the atomic hardlinking commands directly in the shell. The kernel unlinks the duplicate inode and updates the primary inode’s link count (
nlink).
Flyweight Memory & Storage Savings
| Metric | Duplicate File Copies | Hardlink Inode Flyweight |
|---|---|---|
| Physical Storage | $N \times \text{Size}$ blocks | $1 \times \text{Size}$ blocks |
| Linux Page Cache | Loaded $N$ times into RAM buffers | Cached once in RAM across all processes |
| Filesystem Semantics | Modifying one copy breaks parity | Both paths resolve to identical data |
| GoF Pattern Alignment | Ad-hoc duplicate state | Strict Intrinsic / Extrinsic Separation |
By decoupling the file path from the underlying storage blocks, the Flyweight pattern recovers vast amounts of disk space and memory bandwidth across Linux systems.