Year 2000 has passed, no flying cars yet but at least more faster (SSD and NVMe) disk space 😀

All sorts of resources also digital resources and bandwidth, RAM and DiskSpace space are STILL precious and should be treated as such. (one’s Android based smart phone has 12GBytes of RAM, because a lot of (Java)Apps are a bloat)

No matter on server, notebook or smart phone or on the whole planet: Resources should never be wasted and can not be wasted endlessly (thus keep it simple + minimalism + recycle + reuse (almost) every atom).

“WhatsApp Using Up Your Phone Storage? Here’s How to Fix It” (there should be no such problems in the first place)

Imagine two very large identical files, it would be UTTER uselessness to waste blocks on both.

when having duplicates make sense: backups across devices

of course it’s IMPORTANT to have duplicates of important files across multiple devices (harddisks, flash storage, optical disks, tapes in different places in double-and-tripple-metal-casing-to-shield-against-electro-magnetic-interference) to avoid dataloss.

But ON THE SAME DEVICE one copy of each file or data block would be enough, so it is a bit strange that storage waste by duplicate data is (still) an (massive?) issue daily and globally that needs manual intervention (surely Facebook, Amazon & Google have solved this problem server-side, otherwise even their datacenters would be overfilled fast X-D)

There are cool filesystem that do that automatically: ZFS but it’s only for EEC-servers.

There are cool features of ext4 such like hardlinks and softlinks which can help avoid duplicates.

But softlinks are utter useless with the deduplication task, as the connection between link and file is lost the moment the file is renamed or moved.

Cool: the hardlink program (hardlink.man.txt) can AUTOMATICALLY find duplicate files and simply replace one copy with a hardlink to the original data thus depending on how much duplicate data there is, saving massively on diskspace.

filesystems that handle duplicate data efficiently?

(on a filesystem level) :

sort of file system level

https://btrfs.readthedocs.io/en/latest/Deduplication.html

There are two main deduplication types:

• in-band (sometimes also called on-line) — all newly written data are considered for deduplication before writing
• out-of-band (sometimes also called offline) — data for deduplication have to be actively looked for and deduplicated by the user application

Both have their pros and cons. BTRFS implements only out-of-band type.

BTRFS provides the basic building blocks for deduplication allowing other tools to choose the strategy and scope of the deduplication. There are multiple tools that take different approaches to deduplication, offer additional features or make trade-offs. The following table lists tools that are known to be up-to-date, maintained and widely used.

deduplicate data with hardlinks or softlinks?

where’s actually the difference?

hardlinks are pointers to the original data on disk. if one of multiple hardlinks is deleted, the data can still be accessed by the other hardlinks.

not so with softlinks. when the original data is deleted, the softlink becomes “broken”.

# for testing purposes
# create temp dir
mkdir temp
cd temp
# create a file with data
echo "this is a small text file that is not empty" > small.text.testfile
# verify that there is data
xxd -b small.text.testfile |less

# per default ln creates hardlinks
ln -v /path/file hardlinkname
# create an actual hardlink linking to the testfile
ln -v small.text.testfile hardlink.small.text.testfile
# verify it's a hardlink
ls -l hardlink.small.text.testfile
-rw-r--r-- 2 user user 44 2023-10-04 12:52 hardlink.small.text.testfile

# with -s option creates softlinks
ln -sv /path/file softlinkname
# create an actual softlink linking to the testfile
ln -sv small.text.testfile softlink.small.text.testfile

# verify it's a hardlink
ls -l softlink.small.text.testfile
lrwxrwxrwx 1 user user 19 2023-10-04 12:53 softlink.small.text.testfile -> small.text.testfile

# when deleting the original file X-D
rm -rf small.text.testfile

# what is still there?
ls -lah
# this hardlink still works (connection to data still there)
-rw-r--r--  1 user user 44 2023-10-04 12:52 hardlink.small.text.testfile

# verify data still accessible
xxd -b hardlink.small.text.testfile|less

# this softlink is broken (connection to data lost)
lrwxrwxrwx  1 user user 19 2023-10-04 12:53 softlink.small.text.testfile -> small.text.testfile

# verify connection to data is lost
xxd -b softlink.small.text.testfile|less

hard link not allowed for directory?

ln -v /target /linkname; # will result in that error

possible solutions: mount or bindfs: difference?

# this will mount the directory as specific user with uid 1002
bindfs -u 1002 -g 1002 /target /mountpoint

mount --bind /target /mountpoint

# make it permanent, restore on boot
echo '/target /mountpoint none bind 0 0' >> /etc/fstab

• mount -t bind /target /mountpoint
  • Kernel-level bind mount: makes the same filesystem tree visible at another path.
  • Preserves original inode ownership, permissions, SELinux labels, device numbers, etc.
  • Fast, low overhead. Requires root (or CAP_SYS_ADMIN).
  • You cannot remap UIDs/GIDs or change ownership/permission semantics with plain bind.
  • Typical use: exposing the same data under another path (chroots, containers, different mount namespaces).
• bindfs -u 1002 -g 1002 /target /mountpoint
  • FUSE-based user-space filesystem that mirrors a directory but can remap ownership/permissions.
  • -u/-g remap all file owners and groups as presented through the mount (files keep original owners on disk).
  • Allows many option-based changes (uid/gid mapping, permission masks, create-as-owner, treat-as, set-uid, etc.).
  • Slightly higher CPU/latency overhead than kernel bind; can be mounted by non-root if FUSE permissions allow.
  • Useful when you need a view of files that appears owned by a different UID/GID (e.g., give an unprivileged process access without chown).

When to use which

• Use mount –bind when you only need another path for the exact same view and want best performance.
• Use bindfs when you need to present a different ownership/permission view (UID/GID mapping, masks) or when non-root mounting is required.

Notes

• bindfs is not a replacement for changing ownership on disk — mapping is only at the FUSE layer.
• SELinux/AppArmor and some low-level filesystem features may still reflect the underlying attributes and can affect access.

 

how to go about manual deduplication on ext4 based systems then?

GNU Linux -> how to find and remove duplicate files

liked this article?

• only together we can create a truly free world
• plz support dwaves to keep it up & running!
• (yes the info on the internet is (mostly) free but beer is still not free (still have to work on that))
• really really hate advertisement
• contribute: whenever a solution was found, blog about it for others to find!
• talk about, recommend & link to this blog and articles
• thanks to all who contribute!