How to delete all duplicate hardlinks to a file?

Question

I've got a directory tree created by rsnapshot, which contains multiple snapshots of the same directory structure with all identical files replaced by hardlinks.

I would like to delete all those hardlink duplicates and keep only a single copy of every file (so I can later move all files into a sorted archive without having to touch identical files twice).

Is there a tool that does that?
So far I've only found tools that find duplicates and create hardlinks to replace them…
I guess I could list all files and their inode numbers and implement the deduplicating and deleting myself, but I don't want to reinvent the wheel here.

Answer 1

In the end it wasn't too hard to do this manually, based on Stéphane's and xenoid's hints and some prior experience with find.
I had to adapt a few commands to work with FreeBSD's non-GNU tools — GNU find has the -printf option that could have replaced the -exec stat, but FreeBSD's find doesn't have that.

# create a list of "<inode number> <tab> <full file path>"
find rsnapshots -type f -links +1 -exec stat -f '%i%t%R' {} + > inodes.txt

# sort the list by inode number (to have consecutive blocks of duplicate files)
sort -n inodes.txt > inodes.sorted.txt

# remove the first file from each block (we want to keep one link per inode)
awk -F'\t' 'BEGIN {lastinode = 0} {inode = 0+$1; if (inode == lastinode) {print $2}; lastinode = inode}' inodes.sorted.txt > inodes.to-delete.txt

# delete duplicates (watch out for special characters in the filename, and possibly adjust the read command and double quotes accordingly)
cat inodes.to-delete.txt | while read line; do rm -f "$line"; done

Answer 2

To find the inodes that have more that one link:

 find . -exec stat -c '%i' {} \; | sort -n | uniq -d

Then you can iterate that list with

 find -inum {inode_number}

to list the files that share the inode. Which to remove is up to you.

Answer 3

If you know that all the files have hardlinks within a single directory hierarchy only, you can simply do

find inputdir -type f -links +1 -exec rm {} \;

The reason this works is that rm {} \; removes exactly one file immediately the stat() returns count more than 1. As a result, the hard link count of the other parts of the same inode will be decreased by 1 and if the file is then the only copy, the rm will not be run for that last file by the time find runs stat() against that file.

Note that if any file has hardlinked copies outside the inputdir this command will remove all copies within inputdir hierarchy!

Answer 4

I think you are mistaken, since you think deleting all the "other" links to a file will save space. The only space you will save is a directory entry, and even that is questionable.

All hard links to a file are equal. There are no "duplicates". Files on Linux are really identified by which filesystem they are on, and what Inode number they are on that filesystem.

So when you create a file, you create an Inode, where the blocks actually live, and you create a link to that file in some directory. That link just points at that inode. If you do a hard link from that directory entry to another place, you just create a second directory entry someplace pointing to that same file.

If you run ls -i on a file, you will see it's Inode number. If you want to find other hard links to that same inode, simply run:

find /TOP-OF-FILESYSTEM -type f -inum INODE-NUMBER

Where TOP-OF-FILESYSTEM is the mount point for that filesystem, INODE-NUMBER is the inode number for the file in question. Note that "-type f" is not mandatory but just speeds up the search since you only will look for files.

Note that running ls -il on a file also (by default) it's inode number.

You can test all of this by going to a scratch directory and creating a file, then creating another link to it:

cd ~/tmp
date > temp1
ln tmep1 temp2
ls -l temp*

Answer 5

rmlint will find and remove duplicates, including hardlinks. At the moment it has no options to remove only hardlinks. Removal is done via an autogenerated shell script, so you can review that script prior to deletion.

In general be careful when using duplicate file detectors in hardlink mode (eg fdupes -H) since they can sometimes mistakenly identify a file as its own duplicate (see "path doubles" discussion here).

Answer 6

To make a copy which has no hard links you could copy them to a different file system and then back again or just leave them on the other file system.

To the best of my knowledge hard links do not span file systems. I am happy to be told I am wrong on that point as it will fill a hole in my understanding but I am quite confident I am correct (generally) on this point.