Finding duplicate files using bash script

Question

How do you write a bash one-liner that will find binary files with identical contents, permissions, and owner on the same ext4 file-system, from the current working directory recursively, and replace all files with older access times with hard links to the latest accessed file and report saved disk space in kibibytes?

What I achieved until now is not fully sufficient for the requirements of the objective.

#! /bin/sh
fdupes -r -p -o 'time' . | xargs file -i | grep binary | awk '{print $1}' | awk '{print substr($0,3)}' | sed 's/.\{1\}$//' | xargs rdfind -makehardlinks true

Answer 1

hardlink may not satisfy all requirements for this, but it can be used for what it is, to make the hardlinks. It can accept file arguments, not only directories, and it seems it is always linking a group of identical files to the first in order. Also it will ignore zero size files.

fdupes selects exactly what needed, but does not output real file arguments but a paragraph-mode output, with groups of identical files, every group is ended with an empty line.

So in order to be sure that the exact selections of fdupes will be hardlinked, we have to call hardlink separately once per paragraph. To avoid the case where two pairs of the same identicals exists for different owners or with different permissions. And of course files have to be filtered for binaries.

#!/bin/bash
unset arr i
while IFS= read -r f; do

    # move file to array if binary
    if file -i "$f" | grep -q "charset=binary"; then
        arr[++i]="$f"
    fi
    
    # if end of paragraph and array has files, hardlink and unset array
    if [[ "$f" == "" && "${arr[@]}" ]]; then
        printf "\n => Hardlink for %d files:\n" "$i"
        hardlink -n -c -vv "${arr[@]}"
        unset arr i
    fi

done < <(fdupes -rpio time .)

hardlink with -n parameter simulates and does not write anything, so test the above as is and remove -n later.

Also filenames with newlines are not handled, testing with whitespaces seems ok.

Answer 2

Finally got the desired result. Thanks to @thanasisp. For this, you need two programs fdupes and rdfind.

#!/bin/bash
unset arr i; while IFS= read -r f; do if file -i "$f" | grep -q "charset=binary"; then arr[++i]="$f"; fi; if [[ "$f" == "" && "${arr[@]}" ]]; then printf "\n => Hardlink for %d files:\n" "$i";rdfind -makehardlinks true "${arr[@]}" | grep "Total size is" | grep -P "[0-9]+" -o  | head -1 | awk -v count="$i" '{print $1/count;}' | awk '{printf("%s kibibytes saved.\n",$1/1024)}'; unset arr i; fi; done < <(fdupes -rpio time .)