fdf (Fast Duplicate Finder) by Tormod Tjaberg 

email: tormod@tjaberg.com
www  : http://tjaberg.com

Copyright (C) 2006-2020 Tormod Tjaberg

fdf is free software, distributed under the terms of the
GNU General Public License. fdf comes with no warranty at
all. For details see the file COPYING.

description
===========
fdf is a command line program which finds duplicate files on
your file system. fdf works by comparing the contents of files,
instead of their file names. This method allows one to detect
renamed files as well. 

fdf has several features which makes it very powerful:

. only compare files which match a certain mask
. remove the duplicate files found
. perform comparisons between direcory/file sets.
  [more about this later]

fdf is quite fast since only files of equal length
are compared. In addition a shallow compare is done
to eliminate files which have the same length, but differ
in content in the first block.

note_1: 
The executable within the archive is a command line WIN32 binary.

note_2:
Throughout this document I will use '/' as a path delimiter.
Users on WIN systems may of course use 'c:\' or '\' instead.

note_3:
Files of zero length are purpously ignored since many operating
systems use zero length files as locks.

usage
=====
Copy the executable to a directory which is in your PATH. 

Issue the command:

fdf [options] <directory list/filename>

See below for sample usage and detailed option description.

sets
====
One of the most powerful features of fdf is the concept of sets.
Sets are groups of directories or files which you want to compare
against another group of directories or files. This means that
only duplicats across sets will be displayed. Here are a few
examples to clarify the concept:

fdf /usr/foo /usr/bar

The output screen shows the following matches:

Size: 65544 [0xdd719d7c]
[1] /usr/foo/test_1
[1] /usr/bar/xfile_1
Size: 65544 [0x998d1261]
[1] /usr/bar/xfile_2a
[1] /usr/bar/xfile_2b

This is a list of duplicate files found in /usr/foo and in /usr/bar.
In this case /usr/bar contained duplicates within and these are listed
as well. Since no set delimiter was used both sets are a member of set 1.
This is indicated by the "[1]"

Now we will issue the same command again but with the 
set delimiter '-' between the two directories.

fdf /usr/foo - /usr/bar

The screen shows the following:

Size: 65544 [0xdd719d7c]
[1] /usr/foo/test_1
[2] /usr/bar/xfile_1

The duplicate files in /usr/bar are not displayed because
the comparison is only made between the two sets. Notice that the
duplicate files each belong to their own set, in this case "[1]"
and "[2]".

You can use as many sets as you like and as many directories and
files as you like within a set. Note that there might be a limit
on the command line length imposed by your operating system.

If no set delimiter is given, all files will be assigned to set 1.
Each time the delimiter is encountered on the command line the set
number is increased. In the example above "/usr/foo" is in set 1 and 
"/usr/bar" is in set 2.
 
Sets also play an important part when it comes to removal of files.
If you want to remove the duplicates listed above issue the following
command:

fdf -d 2 /usr/foo - /usr/bar

This means: Delete the duplicates in set 2 (/usr/bar). You will
be prompted to delete each file unless you use force (-f) option.

You can also delete files if you only have one set. Consider the
following example (-R means to recurse through the directory):

fdf -R /usr

Size: 65544 [0xdd719d7c]
[1] /usr/bar/muirab
[1] /usr/foo/barium
[1] /usr/tmp/barmiu
[1] /usr/tmp/rabmui

To delete the duplicates you would use the following command

fdf -d 1 -R /usr/foo

You specify the set as normal, but the operation is a bit different.
All the files listed after the first duplicate will be deleted.
In this case the only file which is not deleted will be: /usr/bar/muirab

Note:
The duplicate files will be listed in alphabetical order.

masks
=====
Another powerful feature of fdf is the use of masks. By using 
masks you can single out the files you want to compare.

The mask may be specified in the standard unix glob format which
consists of a set of normal characters and the special glob characters:

* - asterisk, matches any sequence of zero or more characters
? - question mark, matches any single character

The match is case sensitive unless you use the ignore case option (-i).

Note:
It is good practice to quote the masks to prevent the operating
system doing some globbing of its own.

Here are some examples:

fdf -R -m "*.jpg" /local1 

Find all duplicate files with the extension ".jpg" in the directory specified.

fdf -R -i -m "*.jpg" /local1 

Same as above but ignore case.

fdf -R -m "*.jpg" -m "*.fm"  /local1 

Find all duplicates with the extension "*.jpg" and "*.fm" in the directory specified.  

fdf -R -d 2 -i -m "*.jpg" c:\ - d:\ 

Find all the duplicates between drive "c:\" and "d:\" with the extension "*.jpg".
Then perform a prompted delete of all duplicates found on drive "d:\".

You are allowed to specify more than one mask. The default value is 20 masks.

options
=======
If you run fdf with no arguments or with the '-h' option a 
short help screen will be displayed describing all the options.

-R      process subdirectories Recursivly. Recurse through the 
        entire directory structure.

-d [n]  Delete duplicates found in set 'n'

-f      Force deletion of files with no prompt. The default action
        is a prompted deletion.

-m <s>  only compare files which match the Mask 's'. 's' is in
        the normal UN*X glob format and supports '?' & '*'
        
-i      Ignore case when comparing the mask. 

-o <s>  Output results to log file 's'. All program output will be
        written to screen as well as the to the specified file.

-O <s>  Only output the results to log file 's'. No output will be
        written to screen only to the specified file.

-h      display this Help screen
-G      display the GNU General Public Licence

In addition the following options are available on unix based systems:

-A      process All files and directories even hidden ones

-l      follow symbolic Links

compiling
=========
The source code should build on any target which has a 
gcc compiler. It has been tested on:

. Win32/64 (MinGW)
. Linux (Ubuntu 16.04.6)

Use the included makefile

To avoid a lot of complaints under unix flavours unzip the 
archive with the -a option which will auto convert any text files.

credits
=======
. TMH for delightful discussions
. All my testers and proofreaders
. Annette for being what she is...
. AT & KT for just being...
