Comments on: Writing a duplicate file finder in Erlang http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/ A place for hopefully something useful, but I really don't know. Tue, 06 Jan 2009 07:27:24 +0000 http://wordpress.org/?v=MU hourly 1 By: free of slot fortune wheel http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-3550 free of slot fortune wheel Sat, 20 Sep 2008 17:02:08 +0000 http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-3550 download slot fortune wheel of <a href="http://www.al.com/forums/profile.ssf?nickname=owen97" rel="nofollow">wheel fortune slot of download</a> download slot fortune wheel of wheel fortune slot of download

]]> By: Jim Harris http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-3538 Jim Harris Tue, 05 Aug 2008 16:43:54 +0000 http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-3538 I can't get the source. Save never finishes; times out repeatedly, on separate days. Please advise. Thanks for any help. Jim I can’t get the source. Save never finishes; times out repeatedly, on separate days.

Please advise.

Thanks for any help.

Jim

]]> By: thepetite http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-2123 thepetite Sun, 26 Aug 2007 17:44:59 +0000 http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-2123 comment - thepetite comment - thepetite

]]> By: diginux http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-435 diginux Wed, 09 May 2007 12:10:44 +0000 http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-435 Pádraig, Actually not a bad tool now that I have tried it. I just saw it was a GUI, and no information was listed about the required libraries for it, so I just assumed it was a pain. You should make your introductory page of it a little more informative. Thanks! Jordan Wilberding Pádraig,

Actually not a bad tool now that I have tried it. I just saw it was a GUI, and no information was listed about the required libraries for it, so I just assumed it was a pain. You should make your introductory page of it a little more informative.

Thanks!
Jordan Wilberding

]]> By: Pádraig Brady http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-434 Pádraig Brady Wed, 09 May 2007 11:29:09 +0000 http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-434 Hey you didn't provide a link to FSlint. boo hiss :) http://www.pixelbeat.org/fslint/ You can download the tarball of fslint. Also there is an ebuild file linked prominently on that page. I've been very careful about performance in fslint, and I've found nothing as robust or fast even though it's written in shell script. cheers, Pádraig. Hey you didn’t provide a link to FSlint. boo hiss :)
http://www.pixelbeat.org/fslint/

You can download the tarball of fslint.
Also there is an ebuild file linked prominently on that page.

I’ve been very careful about performance in fslint,
and I’ve found nothing as robust or fast even though
it’s written in shell script.

cheers,
Pádraig.

]]> By: diginux http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-64 diginux Tue, 17 Apr 2007 17:30:58 +0000 http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-64 I'll take your ideas into consideration for a version 2. I know it is "cheating", but in general I still got the same accuracy of results in less time. Also, it seemed quicker to take 512 bytes, hash it, then compare the 16 bit hashes, then it took to just compare the 512 bytes. I admit that doesn't make much sense, but I didn't have time to investigate further. Thanks for your comments, I will have to check your implementation out! I’ll take your ideas into consideration for a version 2. I know it is “cheating”, but in general I still got the same accuracy of results in less time. Also, it seemed quicker to take 512 bytes, hash it, then compare the 16 bit hashes, then it took to just compare the 512 bytes. I admit that doesn’t make much sense, but I didn’t have time to investigate further.

Thanks for your comments, I will have to check your implementation out!

]]> By: Justin http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-27 Justin Sat, 14 Apr 2007 03:04:45 +0000 http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-27 It looks like you are only looking at part of the files, which is cheating :) You actually do not need to calculate any md5 hashes in order to implement a duplicate file finding program efficiently. The Perl program you referenced seems to get things almost right, but it looks like it does too much reading. In your case, you could probably just store the 256+256 byte chunks as the "hash" itself, rather than the md5 hash of it. I have a duplicate finder program implemented in python, but the general idea would be able to be implemented in Erlang as well. The method I use is to find all files of the same size, and then compare them all to each other 8K at a time. With this method, the program only reads as many bytes as it needs to to determine if files are different, or the same. It only ends up reading an entire file if it is actually a duplicate. This makes it extremely IO efficient, while also being 100% accurate. http://bouncybouncy.net/dupes.py It looks like you are only looking at part of the files, which is cheating :) You actually do not need to calculate any md5 hashes in order to implement a duplicate file finding program efficiently. The Perl program you referenced seems to get things almost right, but it looks like it does too much reading. In your case, you could probably just store the 256+256 byte chunks as the “hash” itself, rather than the md5 hash of it.

I have a duplicate finder program implemented in python, but the general idea would be able to be implemented in Erlang as well. The method I use is to find all files of the same size, and then compare them all to each other 8K at a time. With this method, the program only reads as many bytes as it needs to to determine if files are different, or the same. It only ends up reading an entire file if it is actually a duplicate. This makes it extremely IO efficient, while also being 100% accurate.

http://bouncybouncy.net/dupes.py

]]> By: diginux http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-6 diginux Thu, 05 Apr 2007 01:27:22 +0000 http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-6 I probably will in the next version if I update it. From my testing on my own files, it just wasn't needed, and I was trying to keep the code as clean as possible and reduce the number of comparisons needed. I will definitely do it though if I update the code. I probably will in the next version if I update it. From my testing on my own files, it just wasn’t needed, and I was trying to keep the code as clean as possible and reduce the number of comparisons needed. I will definitely do it though if I update the code.

]]> By: Dan http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-5 Dan Wed, 04 Apr 2007 22:30:26 +0000 http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-5 For better accuracy, (assuming you didn't already do this) why don't you just take the ones that hash the same with the first and last 256 bytes, and do a full comparison on just those files? For better accuracy, (assuming you didn’t already do this) why don’t you just take the ones that hash the same with the first and last 256 bytes, and do a full comparison on just those files?

]]> By: diginux http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-4 diginux Wed, 04 Apr 2007 18:57:26 +0000 http://blog.diginux.net/2007/04/03/writing-a-duplicate-file-finder-in-erlang/#comment-4 Vince, I appreciate testing it in Windows. I did not have the time to try it out yet. Good to hear that it works well! Vince, I appreciate testing it in Windows. I did not have the time to try it out yet. Good to hear that it works well!

]]>