Page 1 of 1
PB38: Comparing large binary files
Posted: 08 Dec 2009, 22:48
by mdruiter
I tried the performance of Compare Files (Ctrl+Shift+C), on two very different binary files. The large files were 891,352,865 and 417,412,266 bytes.
With Salamander 2.52, I got the usual (but confusing) "Computing Differences" in the caption, immediate results, and after a few seconds the caption said: More than 32 768 differences.
With 2.53 preview build 38, I got this frightening message:
Insufficient memory to allocate 891352880 bytes. Try to release some memory (e.g. close some running application) and click Retry. If it does not help, you can click Ignore to pass memory allocation error to this application or click Abort to terminate this application.
Can this be avoided, perhaps with the 'slower' method as a fallback?

I would really like to be able to compare large binary files...
When I swap the files, it takes a little longer before the error occurs, but it still does, with the same (largest) number, and without showing any differences.
I use Windows Vista x64, with 4Gb physical memory. Is it running out of address space?
Re: PB38: Comparing large binary files
Posted: 08 Dec 2009, 23:03
by therube
Thinking it must be a bug, perhaps on x64?
Cause I can compare different builds of Windows7 ISO's (so ~2.7 GB each) without a problem (& obviously hitting the 32K compare limit).
Re: PB38: Comparing large binary files
Posted: 09 Dec 2009, 12:34
by Jan Patera
mdruiter wrote:With 2.53 preview build 38, I got this frightening message:
Insufficient memory to allocate 891352880 bytes. Try to release some memory (e.g. close some running application) and click Retry. If it does not help, you can click Ignore to pass memory allocation error to this application or click Abort to terminate this application.
I believe this is only possible when comparing as Text files. Can you please double-check your are not actually comparing as Text file? If Autodetect is ON and the files are really binary, could you please send us the first megabyte of the two files? Maybe the autodetection needs some tuning.
Binary compare requires below 5MB, regardless of file size.
mdruiter wrote:When I swap the files, it takes a little longer before the error occurs, but it still does, with the same (largest) number, and without showing any differences.
It could be that it first successfully allocates memory for and reads the smaller file and then allocation for the larger one fails.
mdruiter wrote:I use Windows Vista x64, with 4Gb physical memory. Is it running out of address space?
Yes, it could be. Or just due to address space fragmentation the largest block is not large enough.
Re: PB38: Comparing large binary files
Posted: 09 Dec 2009, 13:43
by therube
I believe this is only possible when comparing as Text files.
That looks to be it.
If I force Text mode, I get the same message.
So either Text is being forced, or Autodetect is detecting as Text.
Re: PB38: Comparing large binary files
Posted: 09 Dec 2009, 13:50
by Jan Patera
therube wrote:So either Text is being forced, or Autodetect is detecting as Text.
So, it is the wrong text/binary detection.
Could you please send us the first megabyte of the two files? Maybe the autodetection needs some tuning.
Re: PB38: Comparing large binary files
Posted: 09 Dec 2009, 13:52
by Petr Solin
We must wait for mdruiter.

Re: PB38: Comparing large binary files
Posted: 09 Dec 2009, 14:18
by therube
PS: You could use the Split & Combine plugin to "make" the 1MB file, disregarding the extra pieces it would generate. Or if you have a UNIX-like version of
head, you could make the 1MB file directly.
Code: Select all
head -c 1000000 filename > outfile
Re: PB38: Comparing large binary files
Posted: 09 Dec 2009, 15:10
by mdruiter
I'm sorry, forced Text comparison was still on.

Thanks for the quick responses, but it's my fault. I mistakenly assumed that configuration was equal between 2.52 and 2.53 PB38.
Oh, and the changing window caption really helps!
Re: PB38: Comparing large binary files
Posted: 09 Dec 2009, 15:19
by Jan Patera
mdruiter wrote:I'm sorry, forced Text comparison was still on.

Thanks for the quick responses, but it's my fault. I mistakenly assumed that configuration was equal between 2.52 and 2.53 PB38.
Great

I'll try to add/change the error message to something like "The comparison in text mode failed because not enough memory is available. You may want to try binary comparison".
Re: PB38: Comparing large binary files
Posted: 09 Dec 2009, 15:32
by mdruiter
For the record: autodetecting file type compares binary, without problems.
Re: PB38: Comparing large binary files
Posted: 12 Dec 2009, 16:54
by Jan Patera
Jan Patera wrote:I'll try to add/change the error message to something like "The comparison in text mode failed because not enough memory is available. You may want to try binary comparison".
Done for the next Preview build.