PB38: Comparing large binary files

[mdruiter]

I tried the performance of Compare Files (Ctrl+Shift+C), on two very different binary files. The large files were 891,352,865 and 417,412,266 bytes.
With Salamander 2.52, I got the usual (but confusing) "Computing Differences" in the caption, immediate results, and after a few seconds the caption said: More than 32 768 differences.
With 2.53 preview build 38, I got this frightening message:
Insufficient memory to allocate 891352880 bytes. Try to release some memory (e.g. close some running application) and click Retry. If it does not help, you can click Ignore to pass memory allocation error to this application or click Abort to terminate this application.

Can this be avoided, perhaps with the 'slower' method as a fallback?
I would really like to be able to compare large binary files...

When I swap the files, it takes a little longer before the error occurs, but it still does, with the same (largest) number, and without showing any differences.
I use Windows Vista x64, with 4Gb physical memory. Is it running out of address space?

[therube]

Thinking it must be a bug, perhaps on x64?
Cause I can compare different builds of Windows7 ISO's (so ~2.7 GB each) without a problem (& obviously hitting the 32K compare limit).

[Jan Patera]

With 2.53 preview build 38, I got this frightening message:
Insufficient memory to allocate 891352880 bytes. Try to release some memory (e.g. close some running application) and click Retry. If it does not help, you can click Ignore to pass memory allocation error to this application or click Abort to terminate this application.

I believe this is only possible when comparing as Text files. Can you please double-check your are not actually comparing as Text file? If Autodetect is ON and the files are really binary, could you please send us the first megabyte of the two files? Maybe the autodetection needs some tuning.
Binary compare requires below 5MB, regardless of file size.

When I swap the files, it takes a little longer before the error occurs, but it still does, with the same (largest) number, and without showing any differences.

It could be that it first successfully allocates memory for and reads the smaller file and then allocation for the larger one fails.

I use Windows Vista x64, with 4Gb physical memory. Is it running out of address space?

Yes, it could be. Or just due to address space fragmentation the largest block is not large enough.

[therube]

I believe this is only possible when comparing as Text files.

That looks to be it.
If I force Text mode, I get the same message.
So either Text is being forced, or Autodetect is detecting as Text.

[Jan Patera]

So either Text is being forced, or Autodetect is detecting as Text.

So, it is the wrong text/binary detection.
Could you please send us the first megabyte of the two files? Maybe the autodetection needs some tuning.

[Petr Solin]

We must wait for mdruiter.

[therube]

PS: You could use the Split & Combine plugin to "make" the 1MB file, disregarding the extra pieces it would generate. Or if you have a UNIX-like version of head, you could make the 1MB file directly.

Code: Select all

head  -c  1000000  filename  >  outfile

[mdruiter]

I'm sorry, forced Text comparison was still on.
Thanks for the quick responses, but it's my fault. I mistakenly assumed that configuration was equal between 2.52 and 2.53 PB38.

Oh, and the changing window caption really helps!

[Jan Patera]

I'm sorry, forced Text comparison was still on.
Thanks for the quick responses, but it's my fault. I mistakenly assumed that configuration was equal between 2.52 and 2.53 PB38.

Great
I'll try to add/change the error message to something like "The comparison in text mode failed because not enough memory is available. You may want to try binary comparison".

[mdruiter]

For the record: autodetecting file type compares binary, without problems.

[Jan Patera]

I'll try to add/change the error message to something like "The comparison in text mode failed because not enough memory is available. You may want to try binary comparison".

Done for the next Preview build.