PB38: Comparing large binary files
Forum rules
Please read forum rules and a few additional rules for this section:
1. Please take into account that Preview Build versions of Salamander and plugins could contain serious bugs. Don't use them in production environments.
2. All problems or comments regarding Preview Build versions please insert into this section of forum.
3. Each new topic mark with PB abbreviation and Preview Build number, for example "PB22: inactive panel below mouse cursor doesn't scroll while mouse wheel tilting".
Please read forum rules and a few additional rules for this section:
1. Please take into account that Preview Build versions of Salamander and plugins could contain serious bugs. Don't use them in production environments.
2. All problems or comments regarding Preview Build versions please insert into this section of forum.
3. Each new topic mark with PB abbreviation and Preview Build number, for example "PB22: inactive panel below mouse cursor doesn't scroll while mouse wheel tilting".
PB38: Comparing large binary files
I tried the performance of Compare Files (Ctrl+Shift+C), on two very different binary files. The large files were 891,352,865 and 417,412,266 bytes.
With Salamander 2.52, I got the usual (but confusing) "Computing Differences" in the caption, immediate results, and after a few seconds the caption said: More than 32 768 differences.
With 2.53 preview build 38, I got this frightening message:
Insufficient memory to allocate 891352880 bytes. Try to release some memory (e.g. close some running application) and click Retry. If it does not help, you can click Ignore to pass memory allocation error to this application or click Abort to terminate this application.
Can this be avoided, perhaps with the 'slower' method as a fallback?
I would really like to be able to compare large binary files...
When I swap the files, it takes a little longer before the error occurs, but it still does, with the same (largest) number, and without showing any differences.
I use Windows Vista x64, with 4Gb physical memory. Is it running out of address space?
With Salamander 2.52, I got the usual (but confusing) "Computing Differences" in the caption, immediate results, and after a few seconds the caption said: More than 32 768 differences.
With 2.53 preview build 38, I got this frightening message:
Insufficient memory to allocate 891352880 bytes. Try to release some memory (e.g. close some running application) and click Retry. If it does not help, you can click Ignore to pass memory allocation error to this application or click Abort to terminate this application.
Can this be avoided, perhaps with the 'slower' method as a fallback?
I would really like to be able to compare large binary files...
When I swap the files, it takes a little longer before the error occurs, but it still does, with the same (largest) number, and without showing any differences.
I use Windows Vista x64, with 4Gb physical memory. Is it running out of address space?
Re: PB38: Comparing large binary files
Thinking it must be a bug, perhaps on x64?
Cause I can compare different builds of Windows7 ISO's (so ~2.7 GB each) without a problem (& obviously hitting the 32K compare limit).
Cause I can compare different builds of Windows7 ISO's (so ~2.7 GB each) without a problem (& obviously hitting the 32K compare limit).
WinXP Pro SP3 or Win7 x86 | SS 2.54
-
- Plugin Developer
- Posts: 707
- Joined: 08 Dec 2005, 14:33
- Location: Prague, Czech Republic
- Contact:
Re: PB38: Comparing large binary files
I believe this is only possible when comparing as Text files. Can you please double-check your are not actually comparing as Text file? If Autodetect is ON and the files are really binary, could you please send us the first megabyte of the two files? Maybe the autodetection needs some tuning.mdruiter wrote:With 2.53 preview build 38, I got this frightening message:
Insufficient memory to allocate 891352880 bytes. Try to release some memory (e.g. close some running application) and click Retry. If it does not help, you can click Ignore to pass memory allocation error to this application or click Abort to terminate this application.
Binary compare requires below 5MB, regardless of file size.
It could be that it first successfully allocates memory for and reads the smaller file and then allocation for the larger one fails.mdruiter wrote:When I swap the files, it takes a little longer before the error occurs, but it still does, with the same (largest) number, and without showing any differences.
Yes, it could be. Or just due to address space fragmentation the largest block is not large enough.mdruiter wrote:I use Windows Vista x64, with 4Gb physical memory. Is it running out of address space?
Re: PB38: Comparing large binary files
That looks to be it.I believe this is only possible when comparing as Text files.
If I force Text mode, I get the same message.
So either Text is being forced, or Autodetect is detecting as Text.
WinXP Pro SP3 or Win7 x86 | SS 2.54
-
- Plugin Developer
- Posts: 707
- Joined: 08 Dec 2005, 14:33
- Location: Prague, Czech Republic
- Contact:
Re: PB38: Comparing large binary files
So, it is the wrong text/binary detection.therube wrote:So either Text is being forced, or Autodetect is detecting as Text.
Could you please send us the first megabyte of the two files? Maybe the autodetection needs some tuning.
-
- ALTAP Staff
- Posts: 1112
- Joined: 08 Dec 2005, 09:13
- Location: Novy Bor, Czech Republic
- Contact:
Re: PB38: Comparing large binary files
We must wait for mdruiter.
Re: PB38: Comparing large binary files
PS: You could use the Split & Combine plugin to "make" the 1MB file, disregarding the extra pieces it would generate. Or if you have a UNIX-like version of head, you could make the 1MB file directly.
Code: Select all
head -c 1000000 filename > outfile
WinXP Pro SP3 or Win7 x86 | SS 2.54
Re: PB38: Comparing large binary files
I'm sorry, forced Text comparison was still on.
Thanks for the quick responses, but it's my fault. I mistakenly assumed that configuration was equal between 2.52 and 2.53 PB38.
Oh, and the changing window caption really helps!
Thanks for the quick responses, but it's my fault. I mistakenly assumed that configuration was equal between 2.52 and 2.53 PB38.
Oh, and the changing window caption really helps!
Last edited by mdruiter on 09 Dec 2009, 15:28, edited 1 time in total.
-
- Plugin Developer
- Posts: 707
- Joined: 08 Dec 2005, 14:33
- Location: Prague, Czech Republic
- Contact:
Re: PB38: Comparing large binary files
Greatmdruiter wrote:I'm sorry, forced Text comparison was still on.
Thanks for the quick responses, but it's my fault. I mistakenly assumed that configuration was equal between 2.52 and 2.53 PB38.
I'll try to add/change the error message to something like "The comparison in text mode failed because not enough memory is available. You may want to try binary comparison".
Re: PB38: Comparing large binary files
For the record: autodetecting file type compares binary, without problems.
-
- Plugin Developer
- Posts: 707
- Joined: 08 Dec 2005, 14:33
- Location: Prague, Czech Republic
- Contact:
Re: PB38: Comparing large binary files
Done for the next Preview build.Jan Patera wrote:I'll try to add/change the error message to something like "The comparison in text mode failed because not enough memory is available. You may want to try binary comparison".