Re: Unpack 7z archive fails due to out of memory problem

mdruiter · Post by **mdruiter** » 01 Jun 2010, 18:03

With some (old and new) 7z archives, Unpack archive does not seem to work.
I just press Alt+F6 on it and Enter, and I get:
---------------------------
Altap Salamander 2.53 beta 2
---------------------------
Insufficient memory to allocate 1231194888 bytes. Try to release some memory (e.g. close some running application) and click Retry. If it does not help, you can click Ignore to pass memory allocation error to this application or click Abort to terminate this application.
---------------------------
Abort Retry Ignore
---------------------------

That's not what I expected. :/
Copying files from the archive does work normally.

An example 7z is attached.

Post by **zarevak** » 01 Jun 2010, 19:24

I've reported problems with memory management in 7zip plugin about a year ago in this Czech thread: 7-Zip komprese - málo paměti (fragmentace, /3GB) (althought it was about compression)

7Zip plugin is based on 7Zip application, which doesn't have problems allocating large blocks of memory. Salamander on the other side has memory space fragmented because of all the different libraries loaded into it and cannot find continuous free block of 1GB size.
Sadly the memory allocation routines are deep in the 7zip codebase and would be very hard to rewrite to allocate the required memory in smaller chunks.

But I may be wrong and your archive may be just corrupted

Slanec · Post by **Slanec** » 01 Jun 2010, 19:28

...but then

Copying files from the archive does work normally.

wouldn't be true. I can't really see a difference between unpacking and copying files out of an archive.

Ether · Post by **Ether** » 02 Jun 2010, 00:46

Unfortunately, it's not caused by corrupted archives. I, too, have encountered a 7-zip archive created by AS that required circa half a gigabyte of memory to unpack (luckily, I was able to free this ridiculous amount of memory and unpack the archive). I can send it by email if required.

On the other hand, I cannot reproduce the problem with mdruiter's attached archive. The files extract just fine using both Alt+F9 on archive and F5 from archive, without any memory hogging or swapping (using simple Task Manager and disk activity observation).

Post by **zarevak** » 02 Jun 2010, 03:12

Just to clarify the memory allocation which is done in two steps:
1) reservation - regular 32-bit program can reserve about 2GB of memory without any actual memory being used or backed up by the page file....
2) commiting - this step makes the memory actualy usable

The reservation system is useful in cases when programmers need to use continuous memory blocks, but don't have the need to use them completely.
- program can reserve 1GB block and then can use basic pointer arithmetics and memory indexing... only the parts which are actualy used can commited
-- array item number 1000 is on memory address:

arrayPtr + 1000 * sizeof(arrayItem)

- easy

- drawback is the reserved block needs to have free continuous address space - if the memory is too fragmented, there may not be enough space to reserve the block (1GB is very large piece in 4GB address space) - the problem almost doesn't exist in 64-bit world, because the address space is huge and 1GB is very small part...

If the program would only reserve and commit smaller blocks, it would need to have much more complicated pointer arithmetics and would require some table of all allocated blocks...
-- array item number 1000 is on memory address:

blocksPtr[1000 / items_in_block] + (1000 % items_in_block) * sizeof(arrayItem)

- much harder and slower

More information about memory allocation can be found in Mark Russinovich's blog post: Pushing the Limits of Windows: Virtual Memory

I don't know the internals of 7-zip code, but I wanted to point out, that program can fail to allocate 1GB of memory even when it will actually use only 1MB of this memory...

Post by **Jan Patera** » 02 Jun 2010, 08:13

Slanec wrote:...but then
Copying files from the archive does work normally.
wouldn't be true. I can't really see a difference between unpacking and copying files out of an archive.

There can be diference if the archive is not solid and mdruiter is copying out of it only some files.

Post by **Jan Patera** » 02 Jun 2010, 08:18

mdruiter wrote:With some (old and new) 7z archives, Unpack archive does not seem to work.
> Insufficient memory to allocate 1231194888 bytes.

Yes, this can happen on PPMD-compressed files with very high dictionary and word sizes. See Zarevak's notes & thread.
However, the largest block needed to extract your particular file 1.7z is just 1MB, it only uses LZMA and only with small dictionary size (see the detailed view). Are you sure it is the right example?

mdruiter · Post by **mdruiter** » 02 Jun 2010, 12:20

I still have the problem on this (solid) archive.
Copying all (two) files works just fine.

Wait, once I restarted Salamander, Unpack worked again. So apparently it depends on (memory?) state...

I'm using Windows Vista x64, by the way.

How can I debug this problem further, if I encounter it again?

Post by **Jan Rysavy** » 02 Jun 2010, 12:33

We know about this problem, it looks there isn't "easy" solution for now.
The x64 version of Altap Salamander will solve it.

Starting a new instance of Salamander, where plugins are not loaded (default state of most AS plugins) could help you now.

mdruiter · Post by **mdruiter** » 02 Jun 2010, 12:43

We know about this problem, it looks there isn't "easy" solution for now.

Jan, do you understand why it asks for so much memory for so small a file?

Post by **Jan Patera** » 02 Jun 2010, 12:55

mdruiter wrote:
We know about this problem, it looks there isn't "easy" solution for now.
Jan, do you understand why it asks for so much memory for so small a file?

No. As I said, the largest block the 7zip plugin and dll try to allocate is 1 MB. Have you tried older versions?

mdruiter · Post by **mdruiter** » 02 Jun 2010, 13:20

I meant to ask Jan Rysavy.

Post by **Jan Rysavy** » 02 Jun 2010, 15:06

I have no idea, Jan Patera knows all about 7-Zip plugin, he did all patches and maintenance for several years.

mdruiter · Post by **mdruiter** » 02 Jun 2010, 20:51

I'm confused.

From Jan Rysavy I understand, that the source of the problem is known. And that it's related to x64, which sounds sensible. At least I run Vista x64 too.
But then again, why does the plugin ask for the enormous amount of memory? How could that be related to Windows-bitness?

As I'm not as convinced as you might be that the problem is understood, I'ld like to know whether I could help to debug it any further, if it occurs again...

Post by **Jan Rysavy** » 02 Jun 2010, 21:05

I looks like my mistake, I'm sorry. Please ignore my post about known problem.