WimLib is crashing on big Files

[MyNameistoLong]

Hello and thank for your great Software,

i use it sind v.1.6 without bigger problems and realy like it.
But now i have a problem i can't solve. I try to build a big Windows All in One WIM with all Versions and Editions that i use to install on different customer PCs. In a frist step i put all Editions of a Version in a WIM.
Windows Vista.wim
Windows 7.wim
Windows 2008.wim
.....

After that, i put all Version WIMs in one MEGA WIM
Until here all works fine. The File is about 44 GB compressed and ~116GB unkompressed.

Now i try to rebuild and compress to the smallest size possible with

Code: Select all

wimlib-imagex.exe export "Win_MEGA_AIO.wim" ALL "F:\Win_MEGA_AIO_OPTIMUM.wim" --solid-compress=LZMS:100 --check --rebuild --recompress --solid

I'am using Wimlib 1.94Beta9 and after 91% is done, wimlib crashes (i've tried multiple times)
Here is a Screenshot.


The Warnings are from the Windows 7 WIMs. I don't know why, because i use the original Microsoft ISOs with corret Checksums.

Someone have an idea? Is the any Limit in Size or Images a WIM can contain?

[MyNameistoLong]

OK, i can give you an Update on this and maybe you can confirm the solution.

The source WIM (not optimized) is about 44 GB in size and the destination file is to be saved on a disk with about 45 GB free space.
With this, there was an reproduceable error at 91%.

I tried to save to another disk with about 200 GB free space and now it was successful
Every test take some hours so i'am not sure if this is the problem.

Is it possible that there must be more free space left for some temp files?
After optimizing, the file size is only about 25,5 GB

[synchronicity]

First of all, the warnings about invalid hard links are expected and you can ignore them. They appear because wimlib had to apply a workaround to correctly read a WIM created by a buggy version of Microsoft's ImageX. I am considering just deleting this warning since it's not really "actionable", and people may think it indicates a problem on wimlib's side.

There is no meaningful limit on the number of images or size of a WIM file, but you may encounter resource limitations such as memory or disk space. However, I don't think memory or disk space was a limiting factor in your situation. According to your second screenshot, the system had 31.7 GB of memory, and based on your numbers, the destination filesystem had easily enough free space to accomodate the output file. wimlib doesn't use any temporary files while exporting. And even if there was a resource limitation, wimlib-imagex should never simply crash --- although it can only be as reliable as the operating system on which it runs.

If you're interested in reproducing the problem again and providing more information, it may be possible to set up your Windows system to collect full crash dumps, and provide me with the crash dump and any other relevant details. See: https://msdn.microsoft.com/en-us/librar ... s.85).aspx). If you do this, please test with the final release of wimlib v1.9.0 (64-bit version, of course). Please note that I am not an expert with Windows crash dumps so I can't really provide more help with the process of collecting the dump itself.

[MyNameistoLong]

Thanks for your fast reply.
I will give it a try but it take some time.

[MyNameistoLong]

So here is an Update.

First i have so say sorry, but what i was writing in my second post was a mistake.

I tried to save to another disk with about 200 GB free space and now it was successful

It looks like it was successful but it turn out, that it climbs up to 100% but won't finish. There was an crash Window like in the first Screenshot (but in the background).

In the last run (it was still 1.9.beta9 because it was running before i read your post)
I've entered the registry Keys like in the MS Article but it won't work for me.
The App Crashes again, now at 100% (this time there was ~90 GB free Space on the disk before starting imlib-Imagex).


I created Memory Dumps of the Process before closing the Window with the Error Report with
Taskmanager
Process Explorer (full Dump won't work)
Procdump with Option MA und ML

The Full Dump is abount 10 GB in size and i think its not necessary but if you need it, i would upload it.
The smaller Dumps are https://goo.gl/VGvhYv for download.

Now i try again with v.1.9.0

EDIT: OK, 1.9.0 crashes on 100% too

[synchronicity]

Thanks. Unfortunately I don't have much chance of tracking down this problem if you can't provide a smaller example that reproduces it. The crash isn't even in wimlib but rather in ntdll, so it could be memory corruption or pretty much anything else that occurred prior to the point of the crash... and if you have any hardware or environment problems you could potentially get crashes like this without a bug in wimlib itself (running this compression algorithm for multiple hours will be very sensitive to errors).

I do have another build you can try if you're (very) interested:

https://wimlib.net/downloads/testing/wi ... 64-bin.zip

It includes some extra logging output (it goes to stderr; you can redirect it to some FILE with 2>FILE), debugging symbols, and a change I made to reduce memory consumption when exporting large numbers of images.

I don't really know what to suggest doing if it takes multiple hours to run each test. I guess that if you are so inclined, you could try running variations on your command (e.g. use wimoptimize instead of wimexport; or use compression level 50 instead of 100; or just export one image instead of all; or using LZX compression), seeing which ones crash, and provide as much information as you can.

[MyNameistoLong]

Great, and thanks for your help.
I downloaded the new Version and run a new Job. I will try to give you the Informations you need. I'am very interestest in solving this because the machine its running on is a VM Host and a Hardware Problem can affect more than one machine.
In the afternoon, i can try to run the same Job on another machine but i have to reduce the threads to 4 because it is a 8Core / 16GB Machine.
I will post every Update.

Thanks for your help.

[MyNameistoLong]

OK, here is an update.
Windows Server 2012 R2
32 GB RAM
Wimlib-ImageX 1.9.1 Beta Build

Comandline

Code: Select all

wimlib-imagex.exe  export "B:\Win_MEGA_AIO.wim" ALL "F:\Win_MEGA_AIO_OPTIMUM.wim" --solid-compress=LZMS:100 --check --rebuild --recompress --solid 

This Version uses much less RAM. Its using about 2,8 GB instead of 10-11 GB (v.1.9.0). But it seems that its running much longer (a few hours).
Maybe i've made some mistake but the redirection with 2>DebugWimLib.txt does not work.
The File is created but empty after the Crash. But there are a few thousand lines of output in the cmd Window and i have samed the last 9999 Lines.

At the end, this Version crashes too with the same message.

If you are interesed in the Dump i can upload it again but i will try a few other settings as you suggested, another Machine and maybe it works with wimlib optimze.

[synchronicity]

I'm very suspicious of the error codes you are seeing:

C0000006 = STATUS_IN_PAGE_ERROR
C00000C4 = STATUS_UNEXPECTED_NETWORK_ERROR

Is either the source WIM file or wimlib-imagex located on a network filesystem, or any sort of device which might be producing I/O errors?

You should also try running the following; it's a readonly operation that won't take as long as an export:

Code: Select all

wimverify B:\Win_META_AIO.wim

That should complete successfully.

[MyNameistoLong]

Now it gets interesting

The last run was executed on my Workstation
Windows 7, 16 GB RAM using Wimlib ImageX 1.9.0 final (because its faster and the Beta was crashing too)
This job was finishing sucessful

Now a little bit more to my environment.

The Server with the errors is a self made system with 32 GB and a few Harddisks.
Its hosting about 5 always running (DC, DHCP, DNS / VPN Server / File Server / Plex / Work...) and a few temporary running VMs.
One VM is a Fileserver and the Clients (in this case the VM Host) are accessing the Files over a SMB / DFS Share mapped as Drive Z:\

So the answer to your question "is WImlib running from a network drive is" -> YES
The Wim Files however are not accessed through the Share. They are on an external drive that is connected over a USB 3 HDD docking station on the VM Host.
This Setup is more stable than i thought and i have only one or zwo crashed over about 2 years of usage (and another 2 or 3 because of power failures and i can't remember any big problems with accessing the Files in the Fileserver.

The sucessful job on my Workstation was accessing Wimlib over Drive Z: (my Netzwork drive).

Maybe the following happens on the Server:
-I was running wimlib from Network drive
-everything works fine and there are much disk IO / memory and CPU useage on the VM Host
-there is so much load on the host, that die FileServer VM gets no resources and the SMB session times out
-at the end of processing WIM Files, WimLib tries to do something that needs to access the wimlib exe or dll (loading a funktion ?!?)
-the Session is closed and the access is not possible and the process craches

OR
i have a hardware failure (memory?) on the Server and i really hope everything is OK.


Now i start a job with Wimlib exe and dll located on the lokal USB Disk with the WIM Files.

I'am so sorry that you troubleshoot my mistakes.
Can i donate a few dollars for your help and the great tool?