[Solved] WimLib random crash when compressing @ LZMS:100, solid, 128M chunk with varying thread count

Comments, questions, bug reports, etc.
Vulpix
Posts: 10
Joined: Fri Jan 25, 2019 7:01 am

Re: WimLib random crash when compressing @ LZMS:100, solid, 128M chunk with varying thread count

Post by Vulpix » Fri Apr 19, 2019 6:47 pm

Unfortunately, the issue is still there; seems to be the same as before. I don't know why the above compression worked, maybe I was just lucky...

I don't really know why it's possible to get such random results though. I wish I could better help you debug this somehow, but I don't know what other information I can supply.

synchronicity
Site Admin
Posts: 301
Joined: Sun Aug 02, 2015 10:31 pm

Re: WimLib random crash when compressing @ LZMS:100, solid, 128M chunk with varying thread count

Post by synchronicity » Tue May 07, 2019 2:56 am

I don't think I can figure out the remaining problem(s) anytime soon without being able to reproduce it myself. I don't have much to go on, and I have limited time. So for now I went ahead and released v1.13.1 with the fix I mentioned, so at least one bug is fixed.

I don't suppose there's any chance you're actually experiencing a hardware problem? The compression workload in question is extremely sensitive to memory errors and uses a huge amount of memory. If there's a bitflip anywhere at any point, all bets are off.

Vulpix
Posts: 10
Joined: Fri Jan 25, 2019 7:01 am

Re: WimLib random crash when compressing @ LZMS:100, solid, 128M chunk with varying thread count

Post by Vulpix » Thu May 09, 2019 9:57 am

Hi!

Well, I may have mentioned this somewhere along these posts but I don't think it's hardware related. Not just because it generally crashes at the same point when it does, but also because the system in question has an uptime of several months and I have had no issues with anything else (including other memory intensive applications). I did run a memory stress test on it for two days just now (aw, my uptime is ruined! :D), but it has found no issues.

I have 128GB of ram and it's true a lot of it is taken by the compression, but I have other programs that use a lot of memory as well and they never complained.

I suppose the only way to truly know is if I had ECC memory, which I unfortunately do not...

Nonetheless thank you for your work! I'll try to figure out some setting which provides good compression but does not crash, although I'm a bit sad I cannot use the "highest" settings :D One day perhaps!

Vulpix
Posts: 10
Joined: Fri Jan 25, 2019 7:01 am

Re: [Solved] WimLib random crash when compressing @ LZMS:100, solid, 128M chunk with varying thread count

Post by Vulpix » Wed May 15, 2019 12:38 pm

So it turns out that while everything was fine with the system when I built it, when I re-ran my stability tests again yesterday (from within windows rather than using the bootable memtest) to make sure everything is OK, I had a few memory "decay" errors (reading memory after writing it, after a certain period of time, did not match the written pattern). The test has failed, which clearly means there was a problem with memory - and as you said, the code there is pretty sensitive to something like that. I have narrowed down and replaced a faulty memory module a few hours ago and all of the commands that previously crashed now work just fine.

It's funny that this must have been the case for a while now, but none of my other memory-intensive programs, virtual machines or other software complained. I ran some pretty heavy stuff as well, hah.

Guess I'm adding wimlib to my test suite from now on! :D

Post Reply