Read Error Recovery for Unsafe Compaction

[zipmagic]

When wimlib is being consumed from Windows PE/RE, sometimes a read error occurs:

WhatsApp Image 2026-02-05 at 21.29.00.jpeg (96.2 KiB) Viewed 83706 times

Which is always fatal during unsafe compaction passes:

WhatsApp Image 2026-02-05 at 00.23.13.jpeg (127.69 KiB) Viewed 83706 times

The thing with these types of failures is, they are almost always intermittent. The above examples were on two different OS instances - both on different sets of hardware (Intel 285HX and AMD 9955HX3D, Phison vs. WD SSD), albeit identical base OS image. While a hardware error cannot be ruled out, it is more than likely a transient failure in Windows and/or the OS image and/or the updates that broke something in it.

I have requested in the past that a workaround be instituted in these scenarious, so as to not render the WIMBoot volume unbootable after this failure. The requests were always turned down, due to the inherent difficulty of reconciling all chunks affected by the read error (it's not just a single file potentially - although the progress report makes it look like it is - instead, it is more often a shared chunk of data that may be in use by countless other files, thanks to built-in data deduplication).

Claude AI to the rescue, then:

wimlib-1.14.5-correct-fix-all-files.zip

(1.13 MiB) Downloaded 3595 times

Would it be possible to check whether Claude did a decent enough job here in providing the requested recovery option for us here?

[zipmagic]

Also, here should be just the modified files, assuming Claude AI hasn't hallucinated too badly, of course:

wimlib-1.14.5-correct-fix-modified-files.zip

(111.1 KiB) Downloaded 3796 times

I truly hope this would be the healing salve for these kinds of (wishfully formerly) fatal read errors?

[zipmagic]

Been working on this since my last post - it's been more than a month already!

Using fashionable vibe coding, I got it to the point where it'd create a WIM that works, for a given definition of works:

- The WIM file is created and the process doesn't abort
- The WIM file can be extracted using DiskZIP (which is based on 7-Zip), but there's many extraction errors on corrupt files (as expected)
- The WIM file can NOT be extracted - either in literal mode or in WIMBOOT mode - with wimlib

Claude tried to fix the extraction, but hasn't had any luck with it.

I consider it a rarity that it was able to fix compression at all - most Claude sessions failed miserably, despite repeated efforts.

Happy to share the vibe coded sources if there's any interest?

I'm wondering if it may make more sense to just attempt to pre-read files and bail if that doesn't work - although, that still wouldn't help with intermittently surfacing read errors due to underlying hardware issues and whatnot.

If this approach is condoned - what kind of code would make sense for trying reads? After all, WIMLIB is already scanning the drive before compression - what extra thing could we be doing that it doesn't pick up to begin with?

I fortunately have two devices - one physical, the other a VM - where I have two different kinds of stable read errors for testing.

The physical machine has an image I keep re-using from backups as necessary, ditto the VM itself, of course.

[zipmagic]

Got it working!!!

[synchronicity]

While I'm interested in better support for recovering from errors, it would be a big change, and I don't work very much on this project anymore. I would also expect any "vibe-coded" patch to have a human who has reviewed the patch, fully understands it, has improved it further where applicable, and stands by it. It's not clear that's the case here.

[zipmagic]

Great to hear from you!

And I share your concerns 100%. While the AI has achieved:

- Ability to apply (in WIMBOOT mode) a WIM which has capture errors in it, extracting the damaged files as zero byte files
- Ability to continue verifying a WIM (via wimverify), when the WIM has one or more damaged blobs in it
- Ability to continue capturing a WIM, when one or more files cannot be read
- Ability to unsafely compact a captured WIM with errors

It hasn't been able to:

- Guarantee minimum damage to the WIM file (each capture on the same "problem VM" results in a random number of damaged files, with damage log data ranging between 140 KB - 1.7 MB for reference)
- Guarantee skipping extracting damaged files (instead of extracting them as zero byte files) when they already pre-exist on disk
- Guarantee that all types of potential read/access errors have been protected and are fully addressed by the fixes
- Guarantee that the changes don't actually break anything (to say nothing of injecting useless slop)

While the work done so far shows real promise and seems to work in our test case, we've been going around in circles with Claude for several months now, with signs that it may be hallucinating unnecessary fixes, making unnecessary changes, without actually solving the remaining issues above.

It would be great to have feedback from the project owner on this, as I am absolutely unfamiliar with the codebase.

I have several copies of changed files available - from the latest fixed versions, to the very first pseudo-stable versions to have first reliably achieved our objectives above. Even getting Claude to correctly log each damaged file path was a major hassle, and none of these later fixes are part of the very early code changes I have published above.

Would you like me to email some or all of the files or upload them here for an investigation?

The changes really are rather localized and not extensive (which itself is sort of a concern), and we applied them on the latest version of your codebase.