Reimaging Corrupted WIM

Comments, questions, bug reports, etc.
Post Reply
zipmagic
Posts: 64
Joined: Thu Aug 06, 2015 7:09 am

Reimaging Corrupted WIM

Post by zipmagic »

So I've got a very interesting scenario. Either due to a corrupt internal SSD, or a corrupt external thumb drive, or corrupt memory (investigations are still ongoing); I ended up with 5,002 unreadable files, and 2,318 checksum mismatched files - inside a 1 TB WIMBOOT archive which contains many millions of files.

The unreadable files are one of two error classes, as reported by running the tool you had so generously built for me probably by now a decade ago:

Code: Select all

wimboot-verify-x64.exe C:\
A device attached to the system is not functioning
or
The WOF driver encountered a corruption in WIM image's Resource Table


The checksum mismatched files seem to contain all NULLs, based on my entirely unscientific random sampling of three files out of several thousand.

Running this command:

Code: Select all

wimverify c:\diskzip.wim
Produces the following output:
[WARNING] "c:\diskzip.wim" does not contain integrity information. Skipping integrity check.
Verifying metadata for image 1 of 1
[ERROR] Failed to decompress data!00 GiB (20%) done

ERROR: "c:\diskzip.wim" failed verification!
ERROR: Exiting with error code 2:
The WIM contains invalid compressed data.
Press any key to continue . . .
Running DiskZIP's own CHKDSK tool, which only checks whether external backing is found, does not report any errors (since all externally backed files do have, an albeit corrupted, WIM on disk).

The unsafe compaction pass which produced this WIMBOOT archive did NOT report any errors in its processing log.

I do have backups of all affected files, so I plan to replace them with their healthy counterparts, and then reimage my system. I've got several questions in this regard:

1) Can I make do with an unsafe compaction pass, despite the fact that the WIM image resource table is corrupt? OR, do I have to perform a full new compaction pass, creating a brand-new WIM from scratch?

2) Is there a way to perform an offline repair on this WIM image, repairing the errors found inside of its Resource Table; while of course dropping any unhealthy files it contains?

3) Is the currently corrupt WIM image re-applicable in WIMBOOT mode? My current operating device (where I'm typing this from presently) definitely seems to suggest so; however I'm curious how this works at all, given the corruption in the WIM image Resource Table. OR, could it be the case that whatever the corruption is, it affected my device AFTER the image was applied in WIMBOOT mode?

4) I thought I'd refrain from asking which hardware component the forensic data at hand might seem to suggest is at fault, but I couldn't help myself and welcome any speculations in this regard.
zipmagic
Posts: 64
Joined: Thu Aug 06, 2015 7:09 am

Re: Reimaging Corrupted WIM

Post by zipmagic »

A follow-up to this question, I enhanced ZIPmagic's CHKDSK code to check for these additional return codes, when reading 1 byte from a WIMBOOT hosted file (per your suggestion from way back when) fails:

Code: Select all

(dw = 31) // A device attached to the system is not functioning
(dw = 4447) // The WOF driver encountered a corruption in WIM image's Resource Table
This helps ZIPmagic's CHKDSK function detect two more sources of corrupt files.

However, it still cannot detect the "CHECKSUM MISMATCH" error returned by your wimboot-verify-x64.exe tool - I suppose the only way to uncover that error is to read the whole file, the way you probably have it done in your code?

OR, could a more efficient implementation be possible?
synchronicity
Site Admin
Posts: 480
Joined: Sun Aug 02, 2015 10:31 pm

Re: Reimaging Corrupted WIM

Post by synchronicity »

Given that the corruption is in the WIM resource table, there isn't really anything that can feasibly be done other than delete the WIM file and start over.

The resource table is basically a list of (SHA-1 hash, offset in file, uncompressed size, compressed size) tuples which list where to find the file data in the WIM file. This information is not stored anywhere else in the WIM. It's *not* like a ZIP file where the equivalent information is stored both in a central directory (equivalent to the WIM resource table) and in a header preceding each file's data.

The only way to recover it, in theory, would be brute force.
1.) Try each resource entry and figure out which ones are corrupted, i.e. which ones don't give data that matches their listed SHA-1 hash
2.) Build a list of every byte in the WIM file that isn't already known to be part of a resource. This could be billions of entries long.
3.) If the image metadata resources weren't found yet, then find them by starting a decompression at every possible starting position (potentially billions). Assuming LZX compression, a valid LZX stream would have a sequence of blocks. After each block successfully decompressed, hypothesize that it's the end of the resource, generate the corresponding uncompressed data, and try to parse the uncompressed data as a WIM image metadata resource. Hopefully you'd end up with exactly one image metadata resource per image, matching the image count given in the WIM file header. In a WIM file containing multiple images, it may be ambiguous which image is which.
4.) From the WIM image metadata resources, collect the set of SHA-1 hashes of every file in the WIM.
5.) Use the same method as in (3) to brute-force check every possible resource location. For each calculate the SHA-1 hash and hope that it matches one of the needed ones. This again could require a computationally expensive check at billions of positions.
6.) If SHA-1 hashes still remain then check for uncompressed resources at every possible remaining starting position, which could be even more computationally expensive. That should cover the remaining ones. But it won't if there was corruption in other places in the WIM file, in which case some will still be unrecoverable.

It might work, in theory. For a large WIM file, it might not be computationally feasible, given the billions of checks that would need to be done to check every possible position. And it's very possible you'd find there was also corruption in another part of the file, not just the WIM resource table, so not everything could be recovered anyway.

My opinion is don't bother. Just delete the WIM file and start over.

Code: Select all

However, it still cannot detect the "CHECKSUM MISMATCH" error returned by your wimboot-verify-x64.exe tool - I suppose the only way to uncover that error is to read the whole file, the way you probably have it done in your code?

OR, could a more efficient implementation be possible?
FSCTL_GET_EXTERNAL_BACKING returns the expected SHA-1 hash of the file, so the verification that wimboot-verify does is just calculate the SHA-1 hash of the file and verify that it matches the expected value. There isn't any shortcut to do this.
zipmagic
Posts: 64
Joined: Thu Aug 06, 2015 7:09 am

Re: Reimaging Corrupted WIM

Post by zipmagic »

Yeah sounds crazy. Since everything else does seem to work (no checksum or other errors) for 99.99% of the image (for now), I'll just replace the broken files with their backups (eliminating the broken WIM backings in the process) and do a safe compaction pass (recompressing the whole disk from scratch). Thank you!
Post Reply