[SOLVED] Issue w wimlib 1.8.1 and later (deploying Win7)

[techryda]

This issue is continued from the comments section over at https://aur4.archlinux.org/packages/wimlib

So, I just tried to capture and deploy a Windows 7 image using wimlib 1.8.2-BETA2

Steps I've taken

• Installed Win 7
• Sysprepped Win 7
• Captured Win 7 using ntfsclone (To be able to quickly get back to a known good image)
• Captured Win 7 using wimlib 1.8.2-BETA2
• Format partition
• Deploy image using wimlib 1.8.2-BETA2
• Same boot loop described before
• Downgrade wimlib to 1.8.0
• Format partition
• Deploy same image using wimlib 1.8.0
• PC boots into Windows

I'm going to deploy the image to same system (/dev/sda1 and /dev/sda2 )
using ntfsclone and wimlib 1.8.2-BETA2 and do a diff as you suggested.

I'll report back with what I find.

[techryda]

Well, it's good to verify you're not crazy every so often...and I just have

Ran a "diff -qr" between the two partitions (/tmp/sdb3 is my "pristine" partition, /tmp/sdb4 was captured and restored with wimlib 1.8.2-BETA2)

Here are results from the diff:

Code: Select all

Only in /tmp/sdb3: hiberfil.sys
Only in /tmp/sdb3: pagefile.sys
Files /tmp/sdb3/Program Files/paint.net/Staging/PaintDotNet_x86_1468493438.msi and /tmp/sdb4/Program Files/paint.net/Staging/PaintDotNet_x86_1468493438.msi differ
Only in /tmp/sdb3: System Volume Information
Only in /tmp/sdb3/Windows: CSC
Files /tmp/sdb3/Windows/Microsoft.NET/Framework/v4.0.30319/SetupCache/v4.5.50938/NetFx_Full_GDR.mzz and /tmp/sdb4/Windows/Microsoft.NET/Framework/v4.0.30319/SetupCache/v4.5.50938/NetFx_Full_GDR.mzz differ
Files /tmp/sdb3/Windows/Microsoft.NET/Framework/v4.0.30319/SetupCache/v4.5.50938/NetFx_Full_LDR.mzz and /tmp/sdb4/Windows/Microsoft.NET/Framework/v4.0.30319/SetupCache/v4.5.50938/NetFx_Full_LDR.mzz differ
File /tmp/sdb3/Windows/System32/config/systemprofile/AppData/LocalLow/Microsoft/CryptnetUrlCache/Content/94308059B57B3142E455B38A6EB92015 is a fifo while file /tmp/sdb4/Windows/System32/config/systemprofile/AppData/LocalLow/Microsoft/CryptnetUrlCache/Content/94308059B57B3142E455B38A6EB92015 is a fifo

And here are directory listings for the "Different" files

PaintDotNet_x86_1468493438.msi

Code: Select all

-rwxrwxrwx 2 root root 28012544 Dec  7  2014 /tmp/sdb3/Program Files/paint.net/Staging/PaintDotNet_x86_1468493438.msi
-rwxrwxrwx 2 root root 0 Dec  7  2014 /tmp/sdb4/Program Files/paint.net/Staging/PaintDotNet_x86_1468493438.msi

NetFx_Full_GDR.mzz

Code: Select all

-rwxrwxrwx 2 root root 213393078 Sep 12  2013 /tmp/sdb3/Windows/Microsoft.NET/Framework/v4.0.30319/SetupCache/v4.5.50938/NetFx_Full_GDR.mzz
-rwxrwxrwx 2 root root 0 Sep 12  2013 /tmp/sdb4/Windows/Microsoft.NET/Framework/v4.0.30319/SetupCache/v4.5.50938/NetFx_Full_GDR.mzz

NetFx_Full_LDR.mzz

Code: Select all

-rwxrwxrwx 2 root root 213393078 Sep 11  2013 /tmp/sdb3/Windows/Microsoft.NET/Framework/v4.0.30319/SetupCache/v4.5.50938/NetFx_Full_LDR.mzz
-rwxrwxrwx 2 root root 0 Sep 11  2013 /tmp/sdb4/Windows/Microsoft.NET/Framework/v4.0.30319/SetupCache/v4.5.50938/NetFx_Full_LDR.mzz

94308059B57B3142E455B38A6EB92015

Code: Select all

prwxrwxrwx 2 root root 0 Jul 13  2009 /tmp/sdb3/Windows/System32/config/systemprofile/AppData/LocalLow/Microsoft/CryptnetUrlCache/Content/94308059B57B3142E455B38A6EB92015
prwxrwxrwx 2 root root 0 Jul 13  2009 /tmp/sdb4/Windows/System32/config/systemprofile/AppData/LocalLow/Microsoft/CryptnetUrlCache/Content/94308059B57B3142E455B38A6EB92015

As you can see, for the first three files the size of data is zero in the wimlib restored partition, so something happened when restoring.
Not sure what the issue is on the last one but it's a special file so I'll leave that to the expert (you).

I'll give you any other data you need later but hopefully this is helpful

Thanks for your assistance.

[techryda]

Not sure if this is helpful but I also did a capture (and apply) of the same filesystem with wimlib 1.8.0 and did a diff of it against the "pristine" partition also.

/tmp/sdb3 is the "Pristine" copy
/tmp/sdb1 is the wimlib 1.8.0 applied filesystem

Here are the results:

Code: Select all

Only in /tmp/sdb3: hiberfil.sys
Only in /tmp/sdb3: pagefile.sys
Only in /tmp/sdb3: System Volume Information
Only in /tmp/sdb3/Windows: CSC
File /tmp/sdb1/Windows/System32/config/systemprofile/AppData/LocalLow/Microsoft/CryptnetUrlCache/Content/94308059B57B3142E455B38A6EB92015 is a fifo while file /tmp/sdb3/Windows/System32/config/systemprofile/AppData/LocalLow/Microsoft/CryptnetUrlCache/Content/94308059B57B3142E455B38A6EB92015 is a fifo

[synchronicity]

Hi techryda,

Thanks for the additional information, and welcome to the new web site! (By the way, I don't yet have email set up on this server, so you won't get emails when posts are made to this forum.) Just to clarify: you captured a WIM image with wimlib v1.8.2-BETA2 and applied it with wimlib v1.8.0, and everything worked properly? If so, the problem must be on the extraction side.

I haven't yet thought of what might be going wrong with the different files. Assuming that the problem is with extraction, one thing you could provide which would be very helpful is a full metadata dump from the WIM image. There isn't a super easy way to do this (yet) but you can run:

Code: Select all

wiminfo --lookup-table WIMFILE | less

and copy off the first 7 lines (they should list WIM_RESHDR_FLAG_METADATA in their flags). Then extract the corresponding compressed metadata from the WIM file. For example, with

Code: Select all

Hash              = 0xb10328e4b1df7b2cf41983f46c1c18d6051888ad
Uncompressed size = 11715840 bytes
Compressed size   = 2450948 bytes
Offset in WIM     = 3845939848 bytes
Part Number       = 1
Reference Count   = 1
Flags             = WIM_RESHDR_FLAG_COMPRESSED  WIM_RESHDR_FLAG_METADATA  

run:

Code: Select all

dd if=WIMFILE ibs=2450948 count=1 iflag=skip_bytes skip=3845939848 of=metadata.bin

and send the metadata.bin file. This will have all the filenames and other metadata but no contents.

[techryda]

Just to clarify: you captured a WIM image with wimlib v1.8.2-BETA2 and applied it with wimlib v1.8.0, and everything worked properly?

That is correct. I captured the image w/ 1.8.2, deployed it w/ 1.8.2, that didn't work. So, on a whim, I downgraded wimlib to 1.8.0, deployed the same image (captured w 1.8.2) and it booted. When you ask if everything worked properly, I assume so, but I wouldn't put it into production. It booted, I logged in....that's about it.

Hopefully I'll be able to get that metadata to you later today. I'll try to get metadata from both wim's, in case that's useful.

[techryda]

MetaData export from wimlib 1.8.0 captured image

Code: Select all

wiminfo --lookup-table Win7-captured-w-wimlib-1.8.0.wim | head -n 7

Hash                = 0xe333c761ed5cfce125e3e9691518b02f94b68996
Uncompressed size   = 13582456 bytes
Compressed size     = 3251690 bytes
Offset in WIM       = 4049803674 bytes
Part Number         = 1
Reference Count     = 1
Flags               = WIM_RESHDR_FLAG_COMPRESSED  WIM_RESHDR_FLAG_METADATA

Code: Select all

dd if=Win7-captured-w-wimlib-1.8.0.wim ibs=3251690 count=1 iflag=skip_bytes skip=4049803674 of=metadata-from-1.8.0-wimfile.bin

MetaData export from wimlib 1.8.2-BETA2 captured image

Code: Select all

wiminfo --lookup-table Win7-captured-w-wimlib-1.8.2.wim | head -n 7

Hash                = 0x8603e6d9611defb623257a9cdddc828da9a757b1
Uncompressed size   = 13586216 bytes
Compressed size     = 3076962 bytes
Offset in WIM       = 4049803674 bytes
Part Number         = 1
Reference Count     = 1
Flags               = WIM_RESHDR_FLAG_COMPRESSED  WIM_RESHDR_FLAG_METADATA

Code: Select all

dd if=Win7-captured-w-wimlib-1.8.2.wim ibs=3076962 count=1 iflag=skip_bytes skip=4049803674 of=metadata-from-1.8.2-wimfile.bin

Unfortunately I get an "Invalid File Extension" error when I try to attach these metadata dumps to this post (I tried renaming the files with several other extensions .txt, etc, ith still wouldn't let me).. If you could fix that problem or give me another way to submit them I'll get it done.

Thanks

[synchronicity]

I've loosened the attachment settings on this forum. If it still doesn't work you could send it to the email address listed if you run 'wimlib-imagex --version'.

[synchronicity]

Hi,

The metadata shows that the WIM image captured with wimlib v1.8.0 contains data streams for the files that you noticed in the diff of the NTFS-3g mounted filesystems, whereas the WIM image captured with wimlib v1.8.2-BETA2 does not.

This is very interesting, but the metadata also shows that this wasn't a fully controlled test because the filesystem was mounted read-write between the two captures (all the last access times are different).

So I'm not sure what to make of it.

These differences also seem to be unrelated to your original problem, because you reported that Windows successfully booted after applying, using wimlib v1.8.0, the WIM image that was captured with wimlib v1.8.2-BETA2, whereas it did not work if you applied the WIM image using wimlib v1.8.2-BETA2.

Unless you are willing to do a full bisection, I suggest that you:

1. Make sure you have wimlib v1.8.0 and v1.8.1 and that they are using the exact same version of libntfs-3g.
2. Capture a WIM image from your source filesystem using wimlib v1.8.0
3. Format the target partition and apply, using wimlib v1.8.0, the WIM image captured in step 2
4. Using wimlib v1.8.0, capture the NTFS volume to which you applied the WIM image, and provide a metadata dump for this new WIM image.
5. Verify that Windows boots from the target partition.
6. Format the target partition and apply, using wimlib v1.8.1, the WIM image captured in step 2
7. Using wimlib v1.8.0, capture the NTFS volume to which you applied the WIM image, and provide a metadata dump for this new WIM image.
8. Verify that Windows does not boot from the target partition.

[techryda]

I'll do some more testing this week (kinda busy with some other things, but anxious for a fix so I'll make time)

I'm not sure what I did when I sent you the files I must've taken the metadata from the wrong partition. I assure you most of my testing was NOT that sloppy. I think I may have rushed the metadata extraction but I'll get you the data from a more controlled test. There is an underlying issue. Software-wise, the only thing that has been changing is the wimlib version. I compiled packages (under Arch) for each version and I switch them out w pacman between tests.

Hopefully I'll find some time today to do some testing. Thanks.

[techryda]

Ok...here it is

Captured the metadata from the same pristine (never been booted) partition.

First capture used wimlib 1.8.0. Second used wimlib 1.8.1

Code: Select all

wiminfo --lookup-table DSS-Win7-32Bit-Base-captured-w-1.8.0.wim | head -n 7
Hash              = 0x02d0421c0121c86b23fd15694b9c3ed0b541af8f
Uncompressed size = 13582456 bytes
Compressed size   = 3075948 bytes
Offset in WIM     = 4049803674 bytes
Part Number       = 1
Reference Count   = 1
Flags             = WIM_RESHDR_FLAG_COMPRESSED  WIM_RESHDR_FLAG_METADATA

dd if=DSS-Win7-32Bit-Base-captured-w-1.8.0.wim ibs=3075948 count=1 iflag=skip_bytes skip=4049803674 of=metadata-from-1.8.0-captured-wim.bin

-----
wiminfo --lookup-table DSS-Win7-32Bit-Base-captured-w-1.8.1.wim | head -n 7
Hash              = 0x8200978ae1ee89cfaabf39c2e0e63be5bd3ed149
Uncompressed size = 13586216 bytes
Compressed size   = 3077030 bytes
Offset in WIM     = 4049803674 bytes
Part Number       = 1
Reference Count   = 1
Flags             = WIM_RESHDR_FLAG_COMPRESSED  WIM_RESHDR_FLAG_METADATA

dd if=DSS-Win7-32Bit-Base-captured-w-1.8.1.wim ibs=3077030 count=1 iflag=skip_bytes skip=4049803674 of=metadata-from-1.8.1-captured-wim.bin

metadata-from-1.8.0-captured-wim.zip

(2.93 MiB) Downloaded 897 times

metadata-from-1.8.1-captured-wim.zip

(2.93 MiB) Downloaded 717 times

Hope this is helpful.