[SOLVED] Issue w wimlib 1.8.1 and later (deploying Win7)

Comments, questions, bug reports, etc.
synchronicity
Site Admin
Posts: 474
Joined: Sun Aug 02, 2015 10:31 pm

Re: Issue w wimlib 1.8.1 and later (deploying Win7)

Post by synchronicity »

Hi,

First, I'm still confused about whether we are dealing with one problem or two
(one with capture, one with apply). The metadata you most recently posted only
shows the difference after a *capture* directly from a sysprepped filesystem, so
an *apply* isn't in the picture at all, correct?

Second, I did manage to identify a case in which a data stream might "go
missing" in a capture with wimlib v1.8.1 compared to a capture with wimlib
v1.8.0. If an NTFS file has more than one unnamed data stream (this is supposed
to be impossible but I won't make any guesses), then v1.8.0 and v1.8.1 may
choose a different one to save in the WIM image. I advise that you re-test
(specifically, the capture scenario) v1.8.1 with and without the following patch
applied:

Code: Select all

diff --git a/src/dentry.c b/src/dentry.c
index 6a75938..17c90aa 100644
--- a/src/dentry.c
+++ b/src/dentry.c
@@ -1842,7 +1842,7 @@ write_dentry(const struct wim_dentry * restrict dentry, u8 * restrict p)
                        if (strm->stream_type == STREAM_TYPE_DATA) {
                                if (stream_is_named(strm))
                                        have_named_data_stream = true;
-                               else
+                               else if (is_zero_hash(unnamed_data_stream_hash))
                                        unnamed_data_stream_hash = stream_hash(strm);
                        } else if (strm->stream_type == STREAM_TYPE_REPARSE_POINT) {
                                have_reparse_point_stream = true;
                                

Alternatively, check out commit a8215b2ff934d1fd0c3f74522deed28c69e36c5d from
the git repository.

For this isn't not necessary to post the full metadata resource --- running 'wimdir
--detailed' on each WIM image and taking the diff of the output should be
sufficient.
techryda
Posts: 20
Joined: Wed Aug 05, 2015 7:07 pm

Re: Issue w wimlib 1.8.1 and later (deploying Win7)

Post by techryda »

The metadata you most recently posted only
shows the difference after a *capture* directly from a sysprepped filesystem, so
an *apply* isn't in the picture at all, correct
Correct, I only captured the image.

Here is the diff you requested:
diff-of-unpatched-and-patched-wimdir.zip
(534 Bytes) Downloaded 591 times
I haven't tried to apply (and boot) the image captured with the patched version yet. I'll give that a try later/tomorrow and let you know.

Thanks
synchronicity
Site Admin
Posts: 474
Joined: Sun Aug 02, 2015 10:31 pm

Re: Issue w wimlib 1.8.1 and later (deploying Win7)

Post by synchronicity »

Thanks. It looks like that confirms what I suspected: the data streams were missing because libntfs-3g reported two unnamed data streams, which is unexpected; and the different versions of wimlib happened to handle this differently. I'm actually planning to work around this in a slightly different way from the one-line patch I posted, but the effect will be the same: if there are multiple unnamed data streams, then wimlib will prefer a nonempty one over an empty one. I'm also going ask the NTFS-3g developers what might be going on.

Let me know if the original problem (possibly with apply, not capture) still exists when applying a WIM image which includes the data streams which had been missing.
synchronicity
Site Admin
Posts: 474
Joined: Sun Aug 02, 2015 10:31 pm

Re: Issue w wimlib 1.8.1 and later (deploying Win7)

Post by synchronicity »

Hi,

One of the NTFS-3g developers confirmed that it's invalid for a file to have multiple unnamed data streams.

They asked about the history of the file (perhaps any named streams being added or deleted), and also whether running 'chkdisk' removes one of the unnamed streams.

Just to confirm, this filesystem was created entirely on Windows 7, and it was a clean shutdown?

If you could somehow provide a byte for byte copy of the filesystem it might be very helpful.

- synchronicity
synchronicity
Site Admin
Posts: 474
Joined: Sun Aug 02, 2015 10:31 pm

Re: Issue w wimlib 1.8.1 and later (deploying Win7)

Post by synchronicity »

wimlib v1.8.2-BETA4, which I just posted, includes a workaround for duplicate NTFS streams.
techryda
Posts: 20
Joined: Wed Aug 05, 2015 7:07 pm

Re: Issue w wimlib 1.8.1 and later (deploying Win7)

Post by techryda »

Sorry for the delay responding...life's been busy.

So, there's still something going on with the capture and/or apply.

From Linux
  • Create 20480MiB Partition (sda1)
  • Make partition active/bootable
From Windows
  • Install Win7 to 20480MiB Partition
  • Just prior to the final step of the install, instead of letting it boot into Windows I boot into Linux
From Linux
  • Capture Image using wimcapture (wimlib 1.8.2)
  • Capture image using ntfsclone (ntfs-3g 2015.3.14)
Restore using wimlib
  • Format sda1 as NTFS
  • Apply image with wimapply
  • Use ms-sys to make partition bootable
  • Reboot
  • Windows doesn't complete boot
Restore using ntfs-3g
  • Format sda1 as NTFS
  • Apply image with ntfsclone
  • Use ms-sys to make partition bootable
  • Reboot
  • Windows completes boot
So, this time there are no applications or strange configurations that I could've done...A pristine Windows 7 install won't boot.
I'm going to try to upload the ntfsclone'd image for you tomorrow.
My thoughts are that you can apply that and capture it yourself with wimlib (any version you want) to see what's happening, otherwise, I'd need to upload a working and non-working image which will take twice the time/storage space.

Let me know if this sounds like the way forward.

Thanks.
synchronicity
Site Admin
Posts: 474
Joined: Sun Aug 02, 2015 10:31 pm

Re: Issue w wimlib 1.8.1 and later (deploying Win7)

Post by synchronicity »

Thanks for testing with wimlib v1.8.2. Yes, uploading the ntfscloned image might be a way forward. An alternative might be to upload a WIM image instead. This is based on your observation that when the same WIM image was applied with wimlib v1.8.0 and wimlib v1.8.2-BETA, the problem only occurred in the wimlib v1.8.2-BETA - applied version. If you could provide a WIM image for which you've confirmed this is true, then I should be able to quickly find out what the problem is. Also, if you want to reduce the size of the WIM file, you have the option of capturing it with the --solid flag.
synchronicity
Site Admin
Posts: 474
Joined: Sun Aug 02, 2015 10:31 pm

Re: Issue w wimlib 1.8.1 and later (deploying Win7)

Post by synchronicity »

Hi,

With the files you provided I was able to identify the problem. wimlib v1.8.1
and later pass a slightly different parameter to a certain libntfs-3g function,
and it violated some assumptions made by libntfs-3g internally. Unfortunately
this usually didn't cause things to completely break, so I never noticed it
earlier; but apparently under some circumstances the resulting filesystem could
not be compatible with the Windows bootloader.

I've fixed this and uploaded a new BETA release (wimlib-v1.8.3-BETA1) which
includes the fix.

Thanks for reporting the problem and helping to track it down!

- synchronicity
techryda
Posts: 20
Joined: Wed Aug 05, 2015 7:07 pm

Re: Issue w wimlib 1.8.1 and later (deploying Win7)

Post by techryda »

Thanks!! I'll run a test sometime this week and report back. Feel free to keep that image for future testing/validation.

-TechRyda
techryda
Posts: 20
Joined: Wed Aug 05, 2015 7:07 pm

Re: Issue w wimlib 1.8.1 and later (deploying Win7)

Post by techryda »

Well, based on one simple restore with wimlib 1.8.3-BETA1, the original issues seems to be fixed. Thanks for your time looking into this!

- TechRyda
Post Reply