Delta WIM

[abbodi86]

Hi
thanks for the magnificent tool

couple of questions about the delta WIM files

- apart from the noticeably smaller size, is there a way or a flag to identify a delta WIM?

- is there a way to ignore some of missing base WIMs and continue applying or exporting delta WIM?

- delta WIM can be only created with capture operation, but if the WIMFILE already exist, delta WIM is appended to new index
whether if it's a normal or accidental behavior, i hope you do not remove it

i ask because of the new Microsoft's upgrade mechanism that's called Unified Update Platform
https://blogs.windows.com/windowsexperi ... tform-uup/

the static complete ESD file is now delivered as multiple smaller ESD and CAB files, based on Edition/Language/Features.. etc

it's literally use the delta WIM concept

one Metadata ESD (i.e. Professional_en-us.esd), contain 3 indexes:
1- setup media layout (ISO files minus boot.wim/install.wim)
2- winre.wim
3- install.wim metadata (the actual delta WIM)

all other files are the base WIMs
however, some of them are delivered as regular CAB files, so it's needed first to expand those CABs and capture them as WIMs in order to use them in reference GLOB
i don't know how the upgrade setup handle those files to build and deploy OS image

i can upload you the files if you are interested

Regards.

[synchronicity]

- apart from the noticeably smaller size, is there a way or a flag to identify a delta WIM?

Not that I am aware of. I could however look at some of the Microsoft-created files and see if there is a flag or anything that I've missed.

- is there a way to ignore some of missing base WIMs and continue applying or exporting delta WIM?

Not if the extraction or export operation requires access to any file data that is missing. It is possible to use wimextract to extract individual files that are not missing, but generally a whole image cannot be extracted. Also note that wimexport does not currently support exporting an image while maintaining "delta" semantics. It will only export the full image, just as if the image were being exported from a regular standalone WIM. It could be possible to add a --delta-from flag to wimexport to allow base WIMs to be specified.

- delta WIM can be only created with capture operation, but if the WIMFILE already exist, delta WIM is appended to new index whether if it's a normal or accidental behavior, i hope you do not remove it

Are you sure about that? The current behavior appears to be that 'wimappend' errors out if --delta-from is passed to it, while 'wimcapture' with --delta-from always creates a new delta WIM file, overwriting any that may already exist. It *might* make sense to change 'wimappend' to allow --delta-from, if people are interested. Right now I can't think of any technical reason why it wouldn't be possible.

[abbodi86]

Not that I am aware of. I could however look at some of the Microsoft-created files and see if there is a flag or anything that I've missed.

I checked it with wimlib-imagex, dism, imagex, nothing special showed up in the reported info.
here is the Metadata ESD if you like to check:
https://mega.nz/#F!vkUXWSKS!_c5jN905IsN9lizVu4BxtQ

here are the whole UUP files:
https://mega.nz/#!up8XzSpb!te5UZjWd1uqT ... OMDhgBpYlg

all are original, no encryption was added like it was in the complete ESD file

Not if the extraction or export operation requires access to any file data that is missing. It is possible to use wimextract to extract individual files that are not missing, but generally a whole image cannot be extracted. Also note that wimexport does not currently support exporting an image while maintaining "delta" semantics. It will only export the full image, just as if the image were being exported from a regular standalone WIM. It could be possible to add a --delta-from flag to wimexport to allow base WIMs to be specified.

Understood.
yes, i ment export (rebuild) the full image, not creating new delta WIM

Are you sure about that? The current behavior appears to be that 'wimappend' errors out if --delta-from is passed to it, while 'wimcapture' with --delta-from always creates a new delta WIM file, overwriting any that may already exist. It *might* make sense to change 'wimappend' to allow --delta-from, if people are interested. Right now I can't think of any technical reason why it wouldn't be possible.

You are absolutely right, wimcapture with --delta-from does overwrites the already existed image in the wim
i just mixed things, sorry about that.

but i was wondering how Microsoft created the delta WIM to be index 3 in Metadata ESD, where index 1 & 2 are full images.

Thanks

[synchronicity]

I took a look at one of the MS-created delta WIMs you referenced, but it didn't have any special flags or XML elements to indicate that it is a delta WIM.

I suppose that a delta WIM could be identified by checking whether there are any files in its images that reference missing data. There are a couple very hack-ish ways you could do this right now with wimlib-imagex, e.g. running 'wimverify' or 'wimapply' and checking for exit status 55 ("A file resource needed to complete the operation was missing from the WIM."), or running 'wimdir --detailed' and checking whether any files that have a hash listed but no location (e.g. size and offset).

I tested allowing the --delta-from option to 'wimappend', and it seems to work as expected as the API already handled this case. So I will plan to allow it in the next beta version.

[abbodi86]

I appreciate that, thanks.

[synchronicity]

I've posted wimlib v1.11.0-BETA7 which includes --delta-from support in wimappend.

[abbodi86]

Works great, thanks again.

[abbodi86]

Sorry to bump this
just wanted to inform you about a recent change related to this matter in Windows 10 WIMGAPI/ImageX starting build 15063

Microsoft implemented support (or recognition) for using cab files as reference globs in applying delta wim/esd

this actually require and make use of a text file inside the cab to work, named $filehashes$.dat which contain the sha1/sha256 hashes for all files within, plus size in hex format, e.g.

Code: Select all

update.mum=D6E22603373E4ABB494BF4E22F46AD2B58D42B6C,AEA

and apparently wimgapi parse it to identify the resources needed for delta wim

all UUP esd files, FOD/LP cab files have it since 15063
here's one to check:
http://download.windowsupdate.com/c/msd ... 8c0b49.cab

this new support also includes specifiying a reference directory that have the expanded UUP packages (known as Express UUP)
in this case $filehashes$.dat is parsed from each subdirectory

both of the above only works for applying delta wim, unfortunately not for exporting to new complete wim
while imagex also recognize cab files, but for some reason (seems intentional), it does not recognize solid ESD files as reference

Code: Select all

[3276] [ReadWimHeader:(1856) -> version/header mismatch] W:\ESD\UUPs\Education_en-us.esd (Error = 11) 
[3276] [WIMCreateFile:(426) -> Fail to read WIM header] W:\ESD\UUPs\Education_en-us.esd (Error = 11) 
[3276] ExportInResourceOrder:(717)
[3276] ExportDirTree:(401)

p.s. DISM still do not have this new ability

------
i know this new change might not be directly related to WIM format, but it would be really great if wimlib have it

BTW, this project allow to download UUP files for various Windows 10 builds, if you are interesed
http://mdluup.ct8.pl/
source: https://gitlab.com/uup-dump

Regards.

--

Edit:
i didn't saw @JFX's thread first
viewtopic.php?f=1&t=333

[synchronicity]

I'm going to look into readonly .cab support, but it may be too much work to add.

Re: ImageX and DISM not supporting solid WIMs (ESD files) as reference: I don't know the exact reason for that. It might be somewhat of an artificial limitation, given that WIMGAPI will only open a solid WIM if a special flag is passed to WIMCreateFile(), and probably not all users have been updated to pass that flag. (wimlib doesn't have such a flag.)

[abbodi86]

No need to take it hard, expand cab/capture wim still very feasible workaround for that matter
thank.

on apply, both DISM & ImageX recognize solid ESD as reference
but on export, DISM recognize ESD, but not CAB, and vice versa for ImageX

thanks again.