wimlib 1.13.0 released

Comments, questions, bug reports, etc.
synchronicity
Site Admin
Posts: 292
Joined: Sun Aug 02, 2015 10:31 pm

wimlib 1.13.0 released

Post by synchronicity » Sat Nov 24, 2018 5:37 am

wimlib 1.13.0 has been released. List of changes:
  • On Windows, wimlib now supports capturing and applying extended attributes (EAs). It is compatible with DISM with the /EA option, available since Windows 10 version 1607. wimlib's EA support is on by default and works on older versions of Windows too.
  • Partially fixed a bug where [ExclusionException] entries didn't take effect when the containing directory is matched by [ExclusionList]. It now works when the [ExclusionException] patterns are absolute. For example, listing "/dir/file" in [ExclusionException] now works even if "/dir" is matched by [ExclusionList].
  • Added a '--create' option to 'wimappend' which makes it create the WIM file (like 'wimcapture') if it doesn't exist yet.
  • Added an '--include-integrity' option to various wimlib-imagex commands. '--include-integrity' is like '--check', but it will just include an integrity table in the output WIM(s), while skipping verification of any existing integrity tables. This can be useful to avoid unwanted verification of large WIM files, e.g. WIMs given by '--delta-from'.
  • 'wimextract' now reads a pathlist file from standard input when "@-" is given as an argument.
  • wimsplit (API: wimlib_split()) now correctly handles a dot in the path to the first split WIM part, prior to the filename extension.
  • 'wimlib-imagex --version' now shows the version of the library it is actually using (in case it is different from wimlib-imagex's version).
The files and their SHA-256 checksums are:

Code: Select all

51d92cafaae0180c7a91b50620f86ffbcec6d6aef360d38dd47e274e65d7970b  wimlib-1.13.0.tar.gz
2dd5e110342ed34c0280a32ee679e7b32e6ca3f08f61ec96284d8907b36ff5ff  wimlib-1.13.0-windows-i686-bin.zip
d40d3d16577346dfdd12e73b0f21634549d6a376585fee03278ce32d4f018007  wimlib-1.13.0-windows-x86_64-bin.zip

zipmagic
Posts: 31
Joined: Thu Aug 06, 2015 7:09 am

Re: wimlib 1.13.0 released

Post by zipmagic » Fri Nov 30, 2018 11:03 am

Congratulations on this new release!

Do the EA's include the NTFS compression attribute?

Is there a way to enforce NTFS compression attributes to be excluded from WIM archives that are being created?

Does the include integrity option act as sort of a safeguard against unsafe compaction and/or WIM corruption, or is it unrelated?

Once again, congratulations on this exciting new release, and thank you for all the years of hard work that have gone into it!

synchronicity
Site Admin
Posts: 292
Joined: Sun Aug 02, 2015 10:31 pm

Re: wimlib 1.13.0 released

Post by synchronicity » Sat Dec 01, 2018 3:07 am

Do the EA's include the NTFS compression attribute?
No, the NTFS compression attribute (FILE_ATTRIBUTE_COMPRESSION) is not an "extended attribute". The NTFS compression attribute was already saved and restored by wimlib.
Is there a way to enforce NTFS compression attributes to be excluded from WIM archives that are being created?
Not currently.
Does the include integrity option act as sort of a safeguard against unsafe compaction and/or WIM corruption, or is it unrelated?
It's a subset of the --check option which wimlib-imagex has already had for years. --include-integrity just enables writing the optional per-WIM integrity table without verifying any existing one, whereas --check enables both actions.

zipmagic
Posts: 31
Joined: Thu Aug 06, 2015 7:09 am

Re: wimlib 1.13.0 released

Post by zipmagic » Sun Dec 02, 2018 12:21 pm

It would be great if there was a way to exclude NTFS compression attributes from a wimlib compression and/or extraction pass.

Consider what happens when applying a WIM archive in WIMBOOT mode with NTFS compression enabled:

1) The file takes up space inside the WIM archive in tightly compressed form.

2) Applying this archive in WIMBOOT form creates pointer data to the tightly compressed file without wasting any space (so far so good).

3) Setting an NTFS compression on this file:

3A) Fully extracts the file from the WIM archive. All space on disk used to store the file in tightly compressed form is wasted inside the WIM archive!
3B) Re-compresses it using NTFS compression, which is far inferior to WIM compression...
3C) Net Result: Two copies of the file on disk, one almost uncompressed (NTFS compression), the other inside the WIM archive (tightly compressed).

Is it possible to work around this issue somehow with callbacks or other tricks, if not directly within wimlib?

Or perhaps just an option in wimboot mode during wimlib extraction? Thank you!

synchronicity
Site Admin
Posts: 292
Joined: Sun Aug 02, 2015 10:31 pm

Re: wimlib 1.13.0 released

Post by synchronicity » Sun Dec 02, 2018 7:36 pm

It would be great if there was a way to exclude NTFS compression attributes from a wimlib compression and/or extraction pass.

Consider what happens when applying a WIM archive in WIMBOOT mode with NTFS compression enabled:

1) The file takes up space inside the WIM archive in tightly compressed form.

2) Applying this archive in WIMBOOT form creates pointer data to the tightly compressed file without wasting any space (so far so good).

3) Setting an NTFS compression on this file:

3A) Fully extracts the file from the WIM archive. All space on disk used to store the file in tightly compressed form is wasted inside the WIM archive!
3B) Re-compresses it using NTFS compression, which is far inferior to WIM compression...
3C) Net Result: Two copies of the file on disk, one almost uncompressed (NTFS compression), the other inside the WIM archive (tightly compressed).
That's not actually what happens. wimlib's order during extraction in --wimboot mode is:

1.) File is created, but no data is written yet.
2.) FILE_ATTRIBUTE_COMPRESSED is set if needed to match the state in the WIM file. Again, no data is written yet.
3.) FSCTL_SET_EXTERNAL_BACKING is used to mark the file's contents as externally backed by the WIM file.

So it already does what you want. You do *not* end up with two copies of the file contents, unless someone later writes to the file in which case you actually do want FILE_ATTRIBUTE_COMPRESSED set so that the changed copy is NTFS-compressed rather than uncompressed.

FYI, this is easily verifiable by a few commands with 'compact' and 'wimlib-imagex'. It would be helpful if you checked whether wimlib really does the wrong thing or not, before making new feature requests.

zipmagic
Posts: 31
Joined: Thu Aug 06, 2015 7:09 am

Re: wimlib 1.13.0 released

Post by zipmagic » Mon Dec 03, 2018 12:12 am

That's a remarkable workflow, I am happy to admit it never occurred to me that such a flow could even be theoretically possible.

So what happens when the NTFS compression attribute is removed, say through File Explorer, before the file is modified?

Then, is the file extracted by the operating system, or is the operating system smart enough to recognize that this is unnecessary?

synchronicity
Site Admin
Posts: 292
Joined: Sun Aug 02, 2015 10:31 pm

Re: wimlib 1.13.0 released

Post by synchronicity » Mon Dec 03, 2018 1:08 am

So what happens when the NTFS compression attribute is removed, say through File Explorer, before the file is modified?

Then, is the file extracted by the operating system, or is the operating system smart enough to recognize that this is unnecessary?
The file will be extracted (converted from an "externally backed" file to a regular one) if it's opened to add or remove the NTFS compression attribute, since changing the NTFS compression attribute requires opening the file with FILE_WRITE_DATA access, which triggers the extraction.

zipmagic
Posts: 31
Joined: Thu Aug 06, 2015 7:09 am

Re: wimlib 1.13.0 released

Post by zipmagic » Mon Dec 03, 2018 8:02 pm

That seems like undesirable operation to me.

As far as any user can tell, the files are reported to be NTFS compressed, when they are not. This certainly got me, and will surely get others too.

Then requesting NTFS decompression leads to the same space waste problem I've described above.

So with things as they are, we've got a "GUI bug" which leads to a "space bleed" bug when a user gets irritated by the "GUI bug".

Is there anything you can suggest for me to work around this misleading situation, and/or to update in the wimlib code directly?

synchronicity
Site Admin
Posts: 292
Joined: Sun Aug 02, 2015 10:31 pm

Re: wimlib 1.13.0 released

Post by synchronicity » Tue Dec 04, 2018 4:17 am

If the user sees that a file is "compressed", then they'd expect decompressing it to increase the space used. Really the other case is the unexpected one: in Explorer, one can select "Compress contents to save disk space" on a WIM-backed file, but this actually causes the space used to increase.

But of course, I can't do anything about Explorer not showing whether files are WIM-backed or not.

zipmagic
Posts: 31
Joined: Thu Aug 06, 2015 7:09 am

Re: wimlib 1.13.0 released

Post by zipmagic » Tue Dec 04, 2018 12:47 pm

Looking at this from more of a performance perspective, a user may see that the file is NTFS compressed (when it is not) and desire to decompress it to increase performance (which actually results in decreased performance as well as wasted disk space - WIMBOOT compression does increase performance, whereas NTFS compression decreases it).

Ultimately, the issue at the root of this all is that WIMLIB is marking files as NTFS compressed, when they are not. NTFS compression is applicable only to non-WIM-backed files. By setting an NTFS compression flag on a file that is actually WIM-backed, we are running into a lot of false-positive, avoidable issues.

Is there a way to work around this issue? The most proper fix to me seems to not mark files as NTFS compressed when they are WIM-backed. This is plain false at best and results into the problems I've been highlighting at worst. Only when applying in WIMBOOT mode, the take-home of this discussion is that files should not be marked as NTFS compressed when they are clearly not NTFS compressed in any shape or form.

Post Reply