wimlib

wimlib version 1.10.0 has been released. Notable changes:

The LZX compression ratio has been slightly improved. The default mode,
LZX level 50, is now almost as good as the old LZX level 100, while
being nearly the same speed as before.

Decompression performance has been slightly improved.

Filenames are now always listed in NTFS collation order.

On UNIX-like systems, wimlib can now process Windows filenames that are
not valid Unicode due to the presence of unpaired surrogates.

On UNIX-like systems, wimlib now always assumes UTF-8 encoding with the
addition of surrogate codepoints. Consequently, the environmental
variable WIMLIB_IMAGEX_USE_UTF8 and the flag
WIMLIB_INIT_FLAG_ASSUME_UTF8 no longer have any effect.

wimlib no longer depends on iconv.

Reduced memory usage slightly.

When a WIM image is applied in NTFS-3G mode, security descriptors are
now created in NTFS v3.0 format when supported by the volume.

Workarounds for bugs in libntfs-3g version 2013.1.13 and earlier have
been removed. Users are advised to upgrade to a later version of
libntfs-3g.

On Windows, wimlib now supports case-sensitive filename extraction when
supported by the underlying operating system and filesystem (operating
system support requires a registry setting).

The files and their SHA-256 checksums are:

989b1b02f246c480dec10469374f4235d15a3d5e5ae054452405305af5007f55 wimlib-1.10.0.tar.gz
b5b0906da0e6c4a3bfa933ba905bac28f566d70976cb434a20c7f1b2d4c17a48 wimlib-1.10.0-windows-i686-bin.zip
a1208f2419e48b7e23b814b4495fa6a27ff7df48c654379d1162db8ca94c33bb wimlib-1.10.0-windows-x86_64-bin.zip

Hi,
Wimlib is already the fastest and most storage efficient archiver we got in our real use case. And you manage to still improve performance

Oh and why not create a fork of ntfs-3G or submit patches? Wimlib is certainly the application which has the most intensive usage of ntfs-3G when applying huge partition images. What you would do with it could only benefit the other applications using it.

Wimlib is already the fastest and most storage efficient archiver we got in our real use case. And you manage to still improve performance

Yes, I keep the compressors and decompressors very well optimized. This is useful not just for wimlib specifically but also for learning about how to optimize LZ77-based algorithms in general and for reusing the code in other projects that need it, such as my NTFS-3G plugin for system compression (https://github.com/ebiggers/ntfs-3g-system-compression).

wimlib-imagex uses LZX by default, and it is a compression format which has aged pretty well. Some newer formats are better, though not by as much as one may expect given that LZX is ~20 years old. The main problem with its use in WIM is actually just the fact that the default "chunk size" is limited to 32 KiB. In other words, every 32 KiB of data is compressed independently. This value is good for somewhat fine-grained random access, but I don't think that's an important use case given that even if you are doing "random" access, you likely should be doing more than 32 KiB of readahead per access anyway. A chunk size like 128 KiB would be more reasonable. For anyone who'd like to try it, wimlib-imagex lets you change the LZX chunk size to a larger value using the --chunk-size option, though the resulting archive will not be compatible with Microsoft's WIM software.

Oh and why not create a fork of ntfs-3G or submit patches? Wimlib is certainly the application which has the most intensive usage of ntfs-3G when applying huge partition images. What you would do with it could only benefit the other applications using it.

I actually have already been contributing bug reports, patches, and other improvements to NTFS-3G! As of the latest stable NTFS-3G release (2016.2.22), at least 11 unique bugs have been fixed as a direct result of testing via the wimlib project. I have also improved NTFS-3G's LZNT1 compressor, and the next NTFS-3G release is planned to support for transparently reading Windows 10 "system compressed" files via an external plugin (see: https://github.com/ebiggers/ntfs-3g-system-compression). I am planning to keep contributing to NTFS-3G, as it very much needs more contributors (unfortunately, several individuals who would be very qualified to contribute to NTFS-3G have been lost to a commercial company which develops a proprietary fork of the software...).

synchronicity wrote:such as my NTFS-3G plugin for system compression (https://github.com/ebiggers/ntfs-3g-system-compression).

Nice If it's now supported in Linux, I'll try to get it working on Windows 7 (I read it is possible to "backport").

synchronicity wrote:A chunk size like 128 KiB would be more reasonable. For anyone who'd like to try it, wimlib-imagex lets you change the LZX chunk size to a larger value using the --chunk-size option, though the resulting archive will not be compatible with Microsoft's WIM software.

Also good to know Since I discovered your implementation, I don't use the original one anymore anyway

synchronicity wrote:I actually have already been contributing bug reports, patches, and other improvements to NTFS-3G! As of the latest stable NTFS-3G release (2016.2.22), at least 11 unique bugs have been fixed as a direct result of testing via the wimlib project. I have also improved NTFS-3G's LZNT1 compressor, and the next NTFS-3G release is planned to support for transparently reading Windows 10 "system compressed" files via an external plugin (see: https://github.com/ebiggers/ntfs-3g-system-compression). I am planning to keep contributing to NTFS-3G, as it very much needs more contributors (unfortunately, several individuals who would be very qualified to contribute to NTFS-3G have been lost to a commercial company which develops a proprietary fork of the software...).

I wish other open source projects would put as much time in polishing as you do. I hope you have a good job allowing you to earn enough and continue to work on your personal projects as they are very useful

By the way, any chance to changed the hard coded 16 (Kbyte or Mbyte don't remember) hole after each file anytime soon in NTFS-3G?

By the way, any chance to changed the hard coded 16 (Kbyte or Mbyte don't remember) hole after each file anytime soon in NTFS-3G?

I am planning to work on this, but I cannot guarantee when it will happen.