From: Eric Biggers Date: Thu, 24 Apr 2014 19:49:12 +0000 (-0500) Subject: Update docs for chunk size and compression formats X-Git-Tag: v1.7.0~270 X-Git-Url: https://wimlib.net/git/?p=wimlib;a=commitdiff_plain;h=e5df7af7c9a9239bcc63d341829485467fa8d09e Update docs for chunk size and compression formats --- diff --git a/doc/man1/imagex-capture.1.in b/doc/man1/imagex-capture.1.in index 5b302e62..060430f5 100644 --- a/doc/man1/imagex-capture.1.in +++ b/doc/man1/imagex-capture.1.in @@ -217,23 +217,22 @@ compression ratio. This currently only has an effect for LZX ("maximum", the default) and LZMS ("recovery") compression. .TP \fB--chunk-size\fR=\fISIZE\fR -Set the WIM compression chunk size to \fISIZE\fR. Larger chunks mean larger +Set the WIM compression chunk size to \fISIZE\fR bytes. Larger chunks mean larger LZ77 dictionaries and better compression ratios on sufficiently large files, but slower random access. \fBUsing this option is generally not recommended because of the compatibility limitations detailed in the next paragraph.\fR But if you -decide to use this option regardless, you may choose a chunk size that is a -power of 2 greater than or equal to 2^15 (32768) up to a maximum determined by -the compression format. For LZX ("maximum") compression, the maximum allowed -chunk size is 2^21 (2097152), and for XPRESS ("fast") and LZMS ("recovery") -compression, the maximum allowed chunk size is 2^30 (1073741824). -.IP "" -For XPRESS and LZX compression, Microsoft's implementation (as of Windows 8) -does not appear to support alternate chunk sizes; although it will still open -such files, it will crash, extract the data incorrectly, or report that the data -is invalid. For LZMS compression, Microsoft's implementation (as of Windows 8) -appears to only support chunk sizes that are powers of 2 between 2^15 (32768) -and 2^20 (1048576) inclusively. In addition, wimlib versions before 1.6.0 do -not support alternate chunk sizes. +decide to use this option regardless, you may choose a chunk size that is +allowed by the compression format. All formats only allow power-of-2 chunk +sizes. For LZX ("maximum") compression the maximum allowed chunk size is 2^21 +(2097152), for XPRESS ("fast") compression the maximum allowed chunk size is +2^26 (67108864), and for LZMS ("recovery") compression the maximum allowed chunk +size is 2^30 (1073741824). +.IP "" +Beware that Microsoft's implementation has limited support for non-default chunk +sizes. Depending on the version, their software may refuse to open the WIM, or +open it and crash, or open it and report the data is invalid, or even extract +the data incorrectly. In addition, wimlib versions before 1.6.0 do not support +alternate chunk sizes. .TP \fB--pack-streams\fR, \fB--solid\fR Create a "solid" archive that compresses multiple unique streams ("files") @@ -244,22 +243,23 @@ mounted with \fB@IMAGEX_PROGNAME@ mount\fR. Also, WIMs created using this option use a different version number in their header and as of Windows 8 are only compatible with Windows Setup and WIMGAPI, not even ImageX and Dism. .IP "" -Packed resources use a compression type and chunk size that is independent of -the WIM's "default compression type" and "default chunk size" (which may be -adjusted by the \fB--compress\fR and \fB--chunk-size\fR options, respectively). -For compatibility reasons, \fB@IMAGEX_PROGNAME@ capture\fR currently has no -option to change the compression type used in packed resources; however, the -\fB--pack-chunk-size\fR option may be used to set the chunk size. +The default compression type and chunk size in packed resources is LZMS with +2^25 (33554432) byte chunks. This is independent of the WIM's main compression +type and chunk size. .TP \fB--pack-chunk-size\fR=\fISIZE\fR, \fB--solid-chunk-size\fR=\fISIZE\fR Like \fB--chunk-size\fR, but set the chunk size used in packed resources. The -compression format is LZMS, so the chunk size can be any power of 2 between 2^15 -and 2^30, inclusively. WIMGAPI (Windows 8) appears to be compatible with these -sizes up to 2^26 inclusively, despite not being compatible with sizes greater -than 2^20 in non-packed resources. The default is currently 2^25 (33554432). -Note: currently, the LZMS compression algorithm uses about 15 times the chunk -size in memory per thread, which is about 500 MB per thread for the default pack -chunk size of 2^25 or 1 GB per thread if you change it to 2^26 (67108864). +default is LZMS compression with 2^25 (33554432) byte chunks. This option only +has an effect when \fB--pack-streams\fR is also specified. For maximum +compatibility with the Microsoft implementation, do not use either of these +options. +.TP +\fB--pack-compress\fR=\fITYPE\fR, \fB--solid-compress\fR=\fITYPE\fR +Like \fB--compress\fR, but set the compression format used in packed resources. +The default is LZMS compression with 2^25 (33554432) byte chunks. This option +only has an effect when \fB--pack-streams\fR is also specified. For maximum +compatibility with the Microsoft implementation, do not use either of these +options. .TP \fB--threads\fR=\fINUM_THREADS\fR Number of threads to use for compressing data. Default: autodetect (number of diff --git a/doc/man1/imagex-export.1.in b/doc/man1/imagex-export.1.in index a024e383..a60e9783 100644 --- a/doc/man1/imagex-export.1.in +++ b/doc/man1/imagex-export.1.in @@ -85,14 +85,25 @@ compression ratio. This currently only has an effect for LZX ("maximum") and LZMS ("recovery") compression. This option does not itself set the compression format. .TP +\fB--chunk-size\fR=\fISIZE\fR +Set the WIM compression chunk size to \fISIZE\fR. See the documentation for +this option to \fB@IMAGEX_PROGNAME@ capture\fR (1) for more details. +.TP \fB--pack-streams\fR, \fB--solid\fR Create a "solid" archive that compresses multiple files together. This can result in a higher compression ratio, but has disadvantages such as reduced -compatibility; see the documentation for this option to -\fB@IMAGEX_PROGNAME@ capture\fR (1) for more details. +compatibility. See the documentation for this option to \fB@IMAGEX_PROGNAME@ +capture\fR (1) for more details. .TP \fB--pack-chunk-size\fR=\fISIZE\fR, \fB--solid-chunk-size\fR=\fISIZE\fR -Like \fB--chunk-size\fR, but set the chunk size used in packed resources. +Like \fB--chunk-size\fR, but set the chunk size used in packed resources. See +the documentation for this option to \fB@IMAGEX_PROGNAME@ capture\fR (1) for +more details. +.TP +\fB--pack-compress\fR=\fITYPE\fR, \fB--solid-compress\fR=\fITYPE\fR +Like \fB--compress\fR, but set the compression format used in packed resources. +See the documentation for this option to \fB@IMAGEX_PROGNAME@ capture\fR (1) for +more details. .TP \fB--threads\fR=\fINUM_THREADS\fR Number of threads to use for compressing data. Default: autodetect (number of diff --git a/doc/man1/imagex-optimize.1.in b/doc/man1/imagex-optimize.1.in index f284612d..a5d7458f 100644 --- a/doc/man1/imagex-optimize.1.in +++ b/doc/man1/imagex-optimize.1.in @@ -63,11 +63,18 @@ this option to \fB@IMAGEX_PROGNAME@ capture\fR (1) for more details. \fB--pack-streams\fR, \fB--solid\fR Create a "solid" archive that compresses multiple files together. This can result in a higher compression ratio, but has disadvantages such as reduced -compatibility; see the documentation for this option to -\fB@IMAGEX_PROGNAME@-capture\fR (1) for more details. +compatibility. See the documentation for this option to \fB@IMAGEX_PROGNAME@ +capture\fR (1) for more details. .TP \fB--pack-chunk-size\fR=\fISIZE\fR, \fB--solid-chunk-size\fR=\fISIZE\fR -Like \fB--chunk-size\fR, but set the chunk size used in packed resources. +Like \fB--chunk-size\fR, but set the chunk size used in packed resources. See +the documentation for this option to \fB@IMAGEX_PROGNAME@ capture\fR (1) for +more details. +.TP +\fB--pack-compress\fR=\fITYPE\fR, \fB--solid-compress\fR=\fITYPE\fR +Like \fB--compress\fR, but set the compression format used in packed resources. +See the documentation for this option to \fB@IMAGEX_PROGNAME@ capture\fR (1) for +more details. .TP \fB--threads\fR=\fINUM_THREADS\fR Number of threads to use for compressing data. Default: autodetect (number of diff --git a/include/wimlib.h b/include/wimlib.h index 77fe0ff9..7a453373 100644 --- a/include/wimlib.h +++ b/include/wimlib.h @@ -3431,11 +3431,6 @@ wimlib_set_output_compression_type(WIMStruct *wim, int ctype); * * Similar to wimlib_set_output_compression_type(), but set the compression type * for writing packed streams (solid blocks). - * - * Note: based on testing, WIMGAPI is seemingly only compatible with LZMS - * compression in packed streams. Therefore the use of this function is not - * recommended. Also, with large chunk sizes, LZMS gives the best compression - * ratio among the alternatives anyway. */ extern int wimlib_set_output_pack_compression_type(WIMStruct *wim, int ctype);