+ WIMLIB
- WIMLIB
+This is wimlib version 1.2.7 (March 2013). wimlib can be used to read,
+write, and mount files in the Windows Imaging Format (WIM files). These files
+are normally created by using the `imagex.exe' utility on Windows, but this
+library provides a free implementation of imagex for UNIX-based systems.
-This is wimlib version 0.6.2 (April 2012). wimlib can be used to read, write,
-and mount files in the Windows Imaging Format (WIM files). These files are
-normally created by using the `imagex.exe' utility on Windows, but this library
-provides a free implementetion of imagex for UNIX-based systems.
+ WIM FILES
-The main use of this library is to create customized images of Windows PE, the
-Windows Preinstallation Environment, without having to rely on Windows. Windows
-PE is a lightweight version of Windows that can run entirely from memory and can
-be used to install Windows from local media or a network drive or perform
-maintenance. Windows PE is the operating systems runs when you boot from the
-Windows DVD.
+A Windows Imaging (WIM) file is an archive. Like some other archive formats
+such as ZIP, files in WIM archives may be compressed. WIM archives support two
+Microsoft-specific compression formats: LZX and XPRESS. Both are based on LZ77
+and Huffman encoding, and both are supported by wimlib.
+
+Unlike ZIP files, WIM files can contain multiple independent toplevel directory
+trees known as images. While each image has its own metadata describing a
+directory tree and file access modes, files are not duplicated for each image;
+instead, each file is included only once in the entire WIM. Microsoft did this
+so that in one WIM file, they could do things like have 5 different versions of
+Windows that are almost exactly the same.
+
+Microsoft provides documentation for the WIM file format, XPRESS compression
+format, and LZX compression format. The XPRESS documentation is acceptable, but
+the LZX documentation is not entirely correct, and the WIM documentation itself
+is incomplete.
+
+A WIM file may be either stand-alone or split into multiple parts.
-You can find Windows PE on the ISO filesystem on the installation DVD for both
-Windows 7 and Windows 8. I don't have a DVD for Vista but it should be on there
-too. The Windows PE image a WIM file, `sources/boot.wim', on the ISO
-filesystem. Windows PE can also be found in the Windows Automated Installation
-Kit (WAIK), which is free to download from Microsoft, inside the `WinPE.cab'
-file, which you can extract if you install the `cabextract' program.
+ PROGRAMS
wimlib provides a public API for other programs to use, but also comes with two
-programs: `imagex' and `mkwinpeimg'.
+programs: `imagex' and `mkwinpeimg'.
`imagex' is intended to be like the imagex.exe program from Windows. `imagex'
can be used to create, extract, and mount WIM files. Both read-only and
`mkwinpeimg' is shell script that makes it easy to create a customized bootable
image of Windows PE that can be put on a CD or USB drive, or published on a
-server for PXE booting. See the main page `doc/mkwinpeiso.1' for more details.
+server for PXE booting. See the main page `doc/mkwinpeimg.1' for more details.
-Wimlib can also be used to handle larger WIM files such as the `install.wim'
-file that comes on the Windows DVD; however, this has not been well tested.
+There is an additional program, `wimapply', that is not installed by default.
+It can be used to build a small executable with the ability to apply a WIM image
+from a standalone WIM, without having to build the whole shared library. This
+could be useful on Linux boot clients that only need to be able to apply a WIM,
+not capture/split/join/append/export/mount a WIM. See `programs/wimapply.c'.
-An earlier version of Wimlib is being used to deploy Windows 7 from the Ultimate
-Deployment Appliance. For more information see
-http://www.ultimatedeployment.org/.
+ COMPRESSION RATIO
+
+wimlib can create XPRESS or LZX compressed WIM archives. Currently, the XPRESS
+compression ratio is slightly better than that provided by Microsoft's software,
+while the LZX compression ratio is approaching that of Microsoft's software but
+is not quite there yet. Running time is as good as or better than Microsoft's
+software, especially with multithreaded compression, available in wimlib v1.1.0
+and later.
+
+The following tables compare the compression ratio and performance for creating
+a compressed Windows PE image (disk usage of about 524 MB, uncompressed WIM size
+361 MB):
+
+ Table 1. WIM size
+
+ XPRESS Compression LZX Compression
+ wimlib imagex (v1.2.1): 138,971,353 bytes 131,379,943 bytes
+ Microsoft imagex.exe: 140,406,981 bytes 127,249,176 bytes
+
+ Table 2. Time to create WIM
+
+ XPRESS Compression LZX Compression
+ wimlib imagex (v1.2.1, 2 threads): 11 sec 17 sec
+ Microsoft imagex.exe: 25 sec 89 sec
+
+ NTFS SUPPORT
+
+As of version 1.0.0, wimlib supports capturing and applying images directly to
+NTFS volumes. This was made possible with the help of libntfs-3g from the
+NTFS-3g project. This feature supports capturing and restoring NTFS-specific
+data such as security descriptors, alternate data streams, and reparse point
+data.
+
+The code for NTFS image capture and image application is complete enough that it
+is possible to apply an image from the "install.wim" contained in recent Windows
+installation media (Vista, Windows 7, or Windows 8) directly to a NTFS volume,
+and then boot Windows from it after preparing the Boot Configuration Data. In
+addition, a Windows installation can be captured (or backed up) into a WIM file,
+and then re-applied later.
+
+ WINDOWS PE
+
+A major use for this library is to create customized images of Windows PE, the
+Windows Preinstallation Environment, without having to rely on Windows. Windows
+PE is a lightweight version of Windows that can run entirely from memory and can
+be used to install Windows from local media or a network drive or perform
+maintenance. Windows PE is the operating system that runs when you boot from
+the Windows installation media.
+
+You can find Windows PE on the installation DVD for Windows Vista, Windows 7, or
+Windows 8, in the file `sources/boot.wim'. Windows PE can also be found in the
+Windows Automated Installation Kit (WAIK), which is free to download from
+Microsoft, inside the `WinPE.cab' file, which you can extract if you install
+either the `cabextract' or `p7zip' programs.
--------------------------------------------------------------------------------
+In addition, Windows installations and recovery partitions frequently contain a
+WIM containing an image of the Windows Recovery Environment, which is similar to
+Windows PE.
+
+ DEPENDENCIES
+
+* libxml2 (required)
+ This is a commonly used free library to read and write XML files. You
+ likely already have it installed as a dependency for some other program.
+ For more information see http://xmlsoft.org/.
+
+* libfuse (optional but highly recommended)
+ Unless configured with --without-fuse, wimlib requires a non-ancient
+ version of libfuse to be installed. Most GNU/Linux distributions
+ already include this, but make sure you have the libfuse package
+ installed, and also libfuse-dev if your distribution distributes header
+ files separately. FUSE also requires a kernel module. If the kernel
+ module is available it will automatically be loaded if you try to mount
+ a WIM file. For more information see http://fuse.sourceforge.net/.
+ FUSE is also available for FreeBSD.
+
+* libntfs-3g (optional but highly recommended)
+ Unless configured with --without-ntfs-3g, wimlib requires the library
+ and headers for libntfs-3g version 2011-4-12 or later to be installed.
+ Versions dated 2010-3-6 and earlier do not work because they are missing
+ the header xattrs.h (and the file xattrs.c, which contains functions we
+ need). libntfs-3g version 2013-1-13 is compatible only with wimlib
+ 1.2.4 and later.
+
+* OpenSSL / libcrypto (optional)
+ wimlib can use the SHA1 message digest code from OpenSSL instead of
+ compiling in yet another SHA1 implementation. (See LICENSE section.)
+
+* cdrkit (optional)
+* mtools (optional)
+* syslinux (optional)
+* cabextract (optional)
+ The `mkwinpeimg' shell script will look for several other programs
+ depending on what options are given to it. Depending on your GNU/Linux
+ distribution, you may already have these programs installed, or they may
+ be in the software repository. Making an ISO filesystem requires
+ `mkisofs' from `cdrkit' (http://www.cdrkit.org). Making a disk image
+ requires `mtools' (http://www.gnu.org/software/mtools) and `syslinux'
+ (http://www.syslinux.org). Retrieving files from the Windows Automated
+ Installation Kit requires `cabextract' (http://www.cabextract.org.uk).
CONFIGURATION
Besides the various well-known options, the following options can be passed to
wimlib's `configure' script:
+--without-ntfs-3g
+ If libntfs-3g is not available or is not version 2011-4-12 or later,
+ wimlib can be built without it, in which case it will not be possible to
+ apply or capture images directly to/from NTFS volumes.
+
--without-fuse
If libfuse or the FUSE kernel module is not available, wimlib can be
compiled with --without-fuse. This will remove the ability to mount and
- unmount WIM files. wimlib_mount() and wimlib_unmount() will fail with
- WIMLIB_ERR_UNSUPPORTED.
+ unmount WIM files.
---disable-libcrypto
+--without-libcrypto
Build in functions for SHA1 rather than using external SHA1 functions
from libcrypto (part of OpenSSL). The default is to use libcrypto if it
is found on the system.
+--enable-xattr, --disable-xattr
+ Enable or disable support for the extended-attributes interface to NTFS
+ alternate data streams in mounted WIMs. To support these, wimlib
+ requires that the setxattr() function and the attr/xattr.h header are
+ available. The default is to autodetect whether support is possible.
+
+--disable-multithreaded-compression
+ By default, data will be compressed using multiple threads when writing
+ a WIM, unless only 1 processor is detected. Specify this option to
+ disable support for this.
+
--enable-ssse3-sha1
Use a very fast assembly language implementation of SHA1 from Intel.
Only use this if the build target supports the SSSE3 instructions.
--disable-custom-memory-allocator
- If this option is given, MALLOC(), FREE(), CALLOC(), and STRDUP() will
- directly call the appropriate functions in the C library.
- wimlib_set_memory_allocator() will fail with WIMLIB_ERR_UNSUPPORTED.
+ If this option is given, a very small amount of space will be saved by
+ removing support for the wimlib_set_memory_allocator() function. imagex
+ will be unaffected.
--disable-verify-compression
- Unless this option is given, every time wimlib compresses a data block
- it will decompress it into a temporary buffer and abort() the program
- with an error message if the decompressed data does not exactly match
- the original data. This is to find bugs.
+ Unless this option is given, every time wimlib compresses a data block,
+ it will decompress it into a temporary buffer and abort the program with
+ an error message if the decompressed data does not exactly match the
+ original data. This only makes compression about 10% slower. However,
+ this checking can probably be safely disabled because there are no known
+ bugs in the compression code, and the SHA1 message digest of every
+ extracted file is checked anyway.
--disable-error-messages
- Removes all error messages from the library. If left in, they still
- have to explicitly turned on with wimlib_set_print_errors() in order to
- see them. Also, error codes will still be returned regardless of
- whether error messages are printed or not.
-
- If --disable-error-messages is given, wimlib_set_print_errors() will
- fail with WIMLIB_ERR_UNSUPPORTED if the action is to turn error messages
- on.
+ Save some space by removing all error messages from the library.
--disable-assertions
- Remove all assertions. Without this option, wimlib will abort() the
- program if an assertion fails. An assertion failure should only occur
- if there is a bug in wimlib.
+ Remove all assertions, even the ones that are included by default.
+
+--enable-more-assertions
+ Enable assertions that are not included by default.
--enable-debug
Include debugging messages. Only use this option if you have found a
Include more debugging messages. Only use this option if you have found
a bug in the library.
--------------------------------------------------------------------------------
-
- DEPENDENCIES
-
-Wimlib requires libxml2 to build. This is a commonly used free library to read
-and write XML files. You likely already have it installed as a dependency for
-some other program. For more information see http://xmlsoft.org/.
-
-Wimlib also requires libfuse to build (unless configured with --without-fuse;
-see above). Most GNU/Linux distributions already include this, but make sure
-you have the libfuse package installed (libfuse-dev if your distribution
-distributes header files separately). FUSE also requires a kernel module. If
-the kernel module is available it will automatically be loaded if you try to
-mount a WIM file. Wimlib has only been tested with the Linux version of FUSE.
-For more information see http://fuse.sourceforge.net/.
-
-The `mkwinpeimg' shell script will look for several other programs depending on
-what options are given to it. Depending on your GNU/Linux distribution, you may
-already have these programs installed, or they may be in the software
-repository. Making an ISO filesystem requires `mkisofs' from `cdrkit'
-(http://www.cdrkit.org). Making a disk image requires `mtools'
-(http://www.gnu.org/software/mtools) and `syslinux' (http://www.syslinux.org).
-Retrieving files from the Windows Automated Installation Kit requires
-`cabextract' (http://www.cabextract.org.uk).
-
-------------------------------------------------------------------------------
-
PORTABILITY
-I have been developing and testing wimlib on x86_64 (64-bit) GNU/Linux.
-
-wimlib has been tested on x86 (32-bit) GNU/Linux occasionally.
+wimlib has mostly been developed and tested on x86_64 (64-bit) GNU/Linux.
-I have not tried to compile wimlib with compilers other than gcc.
+It has been tested on x86 (32-bit) GNU/Linux occasionally.
-It should be possible to compile wimlib on other UNIX systems such as FreeBSD.
-If this doesn't work, let me know. If libfuse is not available, try compiling
-with --without-fuse.
+wimlib may work on FreeBSD and Mac OS X. However, this is not well tested. If
+you do not have libntfs-3g 2011-4-12 or later available, you must configure
+wimlib with --without-ntfs-3g. On FreeBSD, before mounting a WIM you need to
+load the POSIX message queue module (run `kldload mqueuefs').
-While the code has been tested on a little endian machine, it has been written
-to work on big endian machines as well. (See endianness.h). However it has
-never actually been tested on a big endian machine. Let me know if it doesn't
-work.
+The code pays attention to endianness, so it should work on big-endian
+architectures, but I've never tested this so do not expect it to work.
-I am not planning to port wimlib to Windows since the programming interface on
+There are no plans to port wimlib to Windows since the programming interface on
Windows is very different and Microsoft's imagex.exe is already available.
-------------------------------------------------------------------------------
-
- REFERENCES
+ REFERENCES
The WIM file format is specified in a document that can be found in the
Microsoft Download Center. There is a similar document that specifies the LZX
compression format, and a document that specifies the XPRESS compression format.
-However, some aspects of these formats are poorly documented. Some particularly
-poorly documented parts of the formats have had comments added in various places
-in the library.
-
-lzx-decomp.c, the code to decompress WIM file resources that are compressed
+However, many parts of these formats are poorly documented, and some parts have
+no documentation whatsoever. Some particularly poorly documented parts of the
+formats have had comments added in various places in the library. Please see
+the code and/or ask me if you have any questions about the WIM file format as it
+exists in reality and not as it exists in Microsoft's poorly written
+documentation.
+
+The code in ntfs-apply.c and ntfs-capture.c uses the NTFS-3g library, which is a
+library for reading and writing to NTFS filesystems (the filesystem used by
+recent versions of Windows). See
+http://www.tuxera.com/community/ntfs-3g-download/ for more information.
+
+lzx-decompress.c, the code to decompress WIM file resources that are compressed
using LZX compression, is originally based on code from the cabextract project
-(http://www.cabextract.org.uk).
-
-lzx-comp.c, the code to compress WIM file resources using LZX compression, is
-originally based on code written by Matthew Russotto (www.russotto.net/chm/).
+(http://www.cabextract.org.uk).
-lz.c, the code to find LZ77 matches, is based on code from zlib.
+lzx-compress.c, the code to compress WIM file resources using LZX compression,
+is originally based on code written by Matthew Russotto (www.russotto.net/chm/).
-sha1.c and sha1.h, the code to compute SHA1 message digests of WIM resources or
-of the WIM file itself in the case of integrity checks, are based on code from
-GNU coreutils.
+lz77.c, the code to find LZ77 matches (used for both XPRESS and LZX compression),
+is based on code from zlib.
A very limited number of other free programs can handle some parts of the WIM
-file format. 7-zip is able to extract and create WIMs and files in many other
-archive formats. However, WIMLIB is designed specifically to handle WIM files
-and provides features previously only available in Microsoft's imagex.exe, such
-as the ability to mount WIMs read-write.
-
-------------------------------------------------------------------------------
+file format. 7-zip is able to extract and create WIMs (as well as files in many
+other archive formats). However, wimlib is designed specifically to handle WIM
+files and provides features previously only available in Microsoft's imagex.exe,
+such as the ability to mount WIMs read-write as well as read-only, the ability
+to create LZX or XPRESS compressed WIMs, and the correct handling of security
+descriptors and hard links.
+
+An earlier version of wimlib is being used to deploy Windows 7 from the Ultimate
+Deployment Appliance. For more information see
+http://www.ultimatedeployment.org/.
+
+You can see the documentation about Microsoft's version of the imagex program at
+http://technet.microsoft.com/en-us/library/cc749447(v=ws.10).aspx, so you can
+see how it compares to the version provided by this library.
+
+ GNU/Linux equivalents of WIM format
+
+What's the equivalent way to capture the filesystem of a GNU/Linux operating
+system into an archive file? You have a few options:
+
+SquashFS:
+ SquashFS (http://squashfs.sourceforge.net/) provides a compressed,
+ read-only filesystem for Linux, and it's probably the closest equivalent
+ of the WIM format and better designed. Although you can't mount
+ SquashFS read-write, when wimlib does this for WIM files it's really an
+ illusion since the WIM isn't actually modified until the image is
+ unmounted. Multiple top-level images in SquashFS files are not
+ supported, although nothing stops you from just putting each image in a
+ separate directory.
+
+FSArchiver:
+ FSArchiver (http://www.fsarchiver.org/Main_Page) is not widely used, but
+ it appears to have some features quite similar to the WIM format.
+
+Tar:
+ The well-known tar format can usually capture a UNIX filesystem just
+ fine, and compressing the tar file produces a good compression ratio
+ (better than WIM, especially if using XZ compression), but there is no
+ support for random access, file deduplication, multiple images per
+ archive, or extended attributes.
+
+Zip:
+ Zip shares some features with WIM but is not designed to store entire
+ filesystems.
+
+7z:
+ The 7z format has some nice features but is unfortunately not designed
+ with UNIX in mind.
MORE INFORMATION
See the manual pages for `imagex', the manual pages for the subcommands of
`imagex', and the manual page for `mkwinpeimg'.
-As of version 0.5.0, Wimlib's public API is documented. Doxygen is required to
+As of version 0.5.0, wimlib's public API is documented. Doxygen is required to
build the documentation. To build the documentation, run `configure', then
enter the directory `doc' and run `doxygen'. The HTML documentation will be
created in a directory named `html'.
-------------------------------------------------------------------------------
-
LICENSE
-Wimlib is released under the GNU LGPL version 2.1 or later. The files in the
-`programs' directory are released under the GPL version 3.
+As of version 1.0.0, wimlib is released under the GNU GPL version 3.0 or later.
+This includes the files in the `programs' directory as well as the files in the
+`src' directory.
+
+wimlib is independently developed and does not contain any code, data, or files
+copyrighted by Microsoft. It is not known to be affected by any patents.
+
+By default, wimlib will be linked to the system library "libcrypto", which
+probably will be OpenSSL. Some people believe that GPL code cannot be linked to
+OpenSSL without a linking exception. As far as I know, I cannot officially
+include a linking exception with the license of this library because several
+files could be considered derived works of code copyrighted by others. If you
+believe this to be a problem, configure with --without-libcrypto to avoid
+linking with OpenSSL. There is no difference in functionality--- there will
+just be stand-alone SHA1 message digest code built into the library.
-------------------------------------------------------------------------------
+ DISCLAIMER
- DISCLAIMER
+wimlib is experimental. Use Microsoft's `imagex.exe' if you want to make sure
+your WIM files are made correctly (but beware: Microsoft's version contains some
+bugs).
-Wimlib is experimental. Use Microsoft's `imagex.exe' if you want to make sure
-your WIM files are made correctly. Please submit a bug report (to
-ebiggers3@gmail.com) if you find a bug.
+Please submit a bug report (to ebiggers3@gmail.com) if you find a bug in wimlib.
Some parts of the WIM file format are poorly documented or even completely
-undocumented, so these parts had to be reverse engineered.
+undocumented, so I've just had to do the best I can to read and write WIMs that
+appear to be compatible with Microsoft's software.
git://wimlib.net / wimlib / blobdiff