X-Git-Url: https://wimlib.net/git/?p=wimlib;a=blobdiff_plain;f=README;h=e3c7632d7f478d37102117886edc43d681da5def;hp=c210fce9c9fce6948acd09084323e5936bd9cf4a;hb=7d01645312c14c8d79a7e7b059838c3211dfd193;hpb=ad70b091b63707f568cf5066a50aa05436baf253 diff --git a/README b/README index c210fce9..e3c7632d 100644 --- a/README +++ b/README @@ -1,26 +1,38 @@ - WIMLIB + WIMLIB -This is wimlib version 0.6.3 (May 2012). wimlib can be used to read, write, -and mount files in the Windows Imaging Format (WIM files). These files are -normally created by using the `imagex.exe' utility on Windows, but this library -provides a free implementetion of imagex for UNIX-based systems. +This is wimlib version 1.3.0 (March 2013). wimlib can be used to read, +write, and mount files in the Windows Imaging Format (WIM files). These files +are normally created by using the `imagex.exe' utility on Windows, but this +library provides a free implementation of imagex for UNIX-based systems. -The main use of this library is to create customized images of Windows PE, the -Windows Preinstallation Environment, without having to rely on Windows. Windows -PE is a lightweight version of Windows that can run entirely from memory and can -be used to install Windows from local media or a network drive or perform -maintenance. Windows PE is the operating systems runs when you boot from the -Windows DVD. +wimlib 1.3.0 has added experimental support for Windows. See the file +"README.WINDOWS" for more details. + + WIM FILES + +A Windows Imaging (WIM) file is an archive. Like some other archive formats +such as ZIP, files in WIM archives may be compressed. WIM archives support two +Microsoft-specific compression formats: LZX and XPRESS. Both are based on LZ77 +and Huffman encoding, and both are supported by wimlib. + +Unlike ZIP files, WIM files can contain multiple independent toplevel directory +trees known as images. While each image has its own metadata describing a +directory tree and file access modes, files are not duplicated for each image; +instead, each file is included only once in the entire WIM. Microsoft did this +so that in one WIM file, they could do things like have 5 different versions of +Windows that are almost exactly the same. -You can find Windows PE on the ISO filesystem on the installation DVD for both -Windows 7 and Windows 8. I don't have a DVD for Vista but it should be on there -too. The Windows PE image a WIM file, `sources/boot.wim', on the ISO -filesystem. Windows PE can also be found in the Windows Automated Installation -Kit (WAIK), which is free to download from Microsoft, inside the `WinPE.cab' -file, which you can extract if you install the `cabextract' program. +Microsoft provides documentation for the WIM file format, XPRESS compression +format, and LZX compression format. The XPRESS documentation is acceptable, but +the LZX documentation is not entirely correct, and the WIM documentation itself +is incomplete. + +A WIM file may be either stand-alone or split into multiple parts. + + PROGRAMS wimlib provides a public API for other programs to use, but also comes with two -programs: `imagex' and `mkwinpeimg'. +programs: `imagex' and `mkwinpeimg'. `imagex' is intended to be like the imagex.exe program from Windows. `imagex' can be used to create, extract, and mount WIM files. Both read-only and @@ -29,62 +41,172 @@ details. `mkwinpeimg' is shell script that makes it easy to create a customized bootable image of Windows PE that can be put on a CD or USB drive, or published on a -server for PXE booting. See the main page `doc/mkwinpeiso.1' for more details. +server for PXE booting. See the main page `doc/mkwinpeimg.1' for more details. -Wimlib can also be used to handle larger WIM files such as the `install.wim' -file that comes on the Windows DVD; however, this has not been well tested. +There is an additional program, `wimapply', that is not installed by default. +It can be used to build a small executable with the ability to apply a WIM image +from a standalone WIM, without having to build the whole shared library. This +could be useful on Linux boot clients that only need to be able to apply a WIM, +not capture/split/join/append/export/mount a WIM. See `programs/wimapply.c'. -An earlier version of Wimlib is being used to deploy Windows 7 from the Ultimate -Deployment Appliance. For more information see -http://www.ultimatedeployment.org/. + COMPRESSION RATIO + +wimlib can create XPRESS or LZX compressed WIM archives. Currently, the XPRESS +compression ratio is slightly better than that provided by Microsoft's software, +while the LZX compression ratio is approaching that of Microsoft's software but +is not quite there yet. Running time is as good as or better than Microsoft's +software, especially with multithreaded compression, available in wimlib v1.1.0 +and later. + +The following tables compare the compression ratio and performance for creating +a compressed Windows PE image (disk usage of about 524 MB, uncompressed WIM size +361 MB): + + Table 1. WIM size + + XPRESS Compression LZX Compression + wimlib imagex (v1.2.1): 138,971,353 bytes 131,379,943 bytes + Microsoft imagex.exe: 140,406,981 bytes 127,249,176 bytes + + Table 2. Time to create WIM + + XPRESS Compression LZX Compression + wimlib imagex (v1.2.1, 2 threads): 11 sec 17 sec + Microsoft imagex.exe: 25 sec 89 sec + + NTFS SUPPORT + +As of version 1.0.0, wimlib supports capturing and applying images directly to +NTFS volumes. This was made possible with the help of libntfs-3g from the +NTFS-3g project. This feature supports capturing and restoring NTFS-specific +data such as security descriptors, alternate data streams, and reparse point +data. + +The code for NTFS image capture and image application is complete enough that it +is possible to apply an image from the "install.wim" contained in recent Windows +installation media (Vista, Windows 7, or Windows 8) directly to a NTFS volume, +and then boot Windows from it after preparing the Boot Configuration Data. In +addition, a Windows installation can be captured (or backed up) into a WIM file, +and then re-applied later. + + WINDOWS PE + +A major use for this library is to create customized images of Windows PE, the +Windows Preinstallation Environment, without having to rely on Windows. Windows +PE is a lightweight version of Windows that can run entirely from memory and can +be used to install Windows from local media or a network drive or perform +maintenance. Windows PE is the operating system that runs when you boot from +the Windows installation media. + +You can find Windows PE on the installation DVD for Windows Vista, Windows 7, or +Windows 8, in the file `sources/boot.wim'. Windows PE can also be found in the +Windows Automated Installation Kit (WAIK), which is free to download from +Microsoft, inside the `WinPE.cab' file, which you can extract if you install +either the `cabextract' or `p7zip' programs. + +In addition, Windows installations and recovery partitions frequently contain a +WIM containing an image of the Windows Recovery Environment, which is similar to +Windows PE. -------------------------------------------------------------------------------- + DEPENDENCIES + +* libxml2 (required) + This is a commonly used free library to read and write XML files. You + likely already have it installed as a dependency for some other program. + For more information see http://xmlsoft.org/. + +* libfuse (optional but highly recommended) + Unless configured with --without-fuse, wimlib requires a non-ancient + version of libfuse to be installed. Most GNU/Linux distributions + already include this, but make sure you have the libfuse package + installed, and also libfuse-dev if your distribution distributes header + files separately. FUSE also requires a kernel module. If the kernel + module is available it will automatically be loaded if you try to mount + a WIM file. For more information see http://fuse.sourceforge.net/. + FUSE is also available for FreeBSD. + +* libntfs-3g (optional but highly recommended) + Unless configured with --without-ntfs-3g, wimlib requires the library + and headers for libntfs-3g version 2011-4-12 or later to be installed. + Versions dated 2010-3-6 and earlier do not work because they are missing + the header xattrs.h (and the file xattrs.c, which contains functions we + need). libntfs-3g version 2013-1-13 is compatible only with wimlib + 1.2.4 and later. + +* OpenSSL / libcrypto (optional) + wimlib can use the SHA1 message digest code from OpenSSL instead of + compiling in yet another SHA1 implementation. (See LICENSE section.) + +* cdrkit (optional) +* mtools (optional) +* syslinux (optional) +* cabextract (optional) + The `mkwinpeimg' shell script will look for several other programs + depending on what options are given to it. Depending on your GNU/Linux + distribution, you may already have these programs installed, or they may + be in the software repository. Making an ISO filesystem requires + `mkisofs' from `cdrkit' (http://www.cdrkit.org). Making a disk image + requires `mtools' (http://www.gnu.org/software/mtools) and `syslinux' + (http://www.syslinux.org). Retrieving files from the Windows Automated + Installation Kit requires `cabextract' (http://www.cabextract.org.uk). CONFIGURATION Besides the various well-known options, the following options can be passed to wimlib's `configure' script: +--without-ntfs-3g + If libntfs-3g is not available or is not version 2011-4-12 or later, + wimlib can be built without it, in which case it will not be possible to + apply or capture images directly to/from NTFS volumes. + --without-fuse If libfuse or the FUSE kernel module is not available, wimlib can be compiled with --without-fuse. This will remove the ability to mount and - unmount WIM files. wimlib_mount() and wimlib_unmount() will fail with - WIMLIB_ERR_UNSUPPORTED. + unmount WIM files. ---disable-libcrypto +--without-libcrypto Build in functions for SHA1 rather than using external SHA1 functions from libcrypto (part of OpenSSL). The default is to use libcrypto if it is found on the system. +--enable-xattr, --disable-xattr + Enable or disable support for the extended-attributes interface to NTFS + alternate data streams in mounted WIMs. To support these, wimlib + requires that the setxattr() function and the attr/xattr.h header are + available. The default is to autodetect whether support is possible. + +--disable-multithreaded-compression + By default, data will be compressed using multiple threads when writing + a WIM, unless only 1 processor is detected. Specify this option to + disable support for this. + --enable-ssse3-sha1 Use a very fast assembly language implementation of SHA1 from Intel. Only use this if the build target supports the SSSE3 instructions. --disable-custom-memory-allocator - If this option is given, MALLOC(), FREE(), CALLOC(), and STRDUP() will - directly call the appropriate functions in the C library. - wimlib_set_memory_allocator() will fail with WIMLIB_ERR_UNSUPPORTED. + If this option is given, a very small amount of space will be saved by + removing support for the wimlib_set_memory_allocator() function. imagex + will be unaffected. --disable-verify-compression - Unless this option is given, every time wimlib compresses a data block - it will decompress it into a temporary buffer and abort() the program - with an error message if the decompressed data does not exactly match - the original data. This is to find bugs. + Unless this option is given, every time wimlib compresses a data block, + it will decompress it into a temporary buffer and abort the program with + an error message if the decompressed data does not exactly match the + original data. This only makes compression about 10% slower. However, + this checking can probably be safely disabled because there are no known + bugs in the compression code, and the SHA1 message digest of every + extracted file is checked anyway. --disable-error-messages - Removes all error messages from the library. If left in, they still - have to explicitly turned on with wimlib_set_print_errors() in order to - see them. Also, error codes will still be returned regardless of - whether error messages are printed or not. - - If --disable-error-messages is given, wimlib_set_print_errors() will - fail with WIMLIB_ERR_UNSUPPORTED if the action is to turn error messages - on. + Save some space by removing all error messages from the library. --disable-assertions - Remove all assertions. Without this option, wimlib will abort() the - program if an assertion fails. An assertion failure should only occur - if there is a bug in wimlib. + Remove all assertions, even the ones that are included by default. + +--enable-more-assertions + Enable assertions that are not included by default. --enable-debug Include debugging messages. Only use this option if you have found a @@ -94,102 +216,137 @@ wimlib's `configure' script: Include more debugging messages. Only use this option if you have found a bug in the library. -------------------------------------------------------------------------------- - - DEPENDENCIES - -Wimlib requires libxml2 to build. This is a commonly used free library to read -and write XML files. You likely already have it installed as a dependency for -some other program. For more information see http://xmlsoft.org/. - -Wimlib also requires libfuse to build (unless configured with --without-fuse; -see above). Most GNU/Linux distributions already include this, but make sure -you have the libfuse package installed (libfuse-dev if your distribution -distributes header files separately). FUSE also requires a kernel module. If -the kernel module is available it will automatically be loaded if you try to -mount a WIM file. Wimlib has only been tested with the Linux version of FUSE. -For more information see http://fuse.sourceforge.net/. - -The `mkwinpeimg' shell script will look for several other programs depending on -what options are given to it. Depending on your GNU/Linux distribution, you may -already have these programs installed, or they may be in the software -repository. Making an ISO filesystem requires `mkisofs' from `cdrkit' -(http://www.cdrkit.org). Making a disk image requires `mtools' -(http://www.gnu.org/software/mtools) and `syslinux' (http://www.syslinux.org). -Retrieving files from the Windows Automated Installation Kit requires -`cabextract' (http://www.cabextract.org.uk). - ------------------------------------------------------------------------------- - PORTABILITY wimlib has mostly been developed and tested on x86_64 (64-bit) GNU/Linux. It has been tested on x86 (32-bit) GNU/Linux occasionally. -It can also be compiled and run on FreeBSD. +wimlib may work on FreeBSD and Mac OS X. However, this is not well tested. If +you do not have libntfs-3g 2011-4-12 or later available, you must configure +wimlib with --without-ntfs-3g. On FreeBSD, before mounting a WIM you need to +load the POSIX message queue module (run `kldload mqueuefs'). -wimlib should work on big endian machines but it has not been tested. +The code pays attention to endianness, so it should work on big-endian +architectures, but I've never tested this so do not expect it to work. -There are no plans to port wimlib to Windows since the programming interface on -Windows is very different and Microsoft's imagex.exe is already available. +As of wimlib 1.3.0, there is experimental support for Microsoft Windows. I +expect it to work on Windows Vista and later, although I have only tested it on +Windows 7. See README.WINDOWS for more information. ------------------------------------------------------------------------------- - - REFERENCES + REFERENCES The WIM file format is specified in a document that can be found in the Microsoft Download Center. There is a similar document that specifies the LZX compression format, and a document that specifies the XPRESS compression format. -However, some aspects of these formats are poorly documented. Some particularly -poorly documented parts of the formats have had comments added in various places -in the library. - -lzx-decomp.c, the code to decompress WIM file resources that are compressed +However, many parts of these formats are poorly documented, and some parts have +no documentation whatsoever. Some particularly poorly documented parts of the +formats have had comments added in various places in the library. Please see +the code and/or ask me if you have any questions about the WIM file format as it +exists in reality and not as it exists in Microsoft's poorly written +documentation. + +The code in ntfs-apply.c and ntfs-capture.c uses the NTFS-3g library, which is a +library for reading and writing to NTFS filesystems (the filesystem used by +recent versions of Windows). See +http://www.tuxera.com/community/ntfs-3g-download/ for more information. + +lzx-decompress.c, the code to decompress WIM file resources that are compressed using LZX compression, is originally based on code from the cabextract project -(http://www.cabextract.org.uk). - -lzx-comp.c, the code to compress WIM file resources using LZX compression, is -originally based on code written by Matthew Russotto (www.russotto.net/chm/). +(http://www.cabextract.org.uk). -lz.c, the code to find LZ77 matches, is based on code from zlib. +lzx-compress.c, the code to compress WIM file resources using LZX compression, +is originally based on code written by Matthew Russotto (www.russotto.net/chm/). -sha1.c and sha1.h, the code to compute SHA1 message digests of WIM resources or -of the WIM file itself in the case of integrity checks, are based on code from -GNU coreutils. +lz77.c, the code to find LZ77 matches (used for both XPRESS and LZX compression), +is based on code from zlib. A very limited number of other free programs can handle some parts of the WIM -file format. 7-zip is able to extract and create WIMs and files in many other -archive formats. However, WIMLIB is designed specifically to handle WIM files -and provides features previously only available in Microsoft's imagex.exe, such -as the ability to mount WIMs read-write. - ------------------------------------------------------------------------------- +file format. 7-zip is able to extract and create WIMs (as well as files in many +other archive formats). However, wimlib is designed specifically to handle WIM +files and provides features previously only available in Microsoft's imagex.exe, +such as the ability to mount WIMs read-write as well as read-only, the ability +to create LZX or XPRESS compressed WIMs, and the correct handling of security +descriptors and hard links. + +An earlier version of wimlib is being used to deploy Windows 7 from the Ultimate +Deployment Appliance. For more information see +http://www.ultimatedeployment.org/. + +You can see the documentation about Microsoft's version of the imagex program at +http://technet.microsoft.com/en-us/library/cc749447(v=ws.10).aspx, so you can +see how it compares to the version provided by this library. + + GNU/Linux equivalents of WIM format + +What's the equivalent way to capture the filesystem of a GNU/Linux operating +system into an archive file? You have a few options: + +SquashFS: + SquashFS (http://squashfs.sourceforge.net/) provides a compressed, + read-only filesystem for Linux, and it's probably the closest equivalent + of the WIM format and better designed. Although you can't mount + SquashFS read-write, when wimlib does this for WIM files it's really an + illusion since the WIM isn't actually modified until the image is + unmounted. Multiple top-level images in SquashFS files are not + supported, although nothing stops you from just putting each image in a + separate directory. + +FSArchiver: + FSArchiver (http://www.fsarchiver.org/Main_Page) is not widely used, but + it appears to have some features quite similar to the WIM format. + +Tar: + The well-known tar format can usually capture a UNIX filesystem just + fine, and compressing the tar file produces a good compression ratio + (better than WIM, especially if using XZ compression), but there is no + support for random access, file deduplication, multiple images per + archive, or extended attributes. + +Zip: + Zip shares some features with WIM but is not designed to store entire + filesystems. + +7z: + The 7z format has some nice features but is unfortunately not designed + with UNIX in mind. MORE INFORMATION See the manual pages for `imagex', the manual pages for the subcommands of `imagex', and the manual page for `mkwinpeimg'. -As of version 0.5.0, Wimlib's public API is documented. Doxygen is required to +As of version 0.5.0, wimlib's public API is documented. Doxygen is required to build the documentation. To build the documentation, run `configure', then enter the directory `doc' and run `doxygen'. The HTML documentation will be created in a directory named `html'. ------------------------------------------------------------------------------- - LICENSE -Wimlib is released under the GNU LGPL version 2.1 or later. The files in the -`programs' directory are released under the GPL version 3. +As of version 1.0.0, wimlib is released under the GNU GPL version 3.0 or later. +This includes the files in the `programs' directory as well as the files in the +`src' directory. + +wimlib is independently developed and does not contain any code, data, or files +copyrighted by Microsoft. It is not known to be affected by any patents. + +By default, wimlib will be linked to the system library "libcrypto", which +probably will be OpenSSL. Some people believe that GPL code cannot be linked to +OpenSSL without a linking exception. As far as I know, I cannot officially +include a linking exception with the license of this library because several +files could be considered derived works of code copyrighted by others. If you +believe this to be a problem, configure with --without-libcrypto to avoid +linking with OpenSSL. There is no difference in functionality--- there will +just be stand-alone SHA1 message digest code built into the library. ------------------------------------------------------------------------------- + DISCLAIMER - DISCLAIMER +wimlib is experimental. Use Microsoft's `imagex.exe' if you want to make sure +your WIM files are made correctly (but beware: Microsoft's version contains some +bugs). -Wimlib is experimental. Use Microsoft's `imagex.exe' if you want to make sure -your WIM files are made correctly. Please submit a bug report (to -ebiggers3@gmail.com) if you find a bug. +Please submit a bug report (to ebiggers3@gmail.com) if you find a bug in wimlib. Some parts of the WIM file format are poorly documented or even completely -undocumented, so these parts had to be reverse engineered. +undocumented, so I've just had to do the best I can to read and write WIMs that +appear to be compatible with Microsoft's software.