X-Git-Url: https://wimlib.net/git/?a=blobdiff_plain;f=README;h=1106609428566808f3eae6a47c58e0bd0a34b08f;hb=3ceffe51aa820d5e804d859935bf34584be08fe2;hp=0f15a36fe4279c2a034d4c1dda5bcbef899d1ecf;hpb=ac4e9d3b603a8abcc99965ed99576fd0721f8ccb;p=wimlib diff --git a/README b/README index 0f15a36f..11066094 100644 --- a/README +++ b/README @@ -1,297 +1,297 @@ - WIMLIB - -This is wimlib version 1.0.3 (September 2012). wimlib can be used to read, -write, and mount files in the Windows Imaging Format (WIM files). These -files are normally created by using the `imagex.exe' utility on Windows, -but this library provides a free implementation of imagex for UNIX-based -systems. - - WIM FILES - -A Windows Imaging (WIM) file is an archive. Like some other archive formats -such as ZIP, files in WIM archives may be compressed. WIM archives support two -Microsoft-specific compression formats: LZX and XPRESS. Both are based on LZ77 -and Huffman encoding, and both are supported by wimlib. - -Unlike ZIP files, WIM files can contain multiple independent toplevel directory -trees known as images. While each image has its own metadata describing a -directory tree and file access modes, files are not duplicated for each image; -instead, each file is included only once in the entire WIM. Microsoft did this -so that in one WIM file, they could do things like have 5 different versions of -Windows that are almost exactly the same. - -Microsoft provides documentation for the WIM file format, XPRESS compression -format, and LZX compression format. The XPRESS documentation is acceptable, but -the LZX documentation is not entirely correct, and the WIM documentation itself -is very incomplete and is of unacceptable quality. - -A WIM file may be either stand-alone or split into multiple parts. - - WINDOWS PE - -A major use for this library is to create customized images of Windows PE, the -Windows Preinstallation Environment, without having to rely on Windows. Windows -PE is a lightweight version of Windows that can run entirely from memory and can -be used to install Windows from local media or a network drive or perform -maintenance. Windows PE is the operating system that runs when you boot from + INTRODUCTION + +This is wimlib version 1.9.2 (June 2016). wimlib is a C library for +creating, modifying, extracting, and mounting files in the Windows Imaging +Format (WIM files). wimlib and its command-line frontend 'wimlib-imagex' +provide a free and cross-platform alternative to Microsoft's WIMGAPI, ImageX, +and DISM. + + INSTALLATION + +To install wimlib and wimlib-imagex on UNIX-like systems, you can compile from +source (e.g. './configure && make && sudo make install'). Alternatively, check +if a package has already been prepared for your operating system. Example files +for Debian and RPM packaging are in the debian/ and rpm/ directories. + +To install wimlib and wimlib-imagex on Windows, just download and extract the +ZIP file containing the latest binaries. See README.WINDOWS for more details. + +All official wimlib releases are available from https://wimlib.net. + + WIM FILES + +A Windows Imaging (WIM) file is an archive designed primarily for archiving +Windows filesystems. However, it can be used on other platforms as well, with +some limitations. Like some other archive formats such as ZIP, files in WIM +archives may be compressed. WIM archives support multiple compression formats, +including LZX, XPRESS, and LZMS. All these formats are supported by wimlib. + +A WIM archive contains one or more "images", each of which is a logically +independent directory tree. Each image has a 1-based index and usually a name. + +WIM archives provide data deduplication at the level of full file contents. In +other words, each unique "file contents" is only stored once in the archive, +regardless of how many files have that contents across all images. + +A WIM archive may be either stand-alone or split into multiple parts. + +An update of the WIM format --- first added by Microsoft for Windows 8 --- +supports solid-mode compression. This refers to files being compressed together +(e.g. as in a .tar.xz or .7z archive) rather than separately (e.g. as in a .zip +archive). This usually produces a much better compression ratio. Solid +archives are sometimes called "ESD files" by Microsoft and may have the ".esd" +file extension rather than ".wim". They are supported in wimlib since v1.6.0. + + IMAGEX IMPLEMENTATION + +wimlib itself is a C library, and it provides a documented public API (See: +https://wimlib.net/apidoc) for other programs to use. However, it is also +distributed with a command-line program called "wimlib-imagex" that uses this +library to implement an imaging tool similar to Microsoft's ImageX. +wimlib-imagex supports almost all the capabilities of Microsoft's ImageX as well +as additional capabilities. wimlib-imagex works on both UNIX-like systems and +Windows, although some features differ between the platforms. + +Run `wimlib-imagex' with no arguments to see an overview of the available +commands and their syntax. For additional documentation: + + * If you have installed wimlib-imagex on a UNIX-like system, you will find + further documentation in the man pages; run `man wimlib-imagex' to get + started. + + * If you have downloaded the Windows binary distribution, you will find the + documentation for wimlib-imagex in PDF format in the "doc" directory, + ready for viewing with any PDF viewer. Please note that although the PDF + files are converted from UNIX-style "man pages", they do document + Windows-specific behavior when appropriate. + + COMPRESSION + +wimlib (and wimlib-imagex) can create XPRESS, LZX, and LZMS compressed WIM +archives. wimlib's compression codecs usually outperform and outcompress their +closed-source Microsoft equivalents. Multiple compression levels and chunk +sizes as well as solid mode compression are supported. Compression is +multithreaded by default. Detailed benchmark results and descriptions of the +algorithms used can be found at https://wimlib.net/compression.html. + + NTFS SUPPORT + +WIM images may contain data, such as alternate data streams and +compression/encryption flags, that are best represented on the NTFS filesystem +used on Windows. Also, WIM images may contain security descriptors which are +specific to Windows and cannot be represented on other operating systems. +wimlib handles this NTFS-specific or Windows-specific data in a +platform-dependent way: + + * In the Windows version of wimlib and wimlib-imagex, NTFS-specific and + Windows-specific data are supported natively. + + * In the UNIX version of wimlib and wimlib-imagex, NTFS-specific and + Windows-specific data are ordinarily ignored; however, there is also special + support for capturing and extracting images directly to/from unmounted NTFS + volumes. This was made possible with the help of libntfs-3g from the + NTFS-3G project. + +For both platforms the code for NTFS capture and extraction is complete enough +that it is possible to apply an image from the "install.wim" contained in recent +Windows installation media (Vista or later) directly to an NTFS filesystem, and +then boot Windows from it after preparing the Boot Configuration Data. In +addition, a Windows installation can be captured (or backed up) into a WIM file, +and then re-applied later. + + WINDOWS PE + +A major use for wimlib and wimlib-imagex is to create customized images of +Windows PE, the Windows Preinstallation Environment, on either UNIX-like systems +or Windows without having to rely on Microsoft's software and its restrictions +and limitations. + +Windows PE is a lightweight version of Windows that can run entirely from memory +and can be used to install Windows from local media or a network drive or +perform maintenance. It is the operating system that runs when you boot from the Windows installation media. -You can find Windows PE on the installation DVD for Windows Vista, Windows 7, or -Windows 8, in the file `sources/boot.wim'. Windows PE can also be found in the -Windows Automated Installation Kit (WAIK), which is free to download from -Microsoft, inside the `WinPE.cab' file, which you can extract if you install -either the `cabextract' or `p7zip' programs. +You can find Windows PE on the installation media for Windows (Vista or later) +as the file `sources/boot.wim'. Windows PE can also be found in the Windows +Automated Installation Kit (WAIK), which is free to download from Microsoft, +inside the `WinPE.cab' file, which you can extract natively on Windows, or on +UNIX-like systems if you install either the `cabextract' or `p7zip' programs. In addition, Windows installations and recovery partitions frequently contain a WIM containing an image of the Windows Recovery Environment, which is similar to Windows PE. - NTFS SUPPORT - -As of version 1.0.0, wimlib supports capturing and applying images directly to -NTFS volumes. This was made possible with the help of libntfs-3g from the -NTFS-3g project. This feature supports capturing and restoring NTFS-specific -data such as security descriptors, alternate data streams, and reparse point -data. - -The code for NTFS image capture and image application is complete enough that it -is possible to apply an image from the "install.wim" contained in recent Windows -installation media (Vista, Windows 7, or Windows 8) directly to a NTFS volume, -and then boot Windows from it after preparing the Boot Configuration Data. In -addition, a Windows installation can be captured (or backed up) into a WIM file, -and then re-applied later. - - PROGRAMS +A shell script `mkwinpeimg' is distributed with wimlib on UNIX-like systems to +ease the process of creating and customizing a bootable Windows PE image. + + DEPENDENCIES + +This section documents the dependencies of wimlib and the programs distributed +with it, when building for a UNIX-like system from source. If you have +downloaded the Windows binary distribution of wimlib and wimlib-imagex then all +dependencies were already included and this section is irrelevant. + +* libxml2 (required) + This is a commonly used free library to read and write XML documents. + Almost all Linux distributions should include this; however, you may + need to install the header files, which might be in a package named + "libxml2-dev" or similar. For more information see http://xmlsoft.org/. + +* libfuse (optional but recommended) + Unless configured --without-fuse, wimlib requires a non-ancient version + of libfuse. Most Linux distributions already include this, but make + sure you have the libfuse package installed, and also libfuse-dev if + your distribution distributes header files separately. FUSE also + requires a kernel module. If the kernel module is available it should + automatically be loaded if you try to mount a WIM image. For more + information see http://fuse.sourceforge.net/. + +* libattr (optional but recommended) + Unless configured --without-fuse, wimlib also requires libattr. Almost + all Linux distributions should include this; however, you may need to + install the header files, which might be in a package named "attr-dev", + "libattr1-dev", or similar. + +* libntfs-3g (optional but recommended) + Unless configured --without-ntfs-3g, wimlib requires the library and + headers for libntfs-3g to be installed. The minimum required version is + 2011-4-12, but newer versions contain important bug fixes. + +* OpenSSL / libcrypto (optional) + wimlib can use the SHA-1 message digest implementation from libcrypto + (usually provided by OpenSSL) instead of compiling in yet another SHA-1 + implementation. -wimlib provides a public API for other programs to use, but also comes with two -programs: `imagex' and `mkwinpeimg'. +* cdrkit (optional) +* mtools (optional) +* syslinux (optional) +* cabextract (optional) + The `mkwinpeimg' shell script will look for several other programs + depending on what options are given to it. Depending on your Linux + distribution, you may already have these programs installed, or they may + be in the software repository. Making an ISO filesystem requires + `mkisofs' from `cdrkit' (http://www.cdrkit.org). Making a disk image + requires `mtools' (http://www.gnu.org/software/mtools) and `syslinux' + (http://www.syslinux.org). Retrieving files from the Windows Automated + Installation Kit requires `cabextract' (http://www.cabextract.org.uk). -`imagex' is intended to be like the imagex.exe program from Windows. `imagex' -can be used to create, extract, and mount WIM files. Both read-only and -read-write mounts are supported. See the man page `doc/imagex.1' for more -details. + CONFIGURATION -`mkwinpeimg' is shell script that makes it easy to create a customized bootable -image of Windows PE that can be put on a CD or USB drive, or published on a -server for PXE booting. See the main page `doc/mkwinpeiso.1' for more details. +This section documents the most important options that may be passed to the +"configure" script when building from source: - COMPRESSION RATIO +--without-ntfs-3g + If libntfs-3g is not available or is not version 2011-4-12 or later, + wimlib can be built without it, in which case it will not be possible to + capture or apply WIM images directly from/to NTFS volumes. -wimlib can create XPRESS or LZX compressed WIM archives. As of wimlib v1.0.3, -the XPRESS compression ratio is slightly better than that provided by -Microsoft's software, while the LZX compression ratio is approaching that of -Microsoft's software but is not quite there yet. Running time is as good as or -better than Microsoft's software. + The default is --with-ntfs-3g when building for any UNIX-like system, + and --without-ntfs-3g when building for Windows. -The following tables compare the compression ratio and performance for creating -a compressed Windows PE image (disk usage of about 524 MB, uncompressed WIM size -361 MB): +--without-fuse + The --without-fuse option disables support for mounting WIM images. + This removes dependencies on libfuse, librt, and libattr. The wimmount, + wimmountrw, and wimunmount commands will not work. - Table 1. WIM size + The default is --with-fuse when building for Linux, and --without-fuse + otherwise. - XPRESS Compression LZX Compression - wimlib imagex (v1.0.2): 145,283,871 bytes 139,288,293 bytes - wimlib imagex (v1.0.3): 139,288,293 bytes 131,379,869 bytes - Microsoft imagex.exe: 140,406,981 bytes 127,249,176 bytes +--without-libcrypto + Build in functions for SHA-1 rather than using external SHA-1 functions + from libcrypto (usually provided by OpenSSL). - Table 2. Time to create WIM + The default is to use libcrypto if it is found on your system. - XPRESS Compression LZX Compression - wimlib imagex (v1.0.2): 18 sec 49 sec - wimlib imagex (v1.0.3): 19 sec 30 sec - Microsoft imagex.exe: 25 sec 89 sec + PORTABILITY +wimlib works on both UNIX-like systems (Linux, Mac OS X, FreeBSD, etc.) and +Windows (XP and later). - DEPENDENCIES +As much code as possible is shared among all supported platforms, but there +necessarily are some differences in what features are supported on each platform +and how they are implemented. Most notable is that file tree scanning and +extraction are implemented separately for Windows, UNIX, and UNIX (NTFS-3G +mode), to ensure a fast and feature-rich implementation of each platform/mode. -* libxml2 - This is a commonly used free library to read and write XML files. You - likely already have it installed as a dependency for some other program. - For more information see http://xmlsoft.org/. +wimlib is mainly used on x86 and x86_64 CPUs, but it should also work on a +number of other GCC-supported 32-bit or 64-bit architectures. It has been +tested on the ARM and MIPS architectures. -* libfuse - Unless configured with --without-fuse, wimlib requires a non-ancient - version of libfuse to be installed. Most GNU/Linux distributions - already include this, but make sure you have the libfuse package - installed, and also libfuse-dev if your distribution distributes header - files separately. FUSE also requires a kernel module. If the kernel - module is available it will automatically be loaded if you try to mount - a WIM file. For more information see http://fuse.sourceforge.net/. - FUSE is also available for FreeBSD. +Currently, gcc and clang are the only supported compilers. A few nonstandard +extensions are used in the code. -* libntfs-3g - Unless configured with --without-ntfs-3g, wimlib requires the library - and headers for libntfs-3g version 2011-4-12 or later to be installed. - Versions dated 2010-3-6 and earlier do not work because they are missing - the header xattrs.h (and the file xattrs.c, which contains functions we - need). + REFERENCES -* cdrkit (optional) -* mtools (optional) -* syslinux (optional) -* cabextract (optional) - The `mkwinpeimg' shell script will look for several other programs - depending on what options are given to it. Depending on your GNU/Linux - distribution, you may already have these programs installed, or they may - be in the software repository. Making an ISO filesystem requires - `mkisofs' from `cdrkit' (http://www.cdrkit.org). Making a disk image - requires `mtools' (http://www.gnu.org/software/mtools) and `syslinux' - (http://www.syslinux.org). Retrieving files from the Windows Automated - Installation Kit requires `cabextract' (http://www.cabextract.org.uk). +The WIM file format is partially specified in a document that can be found in +the Microsoft Download Center. However, this document really only provides an +overview of the format and is not a formal specification. It also does not +cover later extensions of the format, such as solid resources. +With regards to the supported compression formats: - CONFIGURATION +- Microsoft has official documentation for XPRESS that is of reasonable quality. +- Microsoft has official documentation for LZX, but in two different documents, + neither of which is completely applicable to its use in the WIM format, and + the first of which contains multiple errors. +- There does not seem to be any official documentation for LZMS, so my comments + and code in src/lzms_decompress.c may in fact be the best documentation + available for this particular compression format. -Besides the various well-known options, the following options can be passed to -wimlib's `configure' script: +The algorithms used by wimlib's compression and decompression codecs are +inspired by a variety of sources, including open source projects and computer +science papers. ---without-ntfs-3g - If libntfs-3g is not available or is not the correct version, we can - build without it. wimlib will then not be able to apply or capture - images directly to NTFS volumes. +The code in ntfs-3g_apply.c and ntfs-3g_capture.c uses the NTFS-3G library, +which is a library for reading and writing to NTFS filesystems (the filesystem +used by recent versions of Windows). See +http://www.tuxera.com/community/ntfs-3g-download/ for more information. ---without-fuse - If libfuse or the FUSE kernel module is not available, wimlib can be - compiled with --without-fuse. This will remove the ability to mount and - unmount WIM files. wimlib_mount() and wimlib_unmount() will fail with - WIMLIB_ERR_UNSUPPORTED. +A limited number of other free programs can handle some parts of the WIM +file format: ---without-libcrypto - Build in functions for SHA1 rather than using external SHA1 functions - from libcrypto (part of OpenSSL). The default is to use libcrypto if it - is found on the system. - ---enable-xattr, --disable-xattr - Enable or disable support for the extended-attributes interface to NTFS - alternate data streams in mounted WIMs. To support these, we require - the setxattr() function and the attr/xattr.h header be available. The - default is to autodetect whether support is possible. - ---enable-ssse3-sha1 - Use a very fast assembly language implementation of SHA1 from Intel. - Only use this if the build target supports the SSSE3 instructions. - ---disable-custom-memory-allocator - If this option is given, MALLOC(), FREE(), CALLOC(), and STRDUP() will - directly call the appropriate functions in the C library. - wimlib_set_memory_allocator() will fail with WIMLIB_ERR_UNSUPPORTED. - ---disable-verify-compression - Unless this option is given, every time wimlib compresses a data block - it will decompress it into a temporary buffer and abort() the program - with an error message if the decompressed data does not exactly match - the original data. This is to find bugs. - ---disable-error-messages - Removes all error messages from the library. If left in, they still - have to explicitly turned on with wimlib_set_print_errors() in order to - see them. Also, error codes will still be returned regardless of - whether error messages are printed or not. - - If --disable-error-messages is given, wimlib_set_print_errors() will - fail with WIMLIB_ERR_UNSUPPORTED if the action is to turn error messages - on. - ---disable-assertions - Remove all assertions. Without this option, wimlib will abort() the - program if an assertion fails. An assertion failure should only occur - if there is a bug in wimlib. - ---enable-debug - Include debugging messages. Only use this option if you have found a - bug in the library. - ---enable-more-debug - Include more debugging messages. Only use this option if you have found - a bug in the library. - - PORTABILITY - -wimlib has mostly been developed and tested on x86_64 (64-bit) GNU/Linux. - -It has been tested on x86 (32-bit) GNU/Linux occasionally. - -wimlib may work on FreeBSD. However, this is not well tested. If you do not -have libntfs-3g 2011-4-12 or later available, you must configure with ---without-ntfs-3g. Also, GNU coreutils is needed to run the test suite. Before -mounting a WIM you need to load the POSIX message queue module (run `kldload -mqueuefs'). - -wimlib should work on big endian machines but it has not been tested. - -There are no plans to port wimlib to Windows since the programming interface on -Windows is very different and Microsoft's imagex.exe is already available. - - REFERENCES - -The WIM file format is specified in a document that can be found in the -Microsoft Download Center. There is a similar document that specifies the LZX -compression format, and a document that specifies the XPRESS compression format. -However, many parts of these formats are poorly documented, and some parts have -no documentation whatsoever. Some particularly poorly documented parts of the -formats have had comments added in various places in the library. Please see -the code and/or ask me if you have any questions about the WIM file format as it -exists in reality and not as it exists in Microsoft's poorly written -documentation. - -The code in ntfs-apply.c and ntfs-capture.c uses the NTFS-3g library, which is a -library for reading and writing to NTFS filesystems (the filesystem used by -recent versions of Windows). Additionally, the code in ntfs-3g-security.c is -mostly copied from NTFS-3g, but I'm hoping to get rid of this file eventually. -See http://www.tuxera.com/community/ntfs-3g-download/ for more information. - -lzx-decomp.c, the code to decompress WIM file resources that are compressed -using LZX compression, is originally based on code from the cabextract project -(http://www.cabextract.org.uk). - -lzx-comp.c, the code to compress WIM file resources using LZX compression, is -originally based on code written by Matthew Russotto (www.russotto.net/chm/). - -lz.c, the code to find LZ77 matches (used for both XPRESS and LZX compression), -is based on code from zlib. - -A very limited number of other free programs can handle some parts of the WIM -file format. 7-zip is able to extract and create WIMs (as well as files in many -other archive formats). However, wimlib is designed specifically to handle WIM -files and provides features previously only available in Microsoft's imagex.exe, -such as the ability to mount WIMs read-write as well as read-only, and the -ability to create LZX or XPRESS compressed WIMs. - -An earlier version of wimlib is being used to deploy Windows 7 from the Ultimate -Deployment Appliance. For more information see -http://www.ultimatedeployment.org/. + * 7-zip is able to extract and create WIMs (as well as files in many + other archive formats). However, wimlib is designed specifically to handle + WIM files and provides features previously only available in Microsoft's + implementation, such as the ability to mount WIMs read-write as well as + read-only, the ability to create compressed WIMs, the correct handling of + security descriptors and hard links, support for LZMS compression, and + support for solid archives. + * ImagePyX (https://github.com/maxpat78/ImagePyX) is a Python program that + provides some capabilities of wimlib-imagex, with the help of external + compression codecs. -You can see the documentation about Microsoft's version of the imagex program at -http://technet.microsoft.com/en-us/library/cc749447(v=ws.10).aspx, so you can -see how it compares. +If you are looking for an archive format that provides features similar to WIM +but was designed primarily for UNIX, you may want to consider SquashFS +(http://squashfs.sourceforge.net/). However, you may find that wimlib works +surprisingly well on UNIX. It will store hard links and symbolic links, and it +has optional support for storing UNIX owners, groups, modes, and special files +such as device nodes and FIFOs. Actually, I use it to back up my own files on +Linux! - MORE INFORMATION + HISTORY -See the manual pages for `imagex', the manual pages for the subcommands of -`imagex', and the manual page for `mkwinpeimg'. +wimlib was originally a project started by Carl Thijssen for use on Linux in the +Ultimate Deployment Appliance (http://www.ultimatedeployment.org/). Since then +the code has been entirely rewritten and improved (main author: Eric Biggers). +Windows support has been available since version 1.3.0 (March 2013). A list of +version-to-version changes can be found in the NEWS file. -As of version 0.5.0, wimlib's public API is documented. Doxygen is required to -build the documentation. To build the documentation, run `configure', then -enter the directory `doc' and run `doxygen'. The HTML documentation will be -created in a directory named `html'. + NOTICES - LICENSE +wimlib is free software that comes with NO WARRANTY, to the extent permitted by +law. See the COPYING file for more details. -As of version 1.0.0, wimlib is released under the GNU GPL version 3.0 or later. -This includes the files in the `programs' directory as well as the files in the -`src' directory. +Bug reports, suggestions, and other contributions are appreciated and may be +sent via email to ebiggers3@gmail.com or posted to https://wimlib.net/forums. wimlib is independently developed and does not contain any code, data, or files copyrighted by Microsoft. It is not known to be affected by any patents. - DISCLAIMER - -wimlib is experimental. Use Microsoft's `imagex.exe' if you want to make sure -your WIM files are made correctly (but beware: Microsoft's version contains some -bugs). - -Please submit a bug report (to ebiggers3@gmail.com) if you find a bug in wimlib. +On UNIX-like systems, if you do not want wimlib to be dynamically linked with +libcrypto (OpenSSL), configure with --without-libcrypto. This replaces the SHA1 +implementation with built-in code and there will be no difference in +functionality. -Some parts of the WIM file format are poorly documented or even completely -undocumented, so I've just had to do the best I can to read and write WIMs that -appear to be compatible with Microsoft's software. +Note: copyright years may be listed using range notation, e.g., 2012-2016, +indicating that every year in the range, inclusive, is a copyrightable year that +would otherwise be listed individually.