Multithreaded stream writing fixes

[wimlib] / doc / imagex-capture.1.in

.TH IMAGEX "1" "March 2013" "@IMAGEX_PROGNAME@ @VERSION@" "User Commands"
.SH NAME
@IMAGEX_PROGNAME@-capture, @IMAGEX_PROGNAME@-append \- Create or append a WIM image

.SH SYNOPSIS
\fB@IMAGEX_PROGNAME@ capture\fR \fISOURCE\fR \fIWIMFILE\fR [\fIIMAGE_NAME\fR] \
[\fIDESCRIPTION\fR] [\fIOPTION\fR]...
.br
\fB@IMAGEX_PROGNAME@ append\fR \fISOURCE\fR \fIWIMFILE\fR [\fIIMAGE_NAME\fR] \
[\fIDESCRIPTION\fR] [\fIOPTION\fR]...

.SH DESCRIPTION
.PP

The \fB@IMAGEX_PROGNAME@ capture\fR and \fB@IMAGEX_PROGNAME@ append\fR commands create a Windows
Imaging (WIM) image from a directory tree.  The \fB@IMAGEX_PROGNAME@ capture\fR command
creates a new WIM file containing the captured image, while the \fB@IMAGEX_PROGNAME@
append\fR command appends the captured image to an existing WIM file.

Note: this man page primarily documents the UNIX behavior.  See \fBWINDOWS
VERSION\fR for information specific to the Windows build of wimlib.

A WIM image is an independent directory tree in the WIM file.  A WIM file may
contain any number of separate images.  However, files are stored only one time
in the entire WIM, regardless of how many images the file appears in.

\fISOURCE\fR specifies the location of the files to create the new WIM image
from.  If \fISOURCE\fR is a directory, the WIM image is captured from that
directory.  Alternatively, if \fISOURCE\fR is a regular file or block device, it
is interpreted as a NTFS volume from which a WIM image is to be captured.  Still
alternatively, if the \fB--source-list\fR option is given, \fISOURCE\fR is
interpreted as a file that itself provides a list of files and directories to
include in the new WIM image.

\fIIMAGE_NAME\fR and \fIDESCRIPTION\fR specify the name and description of the
new image.  If \fIIMAGE_NAME\fR is not given, it is taken to be the same as the
base name of \fISOURCE\fR.  If \fIDESCRIPTION\fR is not given, no description is
given to the new image.

.SH NORMAL MODE

The normal image capture mode is entered when \fISOURCE\fR specifies a
directory.  The WIM image will be captured from the directory tree rooted at
this directory.  The directory may be on any type of filesystem.

In this mode, the following information is captured from the directory tree:

.IP \[bu] 4
The "normal" name and contents of each file and directory
.IP \[bu]
File and directory creation, access, and modification timestamps to the nearest
100 nanoseconds, if supported by the underlying filesystem
.IP \[bu]
Hard links and symbolic links

.PP

However, in this mode, the following information is \fInot\fR captured from the
directory tree:

.IP \[bu] 4
File permissions.  The resulting WIM image will not contain any security
descriptors because the format of the security descriptors is Windows-specific,
and they cannot contain UNIX file modes.  (Exception: see the \fB--unix-data\fR
option.)

.IP \[bu]
No alternate data streams will be captured, since these do not exist on
POSIX-compliant filesystems.  The resulting WIM image will not contain any
alternate data streams.  (Exception: see the \fB--unix-data\fR option.)

.SH NTFS MODE

A special image capture mode is entered when \fISOURCE\fR is a regular file or
block device.  \fISOURCE\fR is interpreted as an NTFS volume and opened using
libntfs-3g.  If successful, a WIM image is captured containing the contents of
the NTFS volume, including NTFS-specific data.

Please note that the NTFS image capture mode is \fInot\fR entered if
\fISOURCE\fR is a directory, even if a NTFS filesystem is mounted on
\fISOURCE\fR.  You must specify the NTFS volume itself (and it must be
unmounted, and you must have permission to read from it).

More specifically, in this mode, the following types of information are captured
from the NTFS volume:

.IP \[bu] 4
All data streams of all files, including the un-named data stream as well as all
named data streams.
.IP \[bu]
Reparse points, including symbolic links, junction points, and other reparse
points.
.IP \[bu]
File and directory creation, access, and modification timestamps from NTFS
inodes (these have a resolution of 100 nanoseconds).
.IP \[bu]
The security descriptor for each NTFS inode.
.IP \[bu]
File attribute flags.
.IP \[bu]
All names of all files, including names in the Win32 namespace, DOS namespace,
Win32+DOS namespace, and POSIX namespace.  This includes hard links.

.SH SOURCE LIST MODE

Yet another capture mode is entered when the \fB--source-list\fR option is
given.  It is essentially an extension of the \fBNORMAL MODE\fR that allows
multiple files or directories to be incorporated into a WIM image using a single
\fB@IMAGEX_PROGNAME@ capture\fR or \fB@IMAGEX_PROGNAME@ append\fR command.  See the documentation for
\fB--source-list\fR below.

.SH WINDOWS VERSION

This section documents the differences between \fB@IMAGEX_PROGNAME@ capture\fR and
\fB@IMAGEX_PROGNAME@ append\fR in the Windows builds of wimlib versus the rest of this man
page, which is written to document UNIX build.

\fB@IMAGEX_PROGNAME@ capture\fR and \fB@IMAGEX_PROGNAME@ append\fR do not have separate "normal" and
"NTFS" modes on Windows.  There is simply one mode, and it uses the Windows API
to capture NTFS-specific information, including alternate data streams, reparse
points, hard links, and file attributes.  So, you essentially get the advantages
of the "NTFS mode" documented above, but you can capture a WIM image from any
directory, not just an entire NTFS volume.  This is essentially the same
behavior as Microsoft's ImageX.

The \fB--source-list\fR option is supported on Windows, but the
\fB--dereference\fR option is not.

Except for the differences documented in this section, the Windows build of
\fB@IMAGEX_PROGNAME@ capture\fR and \fB@IMAGEX_PROGNAME@ append\fR should be essentially equivalent to
the UNIX build.  However, one additional thing to note is that wimlib's Windows
ImageX is NOT written to be command-line compatible with Microsoft's ImageX,
although they are very similar.

.SH OPTIONS
.TP 6
\fB--boot\fR
Specifies that the new image is to be made the bootable image of the WIM archive.
.TP
\fB--check\fR
Include an integrity table in the new WIM file or the modified WIM file.  If
this option is not specified, no integrity table is included in \fIWIMFILE\fR,
even if there was one before in the case of \fB@IMAGEX_PROGNAME@ append\fR.
.TP
\fB--compress\fR=\fITYPE\fR
Specifies the compression type for the new WIM file.  This flag is only valid
for \fB@IMAGEX_PROGNAME@ capture\fR, since the compression mode for \fB@IMAGEX_PROGNAME@ append\fR
must be the same as that of the existing WIM.  \fITYPE\fR may be "none",
"maximum", or "fast".  By default, it is "fast".

You may also specify the actual names of the compression algorithms, "XPRESS"
and "LZX", instead of "fast" and "maximum", respectively.
.TP
\fB--threads\fR=\fINUM_THREADS\fR
Number of threads to use for compressing data.  Default: autodetect (number of
processors).  Note: if creating or appending to an uncompressed WIM, additional
threads will not be used, regardless of this parameter, since no compression
needs to be done in this case.
.TP
\fB--rebuild\fR
For \fB@IMAGEX_PROGNAME@ append\fR: rebuild the entire WIM rather than appending the new
data to the end of it.  Rebuilding the WIM is slower, but will save a little bit
of space that would otherwise be left as a hole in the WIM.  Also see \fB@IMAGEX_PROGNAME@
optimize\fR.
.TP
\fB--flags\fR=\fIEDITIONID\fR
Specify a string to use in the <FLAGS> element of the XML data for the new
image.
.TP
\fB--verbose\fR
Print the names of files and directories as they are captured, as well as a few
other informational messages.
.TP
\fB--dereference\fR
Follow symlinks; archive and dump the files they point to.  (The default is to
archive the symlinks themselves.)  This flag is only valid in the normal image
capture mode.
.TP
\fB--config\fR=\fIFILE\fR
Specifies a configuration file for capturing the new image.  The configuration
file specifies files that are to be treated specially during the image capture.

The format of the configuration file is a number of sections containing path
globs one per line, where each section begins with the tag [ExclusionList],
[ExclusionException], [CompressionExclusionList], or [AlignmentList].
Currently, only the [ExclusionList] and [ExclusionException] sections are
implemented.  The [ExclusionList] section specifies a list of path globs to
exclude from capture, while the [ExclusionException] section specifies a list of
path globs to include in the capture even if the matched file or directory name
also appears in the [ExclusionList].

Relative globs with only one path component (e.g. *.mp3) match against a filename in any
directory.  Relative globs with multiple path components (e.g. dir/file),
as well as absolute globs (e.g. /dir/file), are treated as paths starting at the
root directory of capture, or the root of the NTFS volume for NTFS capture mode.
If a directory is matched by a glob in the [ExclusionList], the entire directory
tree rooted at that directory is excluded from the capture, unless
\fB--dereference\fR is specified and there is another path into that directory
through a symbolic link.

For compatibility with Windows, the path separators in the globs may be either
forward slashes or backslashes, and the line separators may be either UNIX-style
or DOS-style.  Globs with spaces in them must be quoted, and leading and
trailing whitespace is not significant.  Empty lines and lines beginning with
'#' or whitespace followed by '#' are ignored.

Paths may not have drive letters in them, as they are all relative to the root
of capture and not absolute external paths.

If this option is not specified the following default configuration file is
used:

.RS
.RS
.PP
.nf
[ExclusionList]
\\$ntfs.log
\\hiberfil.sys
\\pagefile.sys
"\\System Volume Information"
\\RECYCLER
\\Windows\\CSC
.RE
.RE

.TP
\fB--unix-data\fR
Store the UNIX owner, group, and mode of regular files, symbolic links, and
directories.  This is done by adding a special alternate data stream to each
directory entry that contains this information.  Please note that this flag is
for convenience only, in case you want to use \fB@IMAGEX_PROGNAME@\fR to archive files on
UNIX.  Microsoft's software will not understand this special
information.
.TP
\fB--no-acls\fR
In the NTFS capture mode, do not capture security descriptors.  This flag is
also available in the native Win32 build of wimlib.
.TP
\fB--strict-acls\fR
In the Win32 native build of wimlib, fail immediately if the full security
descriptor of any file or directory cannot be read.  The default behavior
without this option is to first try omitting the SACL from the security
descriptor, then to try omitting the security descriptor entirely.  The purpose
of this is to capture as much data as possible without always requiring
Administrator privileges.  However, if you desire that all security descriptors
be captured exactly, you may with to provide this option, although the
Administrator should have permission to read everything anyway.
.TP
\fB--source-list\fR
\fB@IMAGEX_PROGNAME@ capture\fR and \fB@IMAGEX_PROGNAME@ append\fR, as of wimlib 1.3.0, support a new
option to create a WIM image from multiple files or directories.  When
\fB--source-list\fR is specified, the \fISOURCE\fR argument specifies the name
of a text file, each line of which is either 1 or 2 whitespace separated
filenames.  The first filename, the source, specifies the path to a file or
directory to capture into the WIM image.  It may be either absolute or relative
to the current working directory.  The second filename, if provided, is the
target and specifies the path  in the WIM image that this file or directory will
be saved as.  Leading and trailing slashes are ignored.  "/" indicates that
the directory is to become the root of the WIM image.  If not specified, the
target string defaults to the same as the source string.

An example is as follows:

.RS
.RS
.PP
.nf
# Make the WIM image from the 'winpe' directory
winpe   /

# Send the 'overlay' directory to '/overlay' in the WIM image
overlay /overlay

# Overlay a separate directory directly on the root of the WIM image.
# This is only legal if there are no conflicting files.
/data/stuff     /
.RE

Subdirectories in the WIM are created as needed.  Multiple source directories
may share the same target, which implies an overlay; however, an error is issued
if the same file appears in different overlays to the same directory.

Filenames containing whitespace may be quoted with either single quotes or
double quotes.  Quotes may not be escaped.

Lines consisting only of whitespace and lines beginning with '#' preceded by
optional whitespace are ignored.

As a special case, if \fISOURCE\fR is "-", the source list is read from standard
input rather than an external file.

The NTFS capture mode cannot be used with \fB--source-list\fR, as only capturing
a full NTFS volume is supported.

.SH NOTES

\fBimage append\fR does not support appending an image to a split WIM.

The different capture modes only specify the data that is captured and don't
specify a special WIM format.  A WIM file can contain images captured using
different modes.  However, all images in a WIM must have the same compression
type, and \fB@IMAGEX_PROGNAME@\fR always enforces this.

\fB@IMAGEX_PROGNAME@\fR writes WIMs having the version number 0x10d00 and a compressed
stream chunk size of 32768.  The only WIMs I've seen that are different from
this are some pre-Vista WIMs that had a different version number.

Unless \fB--rebuild\fR is specified, aborting an \fB@IMAGEX_PROGNAME@ append\fR command
mid-way through has a small chance of corrupting the WIM file.  However, a
precaution is taken against this, so it should be very unlikely.  In the event
of an aborted \fB@IMAGEX_PROGNAME@ append\fR, \fB@IMAGEX_PROGNAME@ optimize\fR may be run to remove
extra data that may have been partially appended to the physical WIM file but
not yet incorporated into the structure of the WIM.

Capturing or appending an image happens in two main phases: (1) scanning the
directory or NTFS volume to checksum all the files and determine the streams to
be written, and (2) writing the new streams to the WIM file.  Streams are not
stored in memory after (1), since there could easily be gigabytes of data;
instead, they are read again during step (2); however, duplicate streams in the
image, and streams already existing in any other image in the WIM, are not read
again.  In the future, it may be possible to introduce the ability to capture an
image with reading each file only one time, although this mode would have some
limitations--- for example, a stream might be compressed only to be thrown away
as a duplicate once it's been checksummed.

\fISOURCE\fR may be a symbolic link to a directory rather than a directory
itself.  However, additional symbolic links in subdirectories, or in additional
source directories not destined for the WIM image root (with
\fB--source-list\fR), are not dereferenced unless \fB--dereference\fR is
specified.

.SH EXAMPLES
Create a new WIM 'mywim.wim' from the directory 'somedir', using LZX compression and
including an integrity table:
.RS
.PP
@IMAGEX_PROGNAME@ capture somedir mywim.wim --compress=maximum --check
.RE
.PP
Append an image to the WIM we just captured, but do it from a NTFS volume on the
partition /dev/sda2 and name the image "Windows 7".  You do not need to specify
the compression type, because the WIM already is using LZX compression and this
cannot be changed.  You need to specify \fB--check\fR if you don't want the
integrity table to be discarded.
.RS
.PP
@IMAGEX_PROGNAME@ append /dev/sda2 mywim.wim --check "Windows 7"
.RE
.PP

.SH SEE ALSO
.BR @IMAGEX_PROGNAME@ (1)