extract.c: Do endian conversion when checking pipable WIM header

[wimlib] / NEWS

Version 1.7.0-BETA:
        Improved compression, decompression, and extraction performance.

        Improved compatibility with version 3584 WIM / ESD files:
            - Add support for reading and writing multiple solid blocks per
              archive, which WIMGAPI/DISM can create when appending an image.
            - Correctly create solid blocks larger than 4 GiB.

        'add' commands passed to wimupdate will now replace existing
        nondirectory files by default.  Use the --no-replace option to get the
        old behavior.

        The license for the library now contains an exception that allows using
        it under the LGPL.  See the COPYING file for details.

        In reparse-point fixup mode (the default for capture), symbolic links
        and junctions that point outside the tree being captured are no longer
        excluded from capture.

        Added support for "WIMBoot" capture and extraction.  See the
        documentation for the new '--wimboot' option to wimcapture and wimapply
        for more information.

        On UNIX-like systems, you can now backup and restore device nodes, named
        pipes, and sockets.  In addition, 32-bit user and group IDs are now
        supported.

        The way that UNIX data is stored in WIM files has been changed.  If you
        captured any WIMs with the --unix-data option, to upgrade them you'll
        need to apply them with --unix-data using wimlib-imagex v1.6.2, then
        re-capture them with --unix-data using this version.

        wimlib now understands tagged metadata items, such as object IDs, that
        can be stored in WIM directory entries.

        Removed the --hardlink and --symlink options to wimapply, since I don't
        think they are too useful and they got in the way of improving the code.

        WIMs will now retain their GUIDs when rebuilt (e.g. with wimoptimize).

        The 'mkwinpeimg' script now supports writing the ISO image to standard
        output.

        The <ARCH> element in WIM XML data is now exported correctly.

        On Windows, sparse file attributes are no longer set on extracted files.
        Oddly enough, this actually saves disk space in some cases.

        On UNIX, configuring with --disable-xattr or --enable-xattr is no longer
        supported.  Mounting WIM images now always requires extended attribute
        support.  Use --without-fuse to disable support for mounting WIM images;
        this will also disable the need for extended attribute support.

        The shared library version has been bumped up.  The main
        incompatibilities are:

                - WIMLIB_COMPRESSION_TYPE_XPRESS is now 1 and
                  WIMLIB_COMPRESSION_TYPE_LZX is now 2 (so it's the same as
                  WIMGAPI).

                - User-provided progress functions are now registered using a
                  separate function, wimlib_register_progress_function().  The
                  'progress_func' argument to many functions no longer exists.

                - The return value from user-provided progress functions is now
                  significant.

                - A context argument has been added to the prototype of
                  user-provided progress functions.

                - 'struct wimlib_capture_config' has been removed.  The library
                  now takes the path to the configuration file directly.  This
                  affects wimlib_add_image(), wimlib_add_image_multisource(),
                  and wimlib_update_image().  However, a NULL value passed in
                  the argument retains the same meaning.

                - Removed deprecated functions: some (de)compression functions,
                  wimlib_extract_files(), and wimlib_print_metadata().

                - Removed extraction flags: WIMLIB_EXTRACT_FLAG_HARDLINK,
                  WIMLIB_EXTRACT_FLAG_SYMLINK, WIMLIB_EXTRACT_FLAG_FILE_ORDER,
                  and WIMLIB_EXTRACT_FLAG_SEQUENTIAL.

                - Removed some progress messages:
                  WIMLIB_PROGRESS_MSG_APPLY_TIMESTAMPS,
                  WIMLIB_PROGRESS_MSG_EXTRACT_DIR_STRUCTURE_BEGIN,
                  WIMLIB_PROGRESS_MSG_EXTRACT_DIR_STRUCTURE_END.  Numbering
                  stays the same.

                - Removed some error codes.  Numbering stays the same.

                - Replaced WIMLIB_UNMOUNT_FLAG_LAZY with
                  WIMLIB_UNMOUNT_FLAG_FORCE.

                - WIM paths passed to progress functions now have a leading
                  slash.

Version 1.6.2:
        Case-insensitive comparisons of strings (e.g. filenames) containing
        UTF-16 codepoints above 32767 are now done correctly.

        Fixed build failure on Mac OS X.

        wimunmount now provides the '--new-image' option to cause changes to a
        read-write mounted image to be committed as a new image rather than as
        an update of the mounted image.  (The corresponding new library flag is
        WIMLIB_UNMOUNT_FLAG_NEW_IMAGE.)

        The LZMS ("recovery") compression chunk size, or "dictionary size", may
        now be up to 1 GiB (1,073,741,824 bytes).

        The performance of LZX ("maximum") and LZMS ("recovery") compression
        with large chunk sizes has been slightly improved.

Version 1.6.1:
        Stored files with size exactly 4 GiB (4,294,967,296 bytes) are now
        decompressed correctly.

        Fixed a bug in the LZX compressor introduced in v1.5.3.  The bug
        occurred in an unlikely case, and due to validity checks it did not
        affect successfully created archives.

        Fixed a minor compatibility issue with the LZMS compressor and
        decompressor.  This is *not* the default compression type and was only
        introduced in v1.6.0.  In the unlikely event that you created an
        LZMS-compressed WIM with v1.6.0 and a checksum error is reported when
        applying it with v1.6.1, decompress it with v1.6.0 then compress it with
        v1.6.1.

        Memory usage for LZMS and LZX compression has been decreased.

        wimextract now allows wildcard characters in paths specified on the
        command line.  Also, the '--strict-wildcards' option has been removed
        and replaced with the inverse option '--nullglob'.  See the
        documentation for wimextract for more details and changes.

        The wimlib_extract_files() function is now considered deprecated in
        favor of wimlib_extract_paths().

        Fixed more permissions problems when extracting files on Windows.

        A new '--no-attributes' option has been added to wimapply and
        wimextract.  The library flag is WIMLIB_EXTRACT_FLAG_NO_ATTRIBUTES.

        The default chunk size is now set correctly when changing the
        compression type of a WIM, for example with 'wimoptimize'.

        The '--metadata' option to wiminfo has been replaced with the
        '--detailed' option to wimdir.

        In relevant wimlib-imagex commands, '--solid' may now be used as an
        alias for '--pack-streams'.

Version 1.6.0:
        Support for extracting and updating the new version 3584 WIMs has been
        added.  These WIMs typically pack many streams ("files") together into a
        single compressed resource, thereby saving space.  This degrades the
        performance of random access (such as that which occurs on a mounted
        image), but optimizations have been implemented for extraction.  These
        new WIM files also typically use a new compression format (LZMS), which
        is similar to LZMA and can offer a better compression ratio than LZX.
        These new WIM files can be created using `wimcapture' with
        the '--compress=lzms --pack-streams' options.  Note: this new WIM format
        is used by the Windows 8 web downloader, but important segments of the
        raw '.esd' files are encrypted, so wimlib will not be able to extract
        such files until they are first decrypted.

        wimlib now supports extracting files and directories from a WIM image
        based on a "listfile" that itself contains the list of paths to extract.
        For `wimextract', the syntax is to specify @LISTFILE instead of a PATH,
        and for the library itself, the new APIs are wimlib_extract_pathlist()
        and wimlib_extract_paths().  Path globs containing wildcard characters
        are supported.

        For searching WIM files, wimlib now has configurable case sensitivity.
        The default on Windows is still case-insensitive and the default on
        UNIX-like systems is still case-sensitive, but this can be overridden
        on either platform through flags to wimlib_global_init().  For
        `wimlib-imagex', the environmental variable WIMLIB_IMAGEX_IGNORE_CASE
        can be set to 1 or 0 for case-insensitive or case-sensitive behavior,
        respectively.

        Support for compression chunk sizes greater than the default of 32768
        bytes has been added.  A larger chunk size typically results in a better
        compression ratio.  However, the MS implementation is seemingly not
        compatible with all chunk sizes, especially for LZX compression, so the
        defaults remain unchanged, with the exception of the new LZMS-compressed
        WIMs, which use a larger chunk size by default.

        The compression/decompression API exported by wimlib has been changed.
        Now one set of functions handles all supported compression formats.

        `wimcapture' and `wimappend' will now display the progress of scanning
        the directory tree to capture, in addition to the progress of writing
        data to the WIM.  The '--verbose' option no longer does anything.  The
        library API change for this is the addition of several members to
        `struct wimlib_progress_info_scan' available to progress callbacks.

        `mkwinpeimg' now correctly handles the '--start-script' option when the
        start script is not in the working directory.

        Sequential extraction, previously requested by using
        WIMLIB_EXTRACT_FLAG_SEQUENTIAL, is now the default.
        WIMLIB_EXTRACT_FLAG_FILE_ORDER can be used to get the old default
        behavior (extract in file order).

Version 1.5.3:
        The new LZX compressor added in v1.5.2 has been improved and is now
        enabled by default, except when `wimcapture' or `wimappend' is run
        *without* the '--compress' option, in which case the faster LZX
        compressor is used (the same as before).  This behavior is reasonably
        consistent with ImageX which actually uses "fast" (XPRESS) compression
        by default.  In those cases, use '--compress=maximum' to explicitly
        capture a WIM image using the new (slower but better) LZX compressor.

        The '--compress-slow' option still exists to `wimlib-imagex optimize',
        but its new behavior is to tweak the new LZX compressor even more to
        produce an even better compression ratio at the cost of more time spent
        compressing.

        `wimlib-imagex optimize' now supports the '--compress=TYPE' option,
        which recompresses the WIM file using the specified compression TYPE.
        The new library API function used for this is
        wimlib_set_output_compression_type().

        Added the wimlib_get_xml_data() function to allow library clients to
        easily retrieve the raw XML data from a WIM file if needed.

        Fixed a bug that could cause an error code to be incorrectly returned
        when writing XML data containing a <WINDOWS> element.

        Mounted WIM images will now correctly show the default file stream even
        if appears in the alternate data stream entries of the corresponding WIM
        directory entry.

Version 1.5.2:
        Added a new experimental LZX compressor which can be enabled by passing
        '--compress-slow' to `wimlib-imagex capture' or `wimlib-imagex
        optimize'.  (The latter is only applicable if the WIM is already
        LZX-compressed and the '--recompress' option is also given.)  The
        experimental compressor is much slower but compresses the data slightly
        more --- currently usually to within a fraction of a percent of the
        results from WIMGAPI/ImageX.

        A workaround has been added for compatibility with versions of WinPE
        that interpret alternate data stream entries in the boot WIM
        incorrectly.

        An alignment bug that caused a crash in the LZX decompressor on some
        builds was fixed.

        wimlib now attempts to clear the WIM_HDR_FLAG_WRITE_IN_PROGRESS flag in
        the WIM header when restoring the previous state of a WIM it failed to
        successfully update.

        Added a workaround to avoid an access denied error on Windows when
        replacing a WIM file that another process has opened.

Version 1.5.1:
        wimlib can now open WinPE WIMs from WAIK v2.1, which had a quirk that
        needed to be handled.

        A bug in the interpretation of negative IMAGE indices in the
        --update-of=[WIMFILE:]IMAGE option to `wimlib-imagex capture' and
        `wimlib-imagex append' has been fixed.

        A workaround has been added to successfully apply security descriptors
        with empty DACLs when the NTFS-3g apply mode is being used with NTFS-3g
        2013.1.13 or earlier.

        `wimlib-imagex capture' can now accept the '--delta-from' option
        multiple times.

Version 1.5.0:
        Added support for "pipable" WIMs.  Pipable WIMs allow capturing images
        to standard output and applying images from standard input, but they are
        not compatible with Microsoft's software and are not created by default.
        See the documentation for --pipable flag of `wimlib-imagex capture' for
        more information.

        To better support incremental backups, added support for declaring an
        image as a modified form of another image.  See the documentation for
        the '--update-of' option of `wimlib-imagex append' and `wimlib-imagex
        capture'.

        Added supported for "delta" WIMs.  See the documentation for the
        '--delta-from' option of `wimlib-imagex capture'.

        The library support for managing split WIMs has been changed to support
        other arrangements, such as delta WIMs, and be easier to use.  This
        change is visible in `wimlib-imagex', which also can now accept the
        '--ref' option multiple times, and also now supports "delta" WIMs as
        mentioned above.

        wimlib now preserves WIM integrity tables by default, even if
        WIMLIB_WRITE_FLAG_CHECK_INTEGRITY is not specified.  This changes the
        behavior of `wimlib-imagex' whenever the WIM being operated on contains
        an integrity table and the '--check' option is not specified.

        `wimlib-imagex capture' now creates LZX-compressed WIMs by default
        (when --compress is not specified).  This provides the best compression
        ratio by default, which is usually what is desired, at a cost of some
        speed.

        `wimlib-imagex' now supports being invoked as wimCOMMAND, where COMMAND
        is the command as in `wimlib-imagex COMMAND'; for example, it can be
        invoked as `wimapply' as an alternative to `wimlib-imagex apply'.  The
        appropriate hard links are created in UNIX installations of
        `wimlib-imagex', while for the Windows distribution of `wimlib-imagex',
        batch files that emulate this behavior are generated.

        Security descriptors are now extracted correctly on Windows.

        Fixed archiving DOS names in NTFS-3g capture mode.

        The extraction code has been rewritten and it will now be easier to
        support new features on all supported backends (currently Win32, UNIX,
        and NTFS-3g).  For example, hard-linked extraction mode (--hardlink) is
        now supported on all backends, not just UNIX.

        `mkwinpeimg' now supports grabbing files from the WAIK supplement rather
        than the WAIK itself.

        wimlib_global_init() now, by default, attempts to acquire additional
        privileges on Windows, so library clients need not do this.

        This update bumps the shared library version number up to 9, since it is
        not binary compatibible with previous releases.

Version 1.4.2:
        Fixed bug in `wimlib-imagex export' that made it impossible to export an
        image from a WIM that is readonly at the filesystem level.

        Return error code rather than segfaulting when trying to list files from
        a non-first part of a split WIM.

        Joining a WIM will now preserve the RP_FIX and READONLY flags.

Version 1.4.1:
        On Windows, paths given to wimlib-imagex are now treated case
        insensitively.

        Improved behavior regarding invalid filenames; in particular, on
        Windows, wimlib-imagex will, when extracting, now omit (with an option
        to override this default) filenames differing only in case, or filenames
        containing characters not valid on Windows.

        On Windows, wimlib now supports capturing and extracting long paths
        (longer than the so-called MAX_PATH).

        On Windows, `wimlib-imagex update' now acquires proper privileges when
        running as an Administrator.

        `wimlib-imagex update' will now complain if no image is specified when
        trying to update a multi-image WIM.

        `wimlib-imagex update' now supports specifying a single update command
        directly on the command line using the --command option.

        wimlib-imagex will now choose different units for progress messages,
        depending on the amount of data that needs to be processed.

        `wimlib-imagex append' will now generate a unique WIM image name if no
        name is specified and the defaulted name already exists in the WIM.

        wimlib now allows you to create unnamed WIM images, which can then only
        be referred to by index.

        wimlib now allows you to explicitly declare you want write access to a
        WIM by providing the WIMLIB_OPEN_FLAG_WRITE_ACCESS flag to
        wimlib_open_wim().

        wimlib now respects the WIM_HDR_FLAG_READONLY flag when set in the WIM
        header.

        Progress callbacks have been added to wimlib's wimlib_update_image()
        function.

        Added wimlib_get_wim_info(), wimlib_set_wim_info(),
        wimlib_iterate_dir_tree(), and wimlib_iterate_lookup_table() functions
        to the library.

        NTFS-3g capture now only warns about two conditions previously treated
        as errors.

        Fixed a couple issues with using wimlib-imagex on UDF filesystems on
        Windows.

        wimlib now correctly detects and returns an error when reading a WIM
        image with a cyclic directory structure.  (Fun fact: such a WIM will
        crash Microsoft's software.)

Version 1.4.0:
        Added new "extract" and "update" subcommands to wimlib-imagex, along
        with associated APIs in the library.  These commands are intended mainly
        for Windows use but can be used on UNIX as well.

        Many documentation improvements.

        Fixed a bug in the Windows build where relative symbolic links were not
        captured in reparse-point fixup mode.

        Fixed a bug in the Windows build where file handles were left open to
        the WIM file, causing `wimlib_imagex optimize' to fail in some cases.

        Fixed a bug in the Windows build of wimlib-imagex where globbing
        split-WIM parts could cause the program to crash.

        Fixed a bug where the creation time of WIM images would be shown instead
        of the last modification time.

        With the Windows build it is now possible to restore a WIM containing
        symbolic links as a non-Administrator; however you will receive warnings
        about not being able to extract the symbolic links.

Version 1.3.3:
        Capturing a WIM image should now be significantly faster in most cases
        due to improved use of the operating system's cache and avoiding reading
        files twice whenever possible.

        The Windows build should now work on Windows XP.

        The Windows build now supports capturing and restoring hidden,
        compressed, sparse, and encrypted files.

        The Windows build now supports capturing and applying WIM images from
        filesystems other than NTFS (with some reduced functionality).

        The Windows build now extracts short names correctly.

        Added support for "reparse-point" fixups (i.e. fixing up of symbolic
        links).  See docs for --rpfix and --norpfix flags of `wimlib-imagex
        capture' and `wimlib-imagex apply'.

        The performance of splitting and joining WIMs should be slightly
        improved.

        The LZX and XPRESS compression and decompression functions are now
        exported from the library.

Version 1.3.2:
        Improvements and bugfixes for the Windows build.

        Added --strict-acls options.

        Fixed the way that wimlib determines the order of images in the WIM.

Version 1.3.1:
        Since wimlib can now be used on Windows, wimlib's implementation of
        ImageX has been renamed to wimlib-imagex to avoid confusion with
        Microsoft's implementation of ImageX, which would have the same file
        name ("imagex.exe").  If you really don't like this you can pass the
        --with-imagex-progname option to `configure' to choose a different name,
        or even simply rename the binary yourself (but the former way will
        configure the man pages to use the chosen name).

        Various bugs fixed in the Windows build.  Mainly to do with capturing
        and restoring alternate data streams correctly in weird cases, and
        requesting the correct privileges when opening files.  Also added the
        --noacls options to wimlib-imagex capture, append, and apply.

        Windows build again: FindFirstStreamW() and FindNextStreamW() are now
        dynamically loaded, so this may make the library compatible with Windows
        XP (however, there may still be other problems).

Version 1.3.0:
        Added experimental support for native Windows builds.  Binaries can be
        downloaded from the SourceForge page.

        --source-list option added to `imagex capture' and `imagex append'.

        Better support for different character encodings.

Version 1.2.6:
        Storing UNIX file owners, groups, and modes in WIM images is now
        possible using `imagex capture' with the --unix-data flag.

        Minor bug fixes and documentation fixes.

Version 1.2.5:
        NTFS capture: Fixed capturing duplicate reparse points.

        NTFS capture: Capture first unnamed stream if there are more than one
        (print warning instead of error).

        Allow multiple test cases to execute concurrently (e.g. make -j2 check).

Version 1.2.4:
        Added --arch switch to mkwinpeimg script to support getting AMD64 WinPE
        from the WAIK.

        Update to work with ntfs-3g version 2013.1.13.

Version 1.2.3:
        Fixed truncating file to shorter but non-zero length on read-write
        mounted WIM image.

        Various code cleanups and minor documentation fixes.

Version 1.2.2:
        LZX and XPRESS decompression have received some additional optimizations
        and should now be even faster.  (Although, they were already pretty
        fast--- much faster than typical I/O speeds.)

        Fixed a bug introduced in v1.2.1 that would cause a directory tree
        containing hard links to be captured incorrectly in some cases.

Version 1.2.1:
        By default, unmounting a read-write mounted WIM with 'imagex unmount
        --commit' will now change the WIM in-place without needing to write the
        entire WIM again.  Use 'imagex unmount --commit --rebuild' to get the
        old behavior.

        'imagex unmount' no longer has a hard-coded limit of 10 minutes to wait
        for a response from the daemon servicing the mounted WIM.  Instead,
        every second 'imagex unmount' will check if the daemon is still alive,
        and keep waiting if so, otherwise terminate with an error.

        'imagex unmount --commit' on a read-write mounted WIM will now print
        progress information regarding the writing of new or modified streams
        the WIM, just like when capturing or appending a WIM.

        A small change has been made to XPRESS compression and it should improve
        the compression ratio slightly.

        A change was made that may improve performance slightly when applying a
        WIM image to a NTFS volume.

        Microsoft has managed to introduce even more bugs into their software,
        and now the WIMs for Windows 8 have incorrect (too low) reference counts
        for some streams.  This is unsafe because such streams can be removed
        when they are in actuality still referenced in the WIM (perhaps by a
        different image).  wimlib will now work around this problem by fixing
        the stream reference counts.  This is only done when wimlib_delete_image() is
        called ('imagex delete') or when wimlib_mount_image() is called with
        WIMLIB_MOUNT_FLAG_READWRITE ('imagex mountrw').  Please note that this
        requires reading the metadata for all images in the WIM, so this will
        make these operations noticably slower on WIMs with multiple images.

        Various other bugfixes.

Version 1.2.0:
        Appending images to a WIM is now be done by default without re-building
        the whole WIM.  Use the --rebuild flag to get the old behavior (which
        was to re-build the entire WIM when a new image is appended).

        A new command `imagex optimize' is now available to manually re-build a
        WIM that has wasted space due to repeated appends.

        Progress information has been improved, and now arbitrary callback
        functions can be used to show the progress of a WIM operation.

        A possible bug with changing the bootable image of a WIM was fixed.

        Some advisory locking is now done to prevent two processes from
        modifying a WIM at the same time (but only in some cases).  For example,
        you cannot mount two images from a WIM read-write at the same time.

        Some functions have been reorganized:
                * wimlib_mount() renamed to wimlib_mount_image().
                * wimlib_unmount() renamed to wimlib_unmount_image().
                * wimlib_overwrite_xml_and_header() removed as
                wimlib_overwrite() suffices now.
                * wimlib_apply_image_to_ntfs_volume() removed as
                wimlib_extract_image() suffices now.
                * wimlib_add_image_from_ntfs_volume() removed as
                * wimlib_add_image() suffices now.

        Previously, the soname of libwim.so has been 0.0.0, despite many
        interface changes.  The soname is now updated to 1.0.0 and will now be
        updated each release.

Version 1.1.0:
        Resources will now be compressed using multiple threads by default.
        (This applies to `imagex capture', `imagex append', and `imagex
        export').

        Some performance improvements in mounted WIMs.

        More progress information is shown when capturing a WIM.

Version 1.0.4:
        Lots of minor fixes, code cleanups, and some documentation updates.
        Nothing in particular is really noteworthy.

Version 1.0.3:
        LZX and XPRESS compression improvements.

        Fixed calculation of Directory Count, File Count, Total Bytes, and Hard
        Link Bytes of the WIM.

Version 1.0.2:
        Fixed bug when capturing NTFS file with multiple named data streams.

        Internally, we are now using inode structures, even though these don't
        appear literally in the WIM file.  This simplifies some of the code
        (mainly for WIM mounting) and likely fixed a few problems, although it
        needs more testing.

Version 1.0.1:
        Fixed problem when exporting images from XPRESS to LZX compressed WIM or
        vice versa

Version 1.0.0:
        Enough changes to call it version 1.0.0!

        Capturing a WIM directly from a NTFS volume, and applying a WIM directly
        to a NTFS volume, is now supported.

        Hard links and symbolic links have much improved support.  They are
        supported for WIM capture, WIM application, and mounted WIMs (you can
        even make them on read-write mounted WIMs).

        Alternate data streams are now supported on mounted WIMs through an
        xattr or a Windows-style stream interface.  Also they are supported when
        capturing a WIM from NTFS or applying a WIM to NTFS.

        Split WIMs are better supported.  You may now apply an image directly
        from a split WIM, mount an image from a split WIM read-only, or export
        an image from a split WIM.

        Using a capture configuration file is now supported (but not fully yet).

        SHA1 message digests are checked in more places, so we can make sure
        applied and captured data is correct.

        Man pages have been updated and consolidated.

Version 0.7.2:
        Fixed segfault when unmounting read-only WIM.

Version 0.7.1:
        Support for joining and splitting WIMs.
        Also, security data is now preserved by default.

Version 0.6.3:
        Can now build with older gcc and system headers, like on CentOS 5.

Version 0.6.2:
        Fixed bug that made it impossible to overwrite files in read-write
        mount.

Version 0.6.1:
        Write byte-order mark before WIM XML data.  (imagex.exe requires this to
        be there.)