Suggestion: merge capture and append into functionally identical commands

Comments, questions, bug reports, etc.
chungy
Posts: 30
Joined: Mon Feb 15, 2016 3:40 am

Suggestion: merge capture and append into functionally identical commands

Post by chungy »

capture and append are already very similar, but you must use capture to create a WIM first, and must use append to add images, instead of overwriting the existing file. The latter trips me up at times (thankfully I've not been without backups/snapshots to restore the old WIM file or data), where I might use wimcapture a second time with the intent of _adding_ to a WIM file, but end up overwriting it completely... oops. The distinction has become troublesome in scripting, because I have to make sure to special-case the first image, and then do subsequent images with append. For example, instead of doing "for vol in /dev/zd{0,16,32,64}p2; do wimcapture $vol windows.wim; done", I'd need to do /dev/zd0p2 separately.

Basically, I think things might run smoother with expectations if the two commands performed the identical task -- the separate command names might be kept for backwards compatibility with scripts, but the behavior I would like means "capture" would append images, and "append" would be capable of creating new images too. If old behavior is desired, I wouldn't be bothered by just being required to delete the old WIM first.
synchronicity
Site Admin
Posts: 472
Joined: Sun Aug 02, 2015 10:31 pm

Re: Suggestion: merge capture and append into functionally identical commands

Post by synchronicity »

It may have been a good idea to do this originally. There would still need to be an option to always create a new archive, rather that potentially use an existing archive; it just wouldn't be the default behavior. Changing it now is worth considering, but I am concerned that the change would cause confusion and break some existing scripts. It may also be helpful for some users that the current behavior is consistent with Microsoft's ImageX and DISM.

An alternative would be to add an option for the requested behavior, like 'wimappend --create-if-needed'.
chungy
Posts: 30
Joined: Mon Feb 15, 2016 3:40 am

Re: Suggestion: merge capture and append into functionally identical commands

Post by chungy »

I think changing it in a major version (eg, 1.10) is fair enough. Reading release notes for new software versions is always a good idea, after all.

The more conservative approach would be appreciated too, but I don't think we should need a new parameter to wimappend in order to create a new WIM, it could just do it regardless (is there a good reason to fail if a WIM doesn't currently exist?). Likewise, if the commands are kept separate, I think adding "--force" to wimcapture would be a good idea. The current behavior of blowing away the existing WIM archive is very potentially devestating, real data loss might happen by an accidental invoke of the command. At least --force would say more explicitly "It's your own fault" if it still happened.
bliblubli
Posts: 88
Joined: Thu Dec 31, 2015 10:45 am

Re: Suggestion: merge capture and append into functionally identical commands

Post by bliblubli »

Good Idea, in the meantime, asking the user if he want's to overwrite the existing file (for capture) or if he wants to create the archive (in case of append) would be safe. I agree that automatically creating the archive if it doesn't exist for append should be safe. For capture I would also keep it (not completely merge both), a safeguard like --force sounds good for script. For interactive command line usage, I would just ask the user what to do like in every other software (break (b)/overwrite(o) or rename(r) with .001 .002, etc...)
synchronicity
Site Admin
Posts: 472
Joined: Sun Aug 02, 2015 10:31 pm

Re: Suggestion: merge capture and append into functionally identical commands

Post by synchronicity »

Well, 1.10 is actually a minor version, and most people do not read release notes. Granted, I've had a habit of making potentially breaking changes in minor versions anyway, but they usually have been minor changes I felt were unlikely to break anyone in practice.

Not all file archivers ask the user before overwriting files. For example, tar does not, and tar is probably the most commonly used file archiver on UNIX-like systems. For some people, not getting spammed with annoying "are you sure?" prompts is a feature. Microsoft ImageX and DISM do not use prompts either.

I am not necessarily against changing things; I just want to carefully consider the advantages and disadvantages.
bliblubli
Posts: 88
Joined: Thu Dec 31, 2015 10:45 am

Re: Suggestion: merge capture and append into functionally identical commands

Post by bliblubli »

synchronicity wrote: For some people, not getting spammed with annoying "are you sure?" prompts is a feature. Microsoft ImageX and DISM do not use prompts either.
Is understandable. Then capture could be the way to always create a new archive and append would only create one if their is no archive with given name.
synchronicity
Site Admin
Posts: 472
Joined: Sun Aug 02, 2015 10:31 pm

Re: Suggestion: merge capture and append into functionally identical commands

Post by synchronicity »

I am liking this version of the changes the most:
  • * wimappend creates a new WIM archive if one doesn't yet exist, thereby acting like wimcapture in that scenario
    * wimcapture requires confirmation before overwriting an existing archive
    • * --force option causes it to always overwrite
      * If used interactively and --force is not specified, a prompt will be given, asking if the file should be overwritten. The prompt can mention that wimappend may have been intended and that the --force option is available.
The wimcapture change may break some scripts, though they will be easily fixed by adding --force. If someone needs to be compatible with multiple versions of wimcapture, they can delete the file themselves before invoking wimcapture. The change would not affect library-only users.

Confirmation of overwriting files being extracted is a separate issue and would need to be considered separately. It may have some edge cases and be difficult to implement. I also do not, in general, like the idea of offering to "rename" files, since that is rarely the right thing to do.
chungy
Posts: 30
Joined: Mon Feb 15, 2016 3:40 am

Re: Suggestion: merge capture and append into functionally identical commands

Post by chungy »

I don't believe a prompt is a good idea. None of the other wimlib-imagex commands use one, and I do agree that they can be annoying, in addition to harming the possible use of scripts. Exiting with an error code and a stderr message about not overwriting the WIM file should be sufficient (now, standard shell scripts on Unix don't tend to halt on errors by default, but that's just a failure mode the author of a script should deal with...).

The comparison with tar is interesting, and a good example to draw from, even if pop culture sometimes thinks it's confusing. "tar -c" always creates a new archive, by default to stdout, and will always overwrite archives if you use -f. Effectively, "tar -cf wimlib.tar wimlib" is identical to "tar -c wimlib > wimlib.tar". Still, options that merely update an archive, such as "tar -r" or "tar -u" will create a new archive if none already exists.

Drawing example from other archivers is still tricky. They don't behave consistently anyway. Maybe Zip would be another example to draw from, particularly as it is more common than tar in Windows. When you do "zip wimlib.zip wimlib", it would update the entries in the wimlib.zip file with the wimlib file (and merely add a directory entry without doing a recursive add, by default). The behavior closest to wimcapture with Zip, is doing "zip -r --filesync wimlib.zip wimlib", which would always make the resulting wimlib.zip identical to a whole new archive in general. With our current line of thinking, this would be like doing "wimcapture --force wimlib.wim wimlib", requiring an option to confirm the total overwrite of the old (if existing) file.

WIM presents some special challenges to making direct comparisons. Each image in a WIM is effectively an independent archive (or tree), while most other formats hold just a single tree. Even though the images can be updated, there is also a bit of an expectation of remaining fixed. New versions of trees are captured as new images in the WIM, and if you want to delete the old version, you just use delete. The best direct equivalent to "tar -u" or "zip" is "wimupdate --command='add wimlib /' wimlib.wim", awkward in a way that makes treating images as semi-immutable easier than otherwise. The only other archiver that behaves like WIM that I can think of, is ZPAQ, in which the equivalent to WIM images is called "versions" instead, but it's still really wholly independent trees, deduplicated across all other trees. There are no separate "zpaq capture" and "zpaq append" commands, just "zpaq add" for both; create an archive if it doesn't exist, or add a new tree to an existing one. (I wouldn't draw too much inspiration from zpaq though, the command line is awkward in a lot of worse ways than tar, and deleting old trees is intentionally made difficult...)

I'm sorry that I might have rambled on a little. None of this runs contrary to your present line of thought, I actually quite like it. Not as much as merely both commands being identical, but removing the possibility of accidental data loss is good nonetheless. Also good is being able to script adding a whole bunch of trees/devices to independent images (in the same WIM) without needing to special-case the first one just by using wimappend.
synchronicity
Site Admin
Posts: 472
Joined: Sun Aug 02, 2015 10:31 pm

Re: Suggestion: merge capture and append into functionally identical commands

Post by synchronicity »

Yes, different archivers do things differently, and different people have different expectations... so it is hard to decide on a behavior.

To be on the safe side, I might not make these changes before the next release. But I will keep them in mind for later.

In the mean time, for completeness I should mention that wimlib has long supported WIM files containing no images. If you create such a WIM file, then it will always be valid to use 'wimappend' on it. To create such a file on the command line you need to capture an empty directory with 'wimcapture', then delete the image with 'wimdelete'. It's an ugly solution, of course.
synchronicity
Site Admin
Posts: 472
Joined: Sun Aug 02, 2015 10:31 pm

Re: Suggestion: merge capture and append into functionally identical commands

Post by synchronicity »

I decided to just add a --create option to wimappend which makes it create the WIM file if it doesn't already exist. So, 'wimappend --create' has the semantics of the suggested merged command. Existing behavior without --create is unchanged.

You can find this change in wimlib-1.13.0-BETA5 and later.
Post Reply