wimlib

Yes, by definition the contents of two files can only be duplicates if they have the same size. Just before starting to write to the output file, wimlib quickly marks each file contents as "definitely unique" (unique size) or "potentially duplicate" (not unique size). For definit...

wimlib tries to compress all files, regardless of file extension or file format, since it cannot predict, in general, whether the data will be compressible or not. Note that if the data is incompressible, then the compression speed will be much faster than if the data is moderately compressible. In ...

1.) The first thing probably appears to take little CPU time because that particular workload happens to be I/O bound. I don't understand what you mean by "wim archives are being recompressed". 3.) The "Archiving file data" progress is reported in terms of bytes, not files. So it...

Do you mean the XML document inside the WIM file? That might make sense, although it gets a little complicated if you try to work through the details of how a referenced WIM file would be identified and then found again automatically later (by path? by name? by GUID?). Then there are the questions o...

I've posted wimlib 1.9.2-BETA1. It should handle the field correctly.

Thanks for the tip! Yes,it looks like your suggested interpretation is correct. I think I did not check what the file looks like after unregistering a WIM from the volume, since that is the case where the two fields diverge. next_data_source_id becomes greater than the number of entries, since data ...

Was this on a system with a lot of memory? It's possible the operating system was just catching up with previously buffered writes. On modern operating systems, data written by applications is usually stored in memory and flushed to disk asynchronously. Applications do not have much, if any, visibil...

Okay, thanks. It seems this is a rather blatant bug in Windows. The last modification time of a file is not updated each time a write is made. I just verified it myself, but it also has been reported by many other people: https://blogs.technet.microsoft.com/asiasupp/2010/12/14/file-date-modified-pro...

I really have no idea. I don't believe wimlib isn't doing anything unusual with the output WIM file. It opens it for writing and writes to it mostly sequentially. Sometimes it seeks slightly backwards and overwrites bytes it recently wrote. I don't think it's an unusual access pattern at all, so if ...

Well, that's disappointing. If Windows really can do that, then the --update-of option cannot be guaranteed to work on Windows.

Is there any possibility the system was uncleanly shut down? I could imagine that could, in some situations, cause data changes to be persisted before metadata changes.