Files with the same modification date and size in delta backups

[Skyblue]

Hello synchronicity,

Windows does not change the modification date\time and size of some files if they are changed \ updated. Those files will have the same modification time and size in the delta backups created with the options -delta-from and --update-of. Could you add an extra option to solve those conflicts? This option could force an update of those files appearing as identical in the base archive and the delta archive. The new version in the delta archive could replace the same file with the same time stamp and size in the base archive.

Some files affected by this issue :

.evtx log files found in \Windows\System32\winevt\Logs
\Users\Administrator\NTUSER.DAT
\Windows\ServiceProfiles\LocalService\AppData\Local\lastalive0.dat
\Windows\ServiceProfiles\LocalService\AppData\Local\lastalive1.dat
\Windows\System32\wfp\wfpdiag.etl

Test environment : Wimlib version 1.9.1 64-bit , OS : Server 2008 R2

[synchronicity]

Well, that's disappointing. If Windows really can do that, then the --update-of option cannot be guaranteed to work on Windows.

Is there any possibility the system was uncleanly shut down? I could imagine that could, in some situations, cause data changes to be persisted before metadata changes.

[Skyblue]

Hello synchronicity,

The main and delta backups are created with the --snapshot option and no any unclean shutdown.

[Skyblue]

Here is a quick test to reproduce the issue.

First snapshot at time 23:00 archiving the event logs :

Code: Select all

wimcapture.cmd C:\Windows\System32\winevt\Logs archive1.wim --snapshot

To add new records to the system log, stop and start the Windows update service :

Code: Select all

net stop wuauserv
net start wuauserv
net stop wuauserv

Second snapshot at 23:04 :

Code: Select all

wimcapture.cmd C:\Windows\System32\winevt\Logs archive2.wim --snapshot

After extracting the system.evtx files from the archives and renaming them as System1.evtx and System2.evtx, I examined the files in the Cygwin environment :

Code: Select all

$ stat *.evtx | grep Modify
Modify: 2016-04-08 22:46:10.086647500 +0300
Modify: 2016-04-08 22:46:10.086647500 +0300

Checking the event files with the dir command :

Code: Select all

dir *.evtx

08.04.2016  22:46            69.632 System1.evtx
08.04.2016  22:46            69.632 System2.evtx

Both of the files have the same modification time and size but the md5 signatures are different. Windows does not update the modification time of the system log file after stop and restarting the Windows update service.

Code: Select all

1d71a2a784ed97d18b616e4b1bf81d1b *System1.evtx
e063856d1855cbf1551a3fc000bcd5b0 *System2.evtx

Test environment : Wimlib version 1.9.1 , Windows 7 32-bit

Attached are the event log files.

[synchronicity]

Okay, thanks. It seems this is a rather blatant bug in Windows. The last modification time of a file is not updated each time a write is made. I just verified it myself, but it also has been reported by many other people:

https://blogs.technet.microsoft.com/asi ... losing-it/

Of course, I or anyone else outside of Microsoft cannot fix bugs in Windows; it is on Microsoft to fix this. POSIX-compliant operating systems such as Mac OS X and Linux do not have this bug. In addition, much software (backup, file sync, etc.) uses the last modification times to detect incremental changes.

I will tell you that in the --update-of mode, wimlib checks not only the last modification times (to the highest precision reported by Windows), but also the sizes of the files --- and Windows does appear to keep the file size up to date. Only when the last modification time and size of a file are *both* the same does wimlib consider the file unchanged. The event logs seem to be a somewhat unusual case because the process writing to them preallocates space and then overwrites it. This means that their sizes can remain the same over a period of time even if their data is changing.

If one desires, the bug likely can also be worked around by rebooting the computer; this should cause the last modification times to get updated due to the files being closed...

[Skyblue]

Hi synchronicity,

It looks like that MS has no any plan to fix this bug. Kindly, is it possible to add a hash compare option to wimlib? Specified as an additional option in the command-line when it's necessary, wimlib could easily detect those modified files preserving the non-updated time stamp and size. At work, it's not always possible to reboot Windows servers while users are logged in. A typical problematic case is the terminal servers.

[synchronicity]

There's no need for a "hash compare" option, since it's the default behavior. Just don't specify --update-of, and wimlib will hash every file rather than trying to optimize it by checking file timestamps and sizes.

[Skyblue]

Hi synchronicity,

Many thanks for your help. Not specifying the option --update-of while capturing the differential image ( the .dwm file ) does the job.