Hello synchronicity,
Windows does not change the modification date\time and size of some files if they are changed \ updated. Those files will have the same modification time and size in the delta backups created with the options -delta-from and --update-of. Could you add an extra option to solve those conflicts? This option could force an update of those files appearing as identical in the base archive and the delta archive. The new version in the delta archive could replace the same file with the same time stamp and size in the base archive.
Some files affected by this issue :
.evtx log files found in \Windows\System32\winevt\Logs
\Users\Administrator\NTUSER.DAT
\Windows\ServiceProfiles\LocalService\AppData\Local\lastalive0.dat
\Windows\ServiceProfiles\LocalService\AppData\Local\lastalive1.dat
\Windows\System32\wfp\wfpdiag.etl
Test environment : Wimlib version 1.9.1 64-bit , OS : Server 2008 R2
Files with the same modification date and size in delta backups
-
- Site Admin
- Posts: 474
- Joined: Sun Aug 02, 2015 10:31 pm
Re: Files with the same modification date and size in delta backups
Well, that's disappointing. If Windows really can do that, then the --update-of option cannot be guaranteed to work on Windows.
Is there any possibility the system was uncleanly shut down? I could imagine that could, in some situations, cause data changes to be persisted before metadata changes.
Is there any possibility the system was uncleanly shut down? I could imagine that could, in some situations, cause data changes to be persisted before metadata changes.
Re: Files with the same modification date and size in delta backups
Hello synchronicity,
The main and delta backups are created with the --snapshot option and no any unclean shutdown.
The main and delta backups are created with the --snapshot option and no any unclean shutdown.
Re: Files with the same modification date and size in delta backups
Here is a quick test to reproduce the issue.
First snapshot at time 23:00 archiving the event logs :
To add new records to the system log, stop and start the Windows update service :
Second snapshot at 23:04 :
After extracting the system.evtx files from the archives and renaming them as System1.evtx and System2.evtx, I examined the files in the Cygwin environment :
Checking the event files with the dir command :
Both of the files have the same modification time and size but the md5 signatures are different. Windows does not update the modification time of the system log file after stop and restarting the Windows update service.
Test environment : Wimlib version 1.9.1 , Windows 7 32-bit
Attached are the event log files.
First snapshot at time 23:00 archiving the event logs :
Code: Select all
wimcapture.cmd C:\Windows\System32\winevt\Logs archive1.wim --snapshot
Code: Select all
net stop wuauserv
net start wuauserv
net stop wuauserv
Code: Select all
wimcapture.cmd C:\Windows\System32\winevt\Logs archive2.wim --snapshot
Code: Select all
$ stat *.evtx | grep Modify
Modify: 2016-04-08 22:46:10.086647500 +0300
Modify: 2016-04-08 22:46:10.086647500 +0300
Code: Select all
dir *.evtx
08.04.2016 22:46 69.632 System1.evtx
08.04.2016 22:46 69.632 System2.evtx
Code: Select all
1d71a2a784ed97d18b616e4b1bf81d1b *System1.evtx
e063856d1855cbf1551a3fc000bcd5b0 *System2.evtx
Attached are the event log files.
- Attachments
-
- system-event-logs.zip
- (23.06 KiB) Downloaded 576 times
-
- Site Admin
- Posts: 474
- Joined: Sun Aug 02, 2015 10:31 pm
Re: Files with the same modification date and size in delta backups
Okay, thanks. It seems this is a rather blatant bug in Windows. The last modification time of a file is not updated each time a write is made. I just verified it myself, but it also has been reported by many other people:
https://blogs.technet.microsoft.com/asi ... losing-it/
Of course, I or anyone else outside of Microsoft cannot fix bugs in Windows; it is on Microsoft to fix this. POSIX-compliant operating systems such as Mac OS X and Linux do not have this bug. In addition, much software (backup, file sync, etc.) uses the last modification times to detect incremental changes.
I will tell you that in the --update-of mode, wimlib checks not only the last modification times (to the highest precision reported by Windows), but also the sizes of the files --- and Windows does appear to keep the file size up to date. Only when the last modification time and size of a file are *both* the same does wimlib consider the file unchanged. The event logs seem to be a somewhat unusual case because the process writing to them preallocates space and then overwrites it. This means that their sizes can remain the same over a period of time even if their data is changing.
If one desires, the bug likely can also be worked around by rebooting the computer; this should cause the last modification times to get updated due to the files being closed...
https://blogs.technet.microsoft.com/asi ... losing-it/
Of course, I or anyone else outside of Microsoft cannot fix bugs in Windows; it is on Microsoft to fix this. POSIX-compliant operating systems such as Mac OS X and Linux do not have this bug. In addition, much software (backup, file sync, etc.) uses the last modification times to detect incremental changes.
I will tell you that in the --update-of mode, wimlib checks not only the last modification times (to the highest precision reported by Windows), but also the sizes of the files --- and Windows does appear to keep the file size up to date. Only when the last modification time and size of a file are *both* the same does wimlib consider the file unchanged. The event logs seem to be a somewhat unusual case because the process writing to them preallocates space and then overwrites it. This means that their sizes can remain the same over a period of time even if their data is changing.
If one desires, the bug likely can also be worked around by rebooting the computer; this should cause the last modification times to get updated due to the files being closed...
Re: Files with the same modification date and size in delta backups
Hi synchronicity,
It looks like that MS has no any plan to fix this bug. Kindly, is it possible to add a hash compare option to wimlib? Specified as an additional option in the command-line when it's necessary, wimlib could easily detect those modified files preserving the non-updated time stamp and size. At work, it's not always possible to reboot Windows servers while users are logged in. A typical problematic case is the terminal servers.
It looks like that MS has no any plan to fix this bug. Kindly, is it possible to add a hash compare option to wimlib? Specified as an additional option in the command-line when it's necessary, wimlib could easily detect those modified files preserving the non-updated time stamp and size. At work, it's not always possible to reboot Windows servers while users are logged in. A typical problematic case is the terminal servers.
-
- Site Admin
- Posts: 474
- Joined: Sun Aug 02, 2015 10:31 pm
Re: Files with the same modification date and size in delta backups
There's no need for a "hash compare" option, since it's the default behavior. Just don't specify --update-of, and wimlib will hash every file rather than trying to optimize it by checking file timestamps and sizes.
Re: Files with the same modification date and size in delta backups
Hi synchronicity,
Many thanks for your help. Not specifying the option --update-of while capturing the differential image ( the .dwm file ) does the job.
Many thanks for your help. Not specifying the option --update-of while capturing the differential image ( the .dwm file ) does the job.