Difference found on: Delta-from vs. Delta-from + Update-of

[Alacran]

Hi Synchronicity

I have been doing my OSs backup making first capture as OS-backup.wim and latter using update-of=OS-backup.wim:-1 and appending them as a new index on OS-backup.wim, and never had a single problem when deploying an index to HD (after formating it), this approach let me delete any index and rebuild the wim and continue adding new index, all has worked fantastic always.

Now I decided to make a change on my backup estategy, then using as refernce OS-first-backup.wim (single index) of Win10x64 partition made 6 moths ago, and now capturing a delta-from on a separate dwm, this worked very fine, wimlib started archiving:

Archiving file data: 0 GiB of 15 GiB (0%), and finally ended with:
Archiving file data: 851 MiB of 851 MiB (100%)

Time = 284 sec. = 4 min. 44 sec.

This is basically same as it used to do when making update-of (starting archiving as Archiving file data: 0 GiB of 15 GiB (0%) done), so far all is fine.

Then I wanted to try another faster way suggested on documentation delta-from + update-of together, and I found this:

Archiving file data: 0 MiB of 788 MiB (0%), and finally ended with
Archiving file data: 747 MiB of 747 MiB (100%)

Time = 90 sec. = 1.5 min. No doubt it is faster and it should be my prefered approach.

But there is a big diference on archived files:

851 MiB captured when using "delta-from" and 747 MiB captured when using "delta-from + update-of"

Final size of wdm files are 305 MiB and 271 MiB respectively.

Offline OS captured Win10x64 partition
Version used v1.13 beta 5 (--include-integrity option was used on all captures).
OS partition captured had no changes (not booted) when this tests had been ran.
Both captures made from Win7x64 OS, an both repited from WinPE-Win7x64 with exactly same results respectively for each.

It seems to me "delta-from + update-of" not detected changed files as accurately as "delta-from" (alone) does.

EDIT: This are the .cmd used:

Code: Select all

@echo off
"%~dp0\wimlib-imagex" capture G:\ D:\NO_BORRAR\Imagen_Asus\Win10x64_Delta.dwm  --delta-from=D:\NO_BORRAR\Imagen_Asus\Win10x64.wim --include-integrity

@echo off
"%~dp0\wimlib-imagex" capture G:\ D:\NO_BORRAR\Imagen_Asus\Win10x64_Delta+Update.dwm  --delta-from=D:\NO_BORRAR\Imagen_Asus\Win10x64.wim --update-of=D:\NO_BORRAR\Imagen_Asus\Win10x64.wim:-1 --include-integrity

Best Regards

Alacran

[Alacran]

@Synchronicity

If I open both dwm files using 7-zip v18.5, this is a list of files that have different size on Win10x64_Delta.dwm compared to Win10x64_Delta+Update.dwm

Code: Select all

Files on delta that are 0 size on delta + update (25 files):
(Remember 0 size is same as unchanged from reference wim)

Path											Bytes

Windows\appcompat\Programs\Amcache.hve.LOG1						32768

Windows\Prefetch\AgAppLaunch.db								334168

Windows\Prefetch\PfPre_db38097f.mkd							196620

Windows\ServiceProfiles\LocalService\NTUSER.DAT.LOG2					109568

Windows\ServiceProfiles\LocalService\AppData\Local\FontCache\FontCache-FontFace.dat	16777216

Windows\ServiceProfiles\LocalService\AppData\Local\FontCache\FontCache-S-1-5-18.dat	8388608

Windows\ServiceProfiles\LocalService\AppData\Local\FontCache\FontCache-S-1-5-21-3347186733-2820039293-3474590447-1000.dat	8388608

Windows\ServiceProfiles\NetworkService\NTUSER.DAT.LOG2					65536

Windows\System32\config\BBI.LOG2							89088

Windows\System32\config\SAM								65536

Windows\System32\config\SAM.LOG1							65536

Windows\System32\config\SAM.LOG2							65536

Windows\System32\config\SECURITY.LOG1							70656

Windows\System32\config\SECURITY.LOG2							65536

Windows\System32\config\SOFTWARE.LOG2							16252928

Users\Yo\AppData\Local\Microsoft\Internet Explorer\MSIMGSIZ.DAT				49120

Users\Yo\AppData\Local\Microsoft\Windows\Caches\cversions.1.db				16384

Users\Yo\AppData\Local\Microsoft\Windows\Caches\cversions.3.db				16384

Users\Yo\AppData\Local\Microsoft\Windows\Explorer\iconcache_16.db			16384

Users\Yo\AppData\Local\Microsoft\Windows\Explorer\iconcache_48.db			18874368

Users\Yo\AppData\Local\Microsoft\Windows\Explorer\iconcache_256.db			23068672

Users\Yo\AppData\Local\Microsoft\Windows\Explorer\iconcache_idx.db			930960

Users\Yo\AppData\Local\Microsoft\Windows\UsrClass.dat.LOG2				2044928

Users\Yo\AppData\Local\Packages\Microsoft.Windows.ShellExperienceHost_cw5n1h2txyewy\Settings\settings.dat.LOG1	37788

Users\Yo\AppData\Roaming\AVAST Software\Avast\Cache\Cache\index				524656

As you can see all are located on Users and Windows directories.

Best Regards

Alacran

[Alacran]

@Synchronicity

I made same captures of my Win7x64 partition/drive and also found same behaviour but only 4 files have different size:

Code: Select all

Path										Bytes

Users\Propietario\AppData\Roaming\AVAST Software\Avast\Cache\Cache\index	524 656

Windows\Prefetch\AgAppLaunch.db							334 168

Windows\ServiceProfiles\LocalService\AppData\Local\FontCache-FontFace.dat	16 777 216

Windows\System32\SMI\Store\Machine\SCHEMA.DAT					12 582 912

Again all are located on Users and Windows directories.

Then I extracted the files using a file list, those from Delta+Update are exactly the same as on referenced wim (since they are only a reference to it), even SHA1 is the same for both. 

Also extracted same way, the 4 files from Delta alone and found SHA1 was diferent from running OS.

Then it came to me the idea SHA1 on running system may vary every time I run it, to avoid this I booted from WinPE-7x64 and took new SHA1 for the 4 files from Win7x64 partition, also made a new Delta Capture, and from the PE extracted the 4 files from the Delta.dwm using the file list, and made SHA1 of the 4 extracted files, this time all 4 files have exactly the same SHA1 on Win7x64 partition and on Delta, but different from Delta+Update and reference wim.

Attached for download some PDF files of this resuts one for each file:

PDF.7z

(212.53 KiB) Downloaded 612 times

This way now we can verify if files are different or not, to be the identical, SHA1 has to be also the identical, if not there is something different.

And finaly this demonstrate only on Delta alone the 4 files have same SHA1 as system captured (only in that exact moment), and just rebooting the system, the SHA1 of this 4 files may change.

The 4 files on Delta+Update keep same SHA1 of the first captured wim, since they are only a reference to it.

Then only logical conclusion is "Capture ---delta-from" is definitively more accurate than "Capture ---delta-from + --update-of" when Capturing deltas of full OS partitions/drives.

Nevertheless "Capture ---delta-from + --update-of" may be good for capturing deltas of folders that are outside of Windows and Users directories.

IMHO the cause is after file inspection Delta start archiving all files only excluding those on wimscript.ini and Delta+Update exclude those on wimscript.ini + many more by an algoritm (???) and starts archiving from a very much smaller quantity of files, so as it has to read and make deep comparison of less less files that's why it is so faster. Then as IMHO the cause is on the algoritm used to exclude so many files from archiving. Maybe if files on Users and Windows directories are not included on the algoritm both captures match each other, of course we will loose some speed.

At first seen all files listed on previous post and on this one do not seem totally critical, but I am not quilified to judge this.

I will appreciate a lot your comments about this:

Are this files critical?
How can they impact on OS when we deploy our images?
Could you explain more deeply than in documentation the way capture --delta-from + --update-of work?

Thanks in advance

Best Regards

Alacran

[synchronicity]

The problem is that the file last modified timestamp is unreliable on Windows. This is already explained in the documentation for the --update-of option. But perhaps I should also make wimlib print a warning whenever that option is used on Windows, if not make it a no-op...

[Alacran]

Thanks for your answer:

The problem is that the file last modified timestamp is unreliable on Windows. This is already explained in the documentation for the --update-of option. But perhaps I should also make wimlib print a warning whenever that option is used on Windows, if not make it a no-op...

Yes, I have readen it, But that part on documentation also suggest a reboot should avoid the problem, I made all captures from another OS or WinPE, teoretically this should not happend. I checked time stamps an only on Windows\System32\SMI\Store\Machine\SCHEMA.DAT there is a change in Last Access date, but all others have same 3 dates: created, modified and last access. but diferent SHA1 (all 4 files) from first capture.

See picture of SCHEMA.DAT properties:

SCHEMA.DAT.png (190.2 KiB) Viewed 9605 times

Yes, definitely Windows do not change modified date on this files. Left is from first captured wim and also delta + update, central is from Delta and Right is from Win7x64 partition.

About the warning or the no-op.., I don't think this is really necessary, I suggest only clarify on documentation that even after a reboot not all files last modified timestamps will be updated, because I got the wrong idea thinking a reboot can solve totally this issue.

Best Regards

Alacran