Best practises for applying an image to existing volume

[supercilious]

Hi,

I have a couple of tablets with small SSDs. One is a Windows 8.1 tablet with 64GB, and another is a Windows 10 tablet with 32GB. On both machines, I periodically have to capture the installed image, and reapply it with --wimboot or --compact (respectively) to reclaim space used by updates and other various app installations.

Wimapply refuses to overwrite files on the existing volume (claiming permission denied, despite Administrator access), so I have resorted to formatting the volume and then applying the image. This works fine for windows itself, but some software (like Chrome) uses the volume GUID (other other aspect of the NTFS metadata) to encrypt its credentials and I have to reauthorise it to sync each time. Similar issues occur with other software, making it more cumbersome each time...

Is there a better way to update the compact/wimboot deployment to incorporate changes? Can wimlib-imagex intelligently update the existing image or at least apply the newly captured image on top of the existing filesystem replacing data as needed?

Thanks

[synchronicity]

wimapply is already supposed to replace existing files by default. I am very curious as to why you encountered permission denied errors. Can you provide as many details as you can? What was the exact error message, what file did it occur on, and what is the metadata for the file as given by 'wimdir --detail'? Was the target file WIM-backed, system compressed, or neither? Do you have a WIM image I can apply with certain options to reproduce the problem?

[supercilious]

For example, I just tested with a Windows 10 image in a virtual machine (since its the smallest image I have):

I mounted the drive as E: in my VM and captured the image:

Code: Select all

C:\>C:\WIM\wimcapture E:\ Image.wim --compress=xpress
Scanning "E:\"
129 bytes scanned (1 files, 0 directories)
Excluding "E:\hiberfil.sys" from capture

Excluding "E:\pagefile.sys" from capture
461 MiB scanned (3284 files, 656 directories)
Excluding "E:\System Volume Information" from capture
718 MiB scanned (4660 files, 1484 directories)
Excluding "E:\Windows\CSC" from capture
5893 MiB scanned (49731 files, 16213 directories)
Writing XPRESS-compressed data using 4 threads
5574 MiB of 5574 MiB (uncompressed) written (100% done)

I then tried to re-apply the image on the same E: volume:

Code: Select all

C:\>C:\WIM\wimapply C:\Image.wim E:\ --compact=lzx
[WARNING] Ignoring FILE_ATTRIBUTE_SPARSE_FILE of 2 files
Applying image 1 ("E:") from "C:\Image.wim" to directory "E:\"
[ERROR] Can't create directory "E:\\Program Files\WindowsApps\DeletedAllUserPackages\Microsoft.Windows.Photos_15.1001.16470.0_neutral_split.scale-125_8wekyb3d8bbwe" (status=c0000022): Access is denied
ERROR: Exiting with error code 37:
       Failed to create a directory.

I used the previous version of wimlib:

Code: Select all

C:\>C:\WIM\wimlib-imagex --version
wimlib-imagex (distributed with wimlib 1.8.3)
Copyright (C) 2012, 2013, 2014, 2015 Eric Biggers
License GPLv3+; GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Report bugs to ebiggers3@gmail.com.

Here is the wiminfo on the WIM I just created:

Code: Select all

C:\>C:\WIM\wiminfo C:\Image.wim
WIM Information:
----------------
Path:           C:\Image.wim
GUID:           0x1ad4babbccc7fed50f95183df1dbbc28
Version:        68864
Image Count:    1
Compression:    XPRESS
Chunk Size:     32768 bytes
Part Number:    1/1
Boot Index:     0
Size:           2623928330 bytes
Attributes:     Relative path junction

Available Images:
-----------------
Index:                  1
Name:                   E:
Description:
Directory Count:        16149
File Count:             76934
Total Bytes:            11216637642
Hard Link Bytes:        5036730882
Creation Time:          Sun Jan 31 16:02:12 2016 UTC
Last Modification Time: Sun Jan 31 16:02:30 2016 UTC
WIMBoot compatible:     no

[supercilious]

All the above was done from an elevated command prompt of course:

Code: Select all

USER INFORMATION
----------------

User Name            SID                                           
==================== ==============================================
desktop-n5k9p1f\user S-1-5-21-2381263475-1422355460-1692112598-1001


GROUP INFORMATION
-----------------

Group Name                                                    Type             SID          Attributes                                                     
============================================================= ================ ============ ===============================================================
Everyone                                                      Well-known group S-1-1-0      Mandatory group, Enabled by default, Enabled group             
NT AUTHORITY\Local account and member of Administrators group Well-known group S-1-5-114    Mandatory group, Enabled by default, Enabled group             
BUILTIN\Administrators                                        Alias            S-1-5-32-544 Mandatory group, Enabled by default, Enabled group, Group owner
BUILTIN\Users                                                 Alias            S-1-5-32-545 Mandatory group, Enabled by default, Enabled group             
NT AUTHORITY\INTERACTIVE                                      Well-known group S-1-5-4      Mandatory group, Enabled by default, Enabled group             
CONSOLE LOGON                                                 Well-known group S-1-2-1      Mandatory group, Enabled by default, Enabled group             
NT AUTHORITY\Authenticated Users                              Well-known group S-1-5-11     Mandatory group, Enabled by default, Enabled group             
NT AUTHORITY\This Organization                                Well-known group S-1-5-15     Mandatory group, Enabled by default, Enabled group             
NT AUTHORITY\Local account                                    Well-known group S-1-5-113    Mandatory group, Enabled by default, Enabled group             
LOCAL                                                         Well-known group S-1-2-0      Mandatory group, Enabled by default, Enabled group             
NT AUTHORITY\NTLM Authentication                              Well-known group S-1-5-64-10  Mandatory group, Enabled by default, Enabled group             
Mandatory Label\High Mandatory Level                          Label            S-1-16-12288                                                                


PRIVILEGES INFORMATION
----------------------

Privilege Name                  Description                               State   
=============================== ========================================= ========
SeIncreaseQuotaPrivilege        Adjust memory quotas for a process        Disabled
SeSecurityPrivilege             Manage auditing and security log          Disabled
SeTakeOwnershipPrivilege        Take ownership of files or other objects  Disabled
SeLoadDriverPrivilege           Load and unload device drivers            Disabled
SeSystemProfilePrivilege        Profile system performance                Disabled
SeSystemtimePrivilege           Change the system time                    Disabled
SeProfileSingleProcessPrivilege Profile single process                    Disabled
SeIncreaseBasePriorityPrivilege Increase scheduling priority              Disabled
SeCreatePagefilePrivilege       Create a pagefile                         Disabled
SeBackupPrivilege               Back up files and directories             Disabled
SeRestorePrivilege              Restore files and directories             Disabled
SeShutdownPrivilege             Shut down the system                      Disabled
SeDebugPrivilege                Debug programs                            Disabled
SeSystemEnvironmentPrivilege    Modify firmware environment values        Disabled
SeChangeNotifyPrivilege         Bypass traverse checking                  Enabled 
SeRemoteShutdownPrivilege       Force shutdown from a remote system       Disabled
SeUndockPrivilege               Remove computer from docking station      Disabled
SeManageVolumePrivilege         Perform volume maintenance tasks          Disabled
SeImpersonatePrivilege          Impersonate a client after authentication Enabled 
SeCreateGlobalPrivilege         Create global objects                     Enabled 
SeIncreaseWorkingSetPrivilege   Increase a process working set            Disabled
SeTimeZonePrivilege             Change the time zone                      Disabled
SeCreateSymbolicLinkPrivilege   Create symbolic links                     Disabled

[synchronicity]

Can you post the output of the following command?

Code: Select all

wimdir --detail C:\Image.wim 1 --path="\Program Files\WindowsApps\DeletedAllUserPackages\Microsoft.Windows.Photos_15.1001.16470.0_neutral_split.scale-125_8wekyb3d8bbwe" --one-file-only

[supercilious]

Code: Select all

C:\WIM\wimdir --detail C:\Image.wim 1 --path="\Program Files\WindowsApps\DeletedAllUserPackages\Microsoft.Windows.Photos_15.1001.16470.0_neutral_split.scale-125_8wekyb3d8bbwe" --one-file-only
----------------------------------------------------------------------------
Full Path           = "\Program Files\WindowsApps\DeletedAllUserPackages\Microsoft.Windows.Photos_15.1001.16470.0_neutral_split.scale-125_8wekyb3d8bbwe"
Short Name          = "MICROS~3.SCA"
Attributes          = 0x00000010
    FILE_ATTRIBUTE_DIRECTORY is set
Security Descriptor = O:BAG:SYD:AI(A;OICI;0x1200a9;;;BU)(A;OICI;0x1200a9;;;S-1-15-3-2226957697-3030467180-2301525-4248967783-2024719031-2325529081-2915787518)(A;OICI;0x1200a9;;;S-1-15-2-2226957697-3030467180-2301525-4248967783-2024719031-2325529081-2915787518)(A;ID;FA;;;S-1-5-80-956008885-3418522649-1831038044-1853292631-2271478464)(A;CIIOID;GA;;;S-1-5-80-956008885-3418522649-1831038044-1853292631-2271478464)(A;ID;0x1200a9;;;S-1-15-3-1024-3635283841-2530182609-996808640-1887759898-3848208603-3313616867-983405619-2501854204)(A;OICIIOID;GXGR;;;S-1-15-3-1024-3635283841-2530182609-996808640-1887759898-3848208603-3313616867-983405619-2501854204)(A;ID;FA;;;SY)(A;OICIIOID;GA;;;SY)(A;OICIID;0x1200a9;;;BA)(A;OICIID;0x1200a9;;;LS)(A;OICIID;0x1200a9;;;NS)S:
Creation Time       = Fri Oct 30 09:12:08 2015 UTC
Last Write Time     = Sun Jan 31 15:55:11 2016 UTC
Last Access Time    = Sun Jan 31 15:55:11 2016 UTC
Link Group ID       = 0x0000000000000000
Link Count          = 1
	Unnamed data stream:
Hash              = 0x0000000000000000000000000000000000000000
Uncompressed size = 0 bytes
Compressed size   = 0 bytes
Offset in WIM     = 0 bytes
Part Number       = 0
Reference Count   = 0
Flags             = 

[synchronicity]

Thanks. That rules out any sort of strange file attributes causing a problem; it's simply a regular directory.

Several months ago, sometime else actually reported what was likely this some problem. But I was never able to get to the bottom of it.

Now, I finally was able to reproduce the problem with the directory you mentioned, from Windows 10.

I have to say that this is one of the strangest errors I have ever seen from Windows. It has the following characteristics:

* The error occurs only when requesting FILE_WRITE_DATA access on a directory.
* The error only occurs on Windows 10. It does not occur on earlier versions of Windows, even Windows 8.1, and even when accessing the exact same filesystem (that is, if I have Windows 8.1 mount the filesystem from Windows 10).
* The error occurs regardless of whether an instance of WOF is attached to the filesystem or not.
* The error does not occur on every directory, but on some directories it occurs consistently.
* The error occurs regardless of the security descriptor on the directory or its parent.
* The error occurs regardless of the file attributes on the directory.
* The error occurs regardless of whether the directory is opened with backup semantics or not.
* The error occurs regardless of whether the program has full admin and restore privileges.
* The error occurs regardless of whether the directory is opened with a relative path or an absolute path.

Personally I think the evidence points to this being some sort of regression in the Windows kernel or the NTFS driver. Most applications will never run into this problem since they don't request FILE_WRITE_DATA permission on directories. wimlib only does so because it's required for changing the NTFS compression attribute (it wants the final value of the attribute to match that specified in the WIM image). So the obvious workaround will be to just skip FILE_WRITE_DATA permission if Windows isn't willing to grant it, and skipping changing the compression attribute, which usually already has the correct value anyway.

I won't implement this workaround right away since I just released 1.9.0; maybe in a week or two.

Let me know if you notice anything that refutes my theory (did you ever notice the problem on Windows 8.1 on just on Windows 10?)

[supercilious]

* The error does not occur on every directory, but on some directories it occurs consistently.

Is there a way to ignore the error so I can see a list of directories which trigger it? What do these directories have in common?

Something odd I noticed, which may or may not be related, is that certain windows modern app files are "protected" by the windows file protection services and modifications are immediately reverted. I stumbled on to this issue when I realised that if I delete a user's profile data (from the system control panel), it does not remove the C:\program files\windowsapps\... links to the user's profile, and therefore the app is never unlinked and never removed to recover space, even though the user profile it was registered in is gone. The only work-around is to log-in as the user, and with powershell cd into that app's directory:

Code: Select all

Add-AppxPackage -DisableDevelopmentMode -Register appxmanifest.xml

And then "uninstall" that app the conventional way. Only then will windows allow it to be removed. Modifying the files any other way will result in the file protection system kicking-in and restoring the files. I have never known this to happen on external images, only the running image though.

[supercilious]

I just tried the exact same procedure on Windows 8.1 (using the same Windows 10 image mounted at E:\). I got the same problem, so it seems very unlikely to be a kernel error.

[synchronicity]

You are correct that the problem occurs on Windows 8.1 as well. One of my tests must have been done incorrectly. One thing I did notice is that the problem goes away entirely on Windows 8.1 and Windows 10 if you overwrite the folder from Windows 7.

It's actually a kernel level error by definition --- the request to open the file is being made directly to the Windows kernel and it is sending back an error code. It could be a driver that is generating the error and not the kernel itself, though. Interesting theory about a file protection service. I thought that didn't exist anymore and Windows had moved to ACLs. Do you have any more information about how this may be used in the WindowsApps folder?

wimlib-imagex doesn't normally support ignoring errors like this, but for testing purposes I updated it to and attached the list of errors.