Best practises for applying an image to existing volume
-
- Posts: 19
- Joined: Thu Sep 03, 2015 9:24 pm
Best practises for applying an image to existing volume
Hi,
I have a couple of tablets with small SSDs. One is a Windows 8.1 tablet with 64GB, and another is a Windows 10 tablet with 32GB. On both machines, I periodically have to capture the installed image, and reapply it with --wimboot or --compact (respectively) to reclaim space used by updates and other various app installations.
Wimapply refuses to overwrite files on the existing volume (claiming permission denied, despite Administrator access), so I have resorted to formatting the volume and then applying the image. This works fine for windows itself, but some software (like Chrome) uses the volume GUID (other other aspect of the NTFS metadata) to encrypt its credentials and I have to reauthorise it to sync each time. Similar issues occur with other software, making it more cumbersome each time...
Is there a better way to update the compact/wimboot deployment to incorporate changes? Can wimlib-imagex intelligently update the existing image or at least apply the newly captured image on top of the existing filesystem replacing data as needed?
Thanks
I have a couple of tablets with small SSDs. One is a Windows 8.1 tablet with 64GB, and another is a Windows 10 tablet with 32GB. On both machines, I periodically have to capture the installed image, and reapply it with --wimboot or --compact (respectively) to reclaim space used by updates and other various app installations.
Wimapply refuses to overwrite files on the existing volume (claiming permission denied, despite Administrator access), so I have resorted to formatting the volume and then applying the image. This works fine for windows itself, but some software (like Chrome) uses the volume GUID (other other aspect of the NTFS metadata) to encrypt its credentials and I have to reauthorise it to sync each time. Similar issues occur with other software, making it more cumbersome each time...
Is there a better way to update the compact/wimboot deployment to incorporate changes? Can wimlib-imagex intelligently update the existing image or at least apply the newly captured image on top of the existing filesystem replacing data as needed?
Thanks
-
- Site Admin
- Posts: 474
- Joined: Sun Aug 02, 2015 10:31 pm
Re: Best practises for applying an image to existing volume
wimapply is already supposed to replace existing files by default. I am very curious as to why you encountered permission denied errors. Can you provide as many details as you can? What was the exact error message, what file did it occur on, and what is the metadata for the file as given by 'wimdir --detail'? Was the target file WIM-backed, system compressed, or neither? Do you have a WIM image I can apply with certain options to reproduce the problem?
-
- Posts: 19
- Joined: Thu Sep 03, 2015 9:24 pm
Re: Best practises for applying an image to existing volume
For example, I just tested with a Windows 10 image in a virtual machine (since its the smallest image I have):
I mounted the drive as E: in my VM and captured the image:
I then tried to re-apply the image on the same E: volume:
I used the previous version of wimlib:
Here is the wiminfo on the WIM I just created:
I mounted the drive as E: in my VM and captured the image:
Code: Select all
C:\>C:\WIM\wimcapture E:\ Image.wim --compress=xpress
Scanning "E:\"
129 bytes scanned (1 files, 0 directories)
Excluding "E:\hiberfil.sys" from capture
Excluding "E:\pagefile.sys" from capture
461 MiB scanned (3284 files, 656 directories)
Excluding "E:\System Volume Information" from capture
718 MiB scanned (4660 files, 1484 directories)
Excluding "E:\Windows\CSC" from capture
5893 MiB scanned (49731 files, 16213 directories)
Writing XPRESS-compressed data using 4 threads
5574 MiB of 5574 MiB (uncompressed) written (100% done)
Code: Select all
C:\>C:\WIM\wimapply C:\Image.wim E:\ --compact=lzx
[WARNING] Ignoring FILE_ATTRIBUTE_SPARSE_FILE of 2 files
Applying image 1 ("E:") from "C:\Image.wim" to directory "E:\"
[ERROR] Can't create directory "E:\\Program Files\WindowsApps\DeletedAllUserPackages\Microsoft.Windows.Photos_15.1001.16470.0_neutral_split.scale-125_8wekyb3d8bbwe" (status=c0000022): Access is denied
ERROR: Exiting with error code 37:
Failed to create a directory.
Code: Select all
C:\>C:\WIM\wimlib-imagex --version
wimlib-imagex (distributed with wimlib 1.8.3)
Copyright (C) 2012, 2013, 2014, 2015 Eric Biggers
License GPLv3+; GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Report bugs to ebiggers3@gmail.com.
Code: Select all
C:\>C:\WIM\wiminfo C:\Image.wim
WIM Information:
----------------
Path: C:\Image.wim
GUID: 0x1ad4babbccc7fed50f95183df1dbbc28
Version: 68864
Image Count: 1
Compression: XPRESS
Chunk Size: 32768 bytes
Part Number: 1/1
Boot Index: 0
Size: 2623928330 bytes
Attributes: Relative path junction
Available Images:
-----------------
Index: 1
Name: E:
Description:
Directory Count: 16149
File Count: 76934
Total Bytes: 11216637642
Hard Link Bytes: 5036730882
Creation Time: Sun Jan 31 16:02:12 2016 UTC
Last Modification Time: Sun Jan 31 16:02:30 2016 UTC
WIMBoot compatible: no
-
- Posts: 19
- Joined: Thu Sep 03, 2015 9:24 pm
Re: Best practises for applying an image to existing volume
All the above was done from an elevated command prompt of course:
Code: Select all
USER INFORMATION
----------------
User Name SID
==================== ==============================================
desktop-n5k9p1f\user S-1-5-21-2381263475-1422355460-1692112598-1001
GROUP INFORMATION
-----------------
Group Name Type SID Attributes
============================================================= ================ ============ ===============================================================
Everyone Well-known group S-1-1-0 Mandatory group, Enabled by default, Enabled group
NT AUTHORITY\Local account and member of Administrators group Well-known group S-1-5-114 Mandatory group, Enabled by default, Enabled group
BUILTIN\Administrators Alias S-1-5-32-544 Mandatory group, Enabled by default, Enabled group, Group owner
BUILTIN\Users Alias S-1-5-32-545 Mandatory group, Enabled by default, Enabled group
NT AUTHORITY\INTERACTIVE Well-known group S-1-5-4 Mandatory group, Enabled by default, Enabled group
CONSOLE LOGON Well-known group S-1-2-1 Mandatory group, Enabled by default, Enabled group
NT AUTHORITY\Authenticated Users Well-known group S-1-5-11 Mandatory group, Enabled by default, Enabled group
NT AUTHORITY\This Organization Well-known group S-1-5-15 Mandatory group, Enabled by default, Enabled group
NT AUTHORITY\Local account Well-known group S-1-5-113 Mandatory group, Enabled by default, Enabled group
LOCAL Well-known group S-1-2-0 Mandatory group, Enabled by default, Enabled group
NT AUTHORITY\NTLM Authentication Well-known group S-1-5-64-10 Mandatory group, Enabled by default, Enabled group
Mandatory Label\High Mandatory Level Label S-1-16-12288
PRIVILEGES INFORMATION
----------------------
Privilege Name Description State
=============================== ========================================= ========
SeIncreaseQuotaPrivilege Adjust memory quotas for a process Disabled
SeSecurityPrivilege Manage auditing and security log Disabled
SeTakeOwnershipPrivilege Take ownership of files or other objects Disabled
SeLoadDriverPrivilege Load and unload device drivers Disabled
SeSystemProfilePrivilege Profile system performance Disabled
SeSystemtimePrivilege Change the system time Disabled
SeProfileSingleProcessPrivilege Profile single process Disabled
SeIncreaseBasePriorityPrivilege Increase scheduling priority Disabled
SeCreatePagefilePrivilege Create a pagefile Disabled
SeBackupPrivilege Back up files and directories Disabled
SeRestorePrivilege Restore files and directories Disabled
SeShutdownPrivilege Shut down the system Disabled
SeDebugPrivilege Debug programs Disabled
SeSystemEnvironmentPrivilege Modify firmware environment values Disabled
SeChangeNotifyPrivilege Bypass traverse checking Enabled
SeRemoteShutdownPrivilege Force shutdown from a remote system Disabled
SeUndockPrivilege Remove computer from docking station Disabled
SeManageVolumePrivilege Perform volume maintenance tasks Disabled
SeImpersonatePrivilege Impersonate a client after authentication Enabled
SeCreateGlobalPrivilege Create global objects Enabled
SeIncreaseWorkingSetPrivilege Increase a process working set Disabled
SeTimeZonePrivilege Change the time zone Disabled
SeCreateSymbolicLinkPrivilege Create symbolic links Disabled
-
- Site Admin
- Posts: 474
- Joined: Sun Aug 02, 2015 10:31 pm
Re: Best practises for applying an image to existing volume
Can you post the output of the following command?
Code: Select all
wimdir --detail C:\Image.wim 1 --path="\Program Files\WindowsApps\DeletedAllUserPackages\Microsoft.Windows.Photos_15.1001.16470.0_neutral_split.scale-125_8wekyb3d8bbwe" --one-file-only
-
- Posts: 19
- Joined: Thu Sep 03, 2015 9:24 pm
Re: Best practises for applying an image to existing volume
Code: Select all
C:\WIM\wimdir --detail C:\Image.wim 1 --path="\Program Files\WindowsApps\DeletedAllUserPackages\Microsoft.Windows.Photos_15.1001.16470.0_neutral_split.scale-125_8wekyb3d8bbwe" --one-file-only
----------------------------------------------------------------------------
Full Path = "\Program Files\WindowsApps\DeletedAllUserPackages\Microsoft.Windows.Photos_15.1001.16470.0_neutral_split.scale-125_8wekyb3d8bbwe"
Short Name = "MICROS~3.SCA"
Attributes = 0x00000010
FILE_ATTRIBUTE_DIRECTORY is set
Security Descriptor = O:BAG:SYD:AI(A;OICI;0x1200a9;;;BU)(A;OICI;0x1200a9;;;S-1-15-3-2226957697-3030467180-2301525-4248967783-2024719031-2325529081-2915787518)(A;OICI;0x1200a9;;;S-1-15-2-2226957697-3030467180-2301525-4248967783-2024719031-2325529081-2915787518)(A;ID;FA;;;S-1-5-80-956008885-3418522649-1831038044-1853292631-2271478464)(A;CIIOID;GA;;;S-1-5-80-956008885-3418522649-1831038044-1853292631-2271478464)(A;ID;0x1200a9;;;S-1-15-3-1024-3635283841-2530182609-996808640-1887759898-3848208603-3313616867-983405619-2501854204)(A;OICIIOID;GXGR;;;S-1-15-3-1024-3635283841-2530182609-996808640-1887759898-3848208603-3313616867-983405619-2501854204)(A;ID;FA;;;SY)(A;OICIIOID;GA;;;SY)(A;OICIID;0x1200a9;;;BA)(A;OICIID;0x1200a9;;;LS)(A;OICIID;0x1200a9;;;NS)S:
Creation Time = Fri Oct 30 09:12:08 2015 UTC
Last Write Time = Sun Jan 31 15:55:11 2016 UTC
Last Access Time = Sun Jan 31 15:55:11 2016 UTC
Link Group ID = 0x0000000000000000
Link Count = 1
Unnamed data stream:
Hash = 0x0000000000000000000000000000000000000000
Uncompressed size = 0 bytes
Compressed size = 0 bytes
Offset in WIM = 0 bytes
Part Number = 0
Reference Count = 0
Flags =
-
- Site Admin
- Posts: 474
- Joined: Sun Aug 02, 2015 10:31 pm
Re: Best practises for applying an image to existing volume
Thanks. That rules out any sort of strange file attributes causing a problem; it's simply a regular directory.
Several months ago, sometime else actually reported what was likely this some problem. But I was never able to get to the bottom of it.
Now, I finally was able to reproduce the problem with the directory you mentioned, from Windows 10.
I have to say that this is one of the strangest errors I have ever seen from Windows. It has the following characteristics:
* The error occurs only when requesting FILE_WRITE_DATA access on a directory.
* The error only occurs on Windows 10. It does not occur on earlier versions of Windows, even Windows 8.1, and even when accessing the exact same filesystem (that is, if I have Windows 8.1 mount the filesystem from Windows 10).
* The error occurs regardless of whether an instance of WOF is attached to the filesystem or not.
* The error does not occur on every directory, but on some directories it occurs consistently.
* The error occurs regardless of the security descriptor on the directory or its parent.
* The error occurs regardless of the file attributes on the directory.
* The error occurs regardless of whether the directory is opened with backup semantics or not.
* The error occurs regardless of whether the program has full admin and restore privileges.
* The error occurs regardless of whether the directory is opened with a relative path or an absolute path.
Personally I think the evidence points to this being some sort of regression in the Windows kernel or the NTFS driver. Most applications will never run into this problem since they don't request FILE_WRITE_DATA permission on directories. wimlib only does so because it's required for changing the NTFS compression attribute (it wants the final value of the attribute to match that specified in the WIM image). So the obvious workaround will be to just skip FILE_WRITE_DATA permission if Windows isn't willing to grant it, and skipping changing the compression attribute, which usually already has the correct value anyway.
I won't implement this workaround right away since I just released 1.9.0; maybe in a week or two.
Let me know if you notice anything that refutes my theory (did you ever notice the problem on Windows 8.1 on just on Windows 10?)
Several months ago, sometime else actually reported what was likely this some problem. But I was never able to get to the bottom of it.
Now, I finally was able to reproduce the problem with the directory you mentioned, from Windows 10.
I have to say that this is one of the strangest errors I have ever seen from Windows. It has the following characteristics:
* The error occurs only when requesting FILE_WRITE_DATA access on a directory.
* The error only occurs on Windows 10. It does not occur on earlier versions of Windows, even Windows 8.1, and even when accessing the exact same filesystem (that is, if I have Windows 8.1 mount the filesystem from Windows 10).
* The error occurs regardless of whether an instance of WOF is attached to the filesystem or not.
* The error does not occur on every directory, but on some directories it occurs consistently.
* The error occurs regardless of the security descriptor on the directory or its parent.
* The error occurs regardless of the file attributes on the directory.
* The error occurs regardless of whether the directory is opened with backup semantics or not.
* The error occurs regardless of whether the program has full admin and restore privileges.
* The error occurs regardless of whether the directory is opened with a relative path or an absolute path.
Personally I think the evidence points to this being some sort of regression in the Windows kernel or the NTFS driver. Most applications will never run into this problem since they don't request FILE_WRITE_DATA permission on directories. wimlib only does so because it's required for changing the NTFS compression attribute (it wants the final value of the attribute to match that specified in the WIM image). So the obvious workaround will be to just skip FILE_WRITE_DATA permission if Windows isn't willing to grant it, and skipping changing the compression attribute, which usually already has the correct value anyway.
I won't implement this workaround right away since I just released 1.9.0; maybe in a week or two.
Let me know if you notice anything that refutes my theory (did you ever notice the problem on Windows 8.1 on just on Windows 10?)
-
- Posts: 19
- Joined: Thu Sep 03, 2015 9:24 pm
Re: Best practises for applying an image to existing volume
Is there a way to ignore the error so I can see a list of directories which trigger it? What do these directories have in common?synchronicity wrote:* The error does not occur on every directory, but on some directories it occurs consistently.
Something odd I noticed, which may or may not be related, is that certain windows modern app files are "protected" by the windows file protection services and modifications are immediately reverted. I stumbled on to this issue when I realised that if I delete a user's profile data (from the system control panel), it does not remove the C:\program files\windowsapps\... links to the user's profile, and therefore the app is never unlinked and never removed to recover space, even though the user profile it was registered in is gone. The only work-around is to log-in as the user, and with powershell cd into that app's directory:
Code: Select all
Add-AppxPackage -DisableDevelopmentMode -Register appxmanifest.xml
-
- Posts: 19
- Joined: Thu Sep 03, 2015 9:24 pm
Re: Best practises for applying an image to existing volume
I just tried the exact same procedure on Windows 8.1 (using the same Windows 10 image mounted at E:\). I got the same problem, so it seems very unlikely to be a kernel error.
- Attachments
-
- wimapply.png (78.64 KiB) Viewed 37843 times
-
- Site Admin
- Posts: 474
- Joined: Sun Aug 02, 2015 10:31 pm
Re: Best practises for applying an image to existing volume
You are correct that the problem occurs on Windows 8.1 as well. One of my tests must have been done incorrectly. One thing I did notice is that the problem goes away entirely on Windows 8.1 and Windows 10 if you overwrite the folder from Windows 7.
It's actually a kernel level error by definition --- the request to open the file is being made directly to the Windows kernel and it is sending back an error code. It could be a driver that is generating the error and not the kernel itself, though. Interesting theory about a file protection service. I thought that didn't exist anymore and Windows had moved to ACLs. Do you have any more information about how this may be used in the WindowsApps folder?
wimlib-imagex doesn't normally support ignoring errors like this, but for testing purposes I updated it to and attached the list of errors.
It's actually a kernel level error by definition --- the request to open the file is being made directly to the Windows kernel and it is sending back an error code. It could be a driver that is generating the error and not the kernel itself, though. Interesting theory about a file protection service. I thought that didn't exist anymore and Windows had moved to ACLs. Do you have any more information about how this may be used in the WindowsApps folder?
wimlib-imagex doesn't normally support ignoring errors like this, but for testing purposes I updated it to and attached the list of errors.
- Attachments
-
- warnings.txt
- (44.71 KiB) Downloaded 993 times