Page 1 of 2

Comparision of --compact compression formats

Posted: Fri Aug 28, 2015 5:03 pm
by misty
The following is from the wimlib discussion forum on sourceforge (more specifically the Windows 10 compact mode topic) -

I suppose that DISM doesn't run into this problem because it probably only uses XPRESS4K.

With "WIMBoot" I also did not run into this problem, since the Windows 8.1 early boot code apparently did in fact support compression formats other than XPRESS4K. But maybe they have disabled all formats except XPRESS4K for "compact mode".

I'm not sure why Microsoft keeps using XPRESS4K; it's not actually a very good choice of compression format in my opinion.

Nevertheless, perhaps, if a list of files that need to be read before wof.sys is loaded could be hard-coded in wimlib, then it could support "compacting" with formats other than XPRESS4K. It would just use XPRESS4K for the hard-coded files or leave them uncompressed.
JFX has compiled a WimBootCompress.ini file for use with his WinNTSetup project that includes a list of files in the [PrepopulateList] section that will result in a Windows 10 installation that boots using compression formats other than xpress4k - I have tested this with xpress16k and lzx compression formats.

The following is a combination of JFX's WimBootCompress.ini with the WimBootCompress.ini file included in Windows 10 -

Code: Select all

; This is the inbox configuration file used for deploying or capture a
; WIMBoot system. Please do not remove this file because WIMCaptureImage 
; and WIMApplyImage will fail if WIM_FLAG_WIM_BOOT flag is specified.

[CompressionExclusionList]
ntoskrnl.exe

[PrepopulateList]
*winload.*
*winresume.*
\Windows\AppPatch\drvmain.sdb
\Windows\Fonts\vgaoem.fon
\Windows\Fonts\vgasys.fon
\Windows\INF\errata.inf
\Windows\System32\config\*
\Windows\System32\ntkrnlpa.exe
\Windows\System32\ntoskrnl.exe
\Windows\System32\bootvid.dll
\Windows\System32\ci.dll
\Windows\System32\hal*.dll
\Windows\System32\mcupdate_AuthenticAMD.dll
\Windows\System32\mcupdate_GenuineIntel.dll
\Windows\System32\pshed.dll
\Windows\System32\apisetschema.dll
\Windows\System32\api-ms-win*.dll
\Windows\System32\ext-ms-win*.dll
\Windows\System32\KernelBase.dll
\Windows\System32\drivers\*.sys
\Windows\System32\*.nls
\Windows\System32\kbd*.dll
\Windows\System32\kd*.dll
\Windows\System32\clfs.sys
\Windows\System32\CodeIntegrity\driver.stl

[ExclusionList]
\$bootdrive$
\$dwnlvldrive$
\$lsdrive$
\$installdrive$
\$Recycle.Bin\*
\bootsect.bak
\hiberfil.sys
\pagefile.sys
\ProgramData\Microsoft\Windows\SQM
\System Volume Information
\Users\*\AppData\Local\GDIPFONTCACHEV1.DAT
\Users\*\NTUSER.DAT*.TM.blf
\Users\*\NTUSER.DAT*.regtrans-ms
\Users\*\NTUSER.DAT*.log*
\Windows\AppCompat\Programs\Amcache.hve*.TM.blf
\Windows\AppCompat\Programs\Amcache.hve*.regtrans-ms
\Windows\AppCompat\Programs\Amcache.hve*.log*
\Windows\CSC
\Windows\Debug\*
\Windows\Logs\*
\Windows\Panther\*.etl
\Windows\Panther\*.log
\Windows\Panther\FastCleanup
\Windows\Panther\img
\Windows\Panther\Licenses
\Windows\Panther\MigLog*.xml
\Windows\Panther\Resources
\Windows\Panther\Rollback
\Windows\Panther\Setup*
\Windows\Panther\UnattendGC
\Windows\Panther\upgradematrix
\Windows\Prefetch\*
\Windows\ServiceProfiles\LocalService\NTUSER.DAT*.TM.blf
\Windows\ServiceProfiles\LocalService\NTUSER.DAT*.regtrans-ms
\Windows\ServiceProfiles\LocalService\NTUSER.DAT*.log*
\Windows\ServiceProfiles\NetworkService\NTUSER.DAT*.TM.blf
\Windows\ServiceProfiles\NetworkService\NTUSER.DAT*.regtrans-ms
\Windows\ServiceProfiles\NetworkService\NTUSER.DAT*.log*
\Windows\System32\config\RegBack\*
\Windows\System32\config\*.TM.blf
\Windows\System32\config\*.regtrans-ms
\Windows\System32\config\*.log*
\Windows\System32\SMI\Store\Machine\SCHEMA.DAT*.TM.blf
\Windows\System32\SMI\Store\Machine\SCHEMA.DAT*.regtrans-ms
\Windows\System32\SMI\Store\Machine\SCHEMA.DAT*.log*
\Windows\System32\sysprep\Panther
\Windows\System32\winevt\Logs\*
\Windows\System32\winevt\TraceFormat\*
\Windows\Temp\*
\Windows\TSSysprep.log
\Windows\winsxs\poqexec.log
\Windows\winsxs\ManifestCache\*
\Windows\servicing\Sessions\*_*.xml
\Windows\servicing\Sessions\Sessions.back.xml

[CompressionFolderList]
\Windows\System32\WinEvt\Logs
\Windows\Installer
I seriously doubt that all of the files in the [PrepopulateList] section need to be extracted without compression, however I have neither the time or the expertise to trace individual files - I do know that \Windows\System32\ntoskrnl.exe and \Windows\System32\hal*.dll need to be uncompressed (or compressed with xpress4k).

My tests were carried out using install.wim from a Windows 10 Enterprise LTSB source.

Using the unmodified install.wim Windows reported the following used space when right-clicking on the systemdrive and selecting Properties (in the running operating system - after the installation had completed and Windows 10 had booted) -
* 5.14 GB (--compact format=xpress4k)

Using the settings listed in this post in the WimBootCompress.ini file, used space -
* 4.99 GB (--compact format=xpress16k)
* 4.75 GB (--compact format=lzx)

For comparison purposes, the same Windows 10 source applied without the --compact switch -
* 6.45 GB

Regards,

Misty

Re: Comparision of --compact compression formats

Posted: Thu Sep 24, 2015 4:41 am
by synchronicity
Thanks for doing these tests. I've included the file list in wimlib-1.8.3-BETA2, and it now knows to force XPRESS4K compression on all files in the list. Therefore, it should now be possible to apply a full Windows image in "compact mode" using XPRESS8K, XPRESS16K, or LZX compression, with no changes to the default WimBootCompress.ini file, and have Windows successfully boot. Feel free to test this if you have a chance.

Re: Comparision of --compact compression formats

Posted: Thu Sep 24, 2015 4:49 pm
by JFX
Hi synchronicity,

here are 3 more exclusions.

Code: Select all

\Windows\Boot\DVD\*
\Windows\Boot\EFI\*
\Windows\bootstat.dat
It seems they only need by Win8.x in compact mode.

BTW: XPRESS4K will be okay on these files for a Win10 compact setup, but not for Win7/8.x.
Maybe you can apply these files without compression, if the build number in the WIM XML is smaller than Win10.

Re: Comparision of --compact compression formats

Posted: Sun Sep 27, 2015 3:02 pm
by synchronicity
Thanks for the suggestions. I've added the three additional patterns and also made the fallback be "uncompressed" rather than XPRESS4K if the image does not appear to be Windows 10 (or later). I haven't yet had time to fully test the changes but feel free to try them out in wimlib-1.8.3-BETA3.

Re: Comparision of --compact compression formats

Posted: Tue Jan 12, 2016 6:44 am
by bliblubli
In the doc, it says we must add the wofadk.sys driver. Where do I get this file and how can I install it on windows 7?

Re: Comparision of --compact compression formats

Posted: Tue Jan 12, 2016 6:21 pm
by JFX
The wofadk driver is a part of the Windows 10 ADK.
You can also use my GetWaikTools and only download Win10 DISM.

If you want to use a Windows 7/8.x in compact mode you can add the wofadk.sys to \Windows\system32\drivers
and import it's respected registry key

Code: Select all

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\WofAdk]
"DependOnService"=hex(7):46,00,6c,00,74,00,4d,00,67,00,72,00,00,00,00,00
"Discription"="WofAdk Mini-Filter Driver"
"DisplayName"="WofAdk"
"ErrorControl"=dword:00000001
"Group"="FSFilter Compression"
"Start"=dword:00000000
"SupportedFeatures"=dword:00000003
"Type"=dword:00000002
"DebugFlags"=dword:00000000
"AutoAttach"=dword:00000001
"Tag"=dword:00000002
"ImagePath"=hex(2):5c,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,\
  74,00,5c,00,53,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,64,00,72,\
  00,69,00,76,00,65,00,72,00,73,00,5c,00,77,00,6f,00,66,00,61,00,64,00,6b,00,\
  2e,00,73,00,79,00,73,00,00,00

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\WofAdk\Instances]
"DefaultInstance"="WofAdk Instance"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\WofAdk\Instances\WofAdk Instance]
"Altitude"="40730"
"Flags"=dword:00000000


Re: Comparision of --compact compression formats

Posted: Tue Jan 12, 2016 8:06 pm
by bliblubli
Hi JFX,
Nice to see you here. So to clarify the steps, after downloading the driver with your tool:
1) I add the wofadk driver to system32\drivers and add the .reg file.
2) I add the conf given above to \System32\WimBootCompress.ini. Is the .ini above in this thread the right one?
3) I make a capture with --snapshot with wimlib.
4) Then apply with --compact=LZX (is something special required on the system on which the apply command is run to get the system LZX compression?
5) Start the system normally.
6) Save some folder to a wim archive with whatever compression algorithm
7) Call DISM on startup through the task scheduler to mount the wim in the original location on startup.
Is it right?

Re: Comparision of --compact compression formats

Posted: Wed Jan 13, 2016 5:00 pm
by JFX
Hi bliblubli,

step 1 -5 sounds good. Don't really see the need for step 6 and 7.

Re: Comparision of --compact compression formats

Posted: Wed Jan 13, 2016 5:53 pm
by bliblubli
for 6 and 7 it's useless because all files will be already compressed with lzx? Will they stay compressed when changed? Does it work for newly created files?

Re: Comparision of --compact compression formats

Posted: Thu Jan 14, 2016 2:49 pm
by JFX
bliblubli wrote:for 6 and 7 it's useless because all files will be already compressed with lzx?
Yes, also WIM mounting will slow down and files accessed will be extracted, so no space is saved.
bliblubli wrote:Will they stay compressed when changed? Does it work for newly created files?
Good question, I guess it will not compress or re-compress any file.