Comparision of --compact compression formats

Comments, questions, bug reports, etc.
misty
Posts: 12
Joined: Tue Aug 18, 2015 10:23 am

Comparision of --compact compression formats

Post by misty »

The following is from the wimlib discussion forum on sourceforge (more specifically the Windows 10 compact mode topic) -

I suppose that DISM doesn't run into this problem because it probably only uses XPRESS4K.

With "WIMBoot" I also did not run into this problem, since the Windows 8.1 early boot code apparently did in fact support compression formats other than XPRESS4K. But maybe they have disabled all formats except XPRESS4K for "compact mode".

I'm not sure why Microsoft keeps using XPRESS4K; it's not actually a very good choice of compression format in my opinion.

Nevertheless, perhaps, if a list of files that need to be read before wof.sys is loaded could be hard-coded in wimlib, then it could support "compacting" with formats other than XPRESS4K. It would just use XPRESS4K for the hard-coded files or leave them uncompressed.
JFX has compiled a WimBootCompress.ini file for use with his WinNTSetup project that includes a list of files in the [PrepopulateList] section that will result in a Windows 10 installation that boots using compression formats other than xpress4k - I have tested this with xpress16k and lzx compression formats.

The following is a combination of JFX's WimBootCompress.ini with the WimBootCompress.ini file included in Windows 10 -

Code: Select all

; This is the inbox configuration file used for deploying or capture a
; WIMBoot system. Please do not remove this file because WIMCaptureImage 
; and WIMApplyImage will fail if WIM_FLAG_WIM_BOOT flag is specified.

[CompressionExclusionList]
ntoskrnl.exe

[PrepopulateList]
*winload.*
*winresume.*
\Windows\AppPatch\drvmain.sdb
\Windows\Fonts\vgaoem.fon
\Windows\Fonts\vgasys.fon
\Windows\INF\errata.inf
\Windows\System32\config\*
\Windows\System32\ntkrnlpa.exe
\Windows\System32\ntoskrnl.exe
\Windows\System32\bootvid.dll
\Windows\System32\ci.dll
\Windows\System32\hal*.dll
\Windows\System32\mcupdate_AuthenticAMD.dll
\Windows\System32\mcupdate_GenuineIntel.dll
\Windows\System32\pshed.dll
\Windows\System32\apisetschema.dll
\Windows\System32\api-ms-win*.dll
\Windows\System32\ext-ms-win*.dll
\Windows\System32\KernelBase.dll
\Windows\System32\drivers\*.sys
\Windows\System32\*.nls
\Windows\System32\kbd*.dll
\Windows\System32\kd*.dll
\Windows\System32\clfs.sys
\Windows\System32\CodeIntegrity\driver.stl

[ExclusionList]
\$bootdrive$
\$dwnlvldrive$
\$lsdrive$
\$installdrive$
\$Recycle.Bin\*
\bootsect.bak
\hiberfil.sys
\pagefile.sys
\ProgramData\Microsoft\Windows\SQM
\System Volume Information
\Users\*\AppData\Local\GDIPFONTCACHEV1.DAT
\Users\*\NTUSER.DAT*.TM.blf
\Users\*\NTUSER.DAT*.regtrans-ms
\Users\*\NTUSER.DAT*.log*
\Windows\AppCompat\Programs\Amcache.hve*.TM.blf
\Windows\AppCompat\Programs\Amcache.hve*.regtrans-ms
\Windows\AppCompat\Programs\Amcache.hve*.log*
\Windows\CSC
\Windows\Debug\*
\Windows\Logs\*
\Windows\Panther\*.etl
\Windows\Panther\*.log
\Windows\Panther\FastCleanup
\Windows\Panther\img
\Windows\Panther\Licenses
\Windows\Panther\MigLog*.xml
\Windows\Panther\Resources
\Windows\Panther\Rollback
\Windows\Panther\Setup*
\Windows\Panther\UnattendGC
\Windows\Panther\upgradematrix
\Windows\Prefetch\*
\Windows\ServiceProfiles\LocalService\NTUSER.DAT*.TM.blf
\Windows\ServiceProfiles\LocalService\NTUSER.DAT*.regtrans-ms
\Windows\ServiceProfiles\LocalService\NTUSER.DAT*.log*
\Windows\ServiceProfiles\NetworkService\NTUSER.DAT*.TM.blf
\Windows\ServiceProfiles\NetworkService\NTUSER.DAT*.regtrans-ms
\Windows\ServiceProfiles\NetworkService\NTUSER.DAT*.log*
\Windows\System32\config\RegBack\*
\Windows\System32\config\*.TM.blf
\Windows\System32\config\*.regtrans-ms
\Windows\System32\config\*.log*
\Windows\System32\SMI\Store\Machine\SCHEMA.DAT*.TM.blf
\Windows\System32\SMI\Store\Machine\SCHEMA.DAT*.regtrans-ms
\Windows\System32\SMI\Store\Machine\SCHEMA.DAT*.log*
\Windows\System32\sysprep\Panther
\Windows\System32\winevt\Logs\*
\Windows\System32\winevt\TraceFormat\*
\Windows\Temp\*
\Windows\TSSysprep.log
\Windows\winsxs\poqexec.log
\Windows\winsxs\ManifestCache\*
\Windows\servicing\Sessions\*_*.xml
\Windows\servicing\Sessions\Sessions.back.xml

[CompressionFolderList]
\Windows\System32\WinEvt\Logs
\Windows\Installer
I seriously doubt that all of the files in the [PrepopulateList] section need to be extracted without compression, however I have neither the time or the expertise to trace individual files - I do know that \Windows\System32\ntoskrnl.exe and \Windows\System32\hal*.dll need to be uncompressed (or compressed with xpress4k).

My tests were carried out using install.wim from a Windows 10 Enterprise LTSB source.

Using the unmodified install.wim Windows reported the following used space when right-clicking on the systemdrive and selecting Properties (in the running operating system - after the installation had completed and Windows 10 had booted) -
* 5.14 GB (--compact format=xpress4k)

Using the settings listed in this post in the WimBootCompress.ini file, used space -
* 4.99 GB (--compact format=xpress16k)
* 4.75 GB (--compact format=lzx)

For comparison purposes, the same Windows 10 source applied without the --compact switch -
* 6.45 GB

Regards,

Misty
synchronicity
Site Admin
Posts: 472
Joined: Sun Aug 02, 2015 10:31 pm

Re: Comparision of --compact compression formats

Post by synchronicity »

Thanks for doing these tests. I've included the file list in wimlib-1.8.3-BETA2, and it now knows to force XPRESS4K compression on all files in the list. Therefore, it should now be possible to apply a full Windows image in "compact mode" using XPRESS8K, XPRESS16K, or LZX compression, with no changes to the default WimBootCompress.ini file, and have Windows successfully boot. Feel free to test this if you have a chance.
JFX
Posts: 40
Joined: Tue Aug 18, 2015 3:59 pm

Re: Comparision of --compact compression formats

Post by JFX »

Hi synchronicity,

here are 3 more exclusions.

Code: Select all

\Windows\Boot\DVD\*
\Windows\Boot\EFI\*
\Windows\bootstat.dat
It seems they only need by Win8.x in compact mode.

BTW: XPRESS4K will be okay on these files for a Win10 compact setup, but not for Win7/8.x.
Maybe you can apply these files without compression, if the build number in the WIM XML is smaller than Win10.
synchronicity
Site Admin
Posts: 472
Joined: Sun Aug 02, 2015 10:31 pm

Re: Comparision of --compact compression formats

Post by synchronicity »

Thanks for the suggestions. I've added the three additional patterns and also made the fallback be "uncompressed" rather than XPRESS4K if the image does not appear to be Windows 10 (or later). I haven't yet had time to fully test the changes but feel free to try them out in wimlib-1.8.3-BETA3.
bliblubli
Posts: 88
Joined: Thu Dec 31, 2015 10:45 am

Re: Comparision of --compact compression formats

Post by bliblubli »

In the doc, it says we must add the wofadk.sys driver. Where do I get this file and how can I install it on windows 7?
JFX
Posts: 40
Joined: Tue Aug 18, 2015 3:59 pm

Re: Comparision of --compact compression formats

Post by JFX »

The wofadk driver is a part of the Windows 10 ADK.
You can also use my GetWaikTools and only download Win10 DISM.

If you want to use a Windows 7/8.x in compact mode you can add the wofadk.sys to \Windows\system32\drivers
and import it's respected registry key

Code: Select all

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\WofAdk]
"DependOnService"=hex(7):46,00,6c,00,74,00,4d,00,67,00,72,00,00,00,00,00
"Discription"="WofAdk Mini-Filter Driver"
"DisplayName"="WofAdk"
"ErrorControl"=dword:00000001
"Group"="FSFilter Compression"
"Start"=dword:00000000
"SupportedFeatures"=dword:00000003
"Type"=dword:00000002
"DebugFlags"=dword:00000000
"AutoAttach"=dword:00000001
"Tag"=dword:00000002
"ImagePath"=hex(2):5c,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,\
  74,00,5c,00,53,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,64,00,72,\
  00,69,00,76,00,65,00,72,00,73,00,5c,00,77,00,6f,00,66,00,61,00,64,00,6b,00,\
  2e,00,73,00,79,00,73,00,00,00

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\WofAdk\Instances]
"DefaultInstance"="WofAdk Instance"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\WofAdk\Instances\WofAdk Instance]
"Altitude"="40730"
"Flags"=dword:00000000

bliblubli
Posts: 88
Joined: Thu Dec 31, 2015 10:45 am

Re: Comparision of --compact compression formats

Post by bliblubli »

Hi JFX,
Nice to see you here. So to clarify the steps, after downloading the driver with your tool:
1) I add the wofadk driver to system32\drivers and add the .reg file.
2) I add the conf given above to \System32\WimBootCompress.ini. Is the .ini above in this thread the right one?
3) I make a capture with --snapshot with wimlib.
4) Then apply with --compact=LZX (is something special required on the system on which the apply command is run to get the system LZX compression?
5) Start the system normally.
6) Save some folder to a wim archive with whatever compression algorithm
7) Call DISM on startup through the task scheduler to mount the wim in the original location on startup.
Is it right?
JFX
Posts: 40
Joined: Tue Aug 18, 2015 3:59 pm

Re: Comparision of --compact compression formats

Post by JFX »

Hi bliblubli,

step 1 -5 sounds good. Don't really see the need for step 6 and 7.
bliblubli
Posts: 88
Joined: Thu Dec 31, 2015 10:45 am

Re: Comparision of --compact compression formats

Post by bliblubli »

for 6 and 7 it's useless because all files will be already compressed with lzx? Will they stay compressed when changed? Does it work for newly created files?
JFX
Posts: 40
Joined: Tue Aug 18, 2015 3:59 pm

Re: Comparision of --compact compression formats

Post by JFX »

bliblubli wrote:for 6 and 7 it's useless because all files will be already compressed with lzx?
Yes, also WIM mounting will slow down and files accessed will be extracted, so no space is saved.
bliblubli wrote:Will they stay compressed when changed? Does it work for newly created files?
Good question, I guess it will not compress or re-compress any file.
Post Reply