ARM64 Support

[synchronicity]

Hi,

I wanted to mention that I've made a number of improvements very recently:

• Eliminated the dependencies on libxml2 and winpthreads, so wimlib is now easier to build for Windows.
• Added dllexport to the declarations in wimlib.h to avoid warnings when building for Windows with clang.
• Updated README.WINDOWS to show how to use MSYS2 (with the MINGW64, CLANG64, MINGW32, CLANG32, or CLANGARM64 environment). This replaces the previous instructions that used Cygwin and only had mingw64 and mingw32.
• Improved the tools/make-windows-release script.

That being said, I haven't yet actually tested CLANGARM64. It seems that build can only be done on a Windows ARM64 system. I will try to find a way to test it, but if anyone else wants to give it a try please go ahead. It *should* be as simple as following the new "building from source" instructions on a Windows ARM64 system, choosing "MSYS2 CLANGARM64" when you get to the step where you choose a MSYS2 environment.

All these changes can be found on the master branch in git (which will eventually become v1.14.0).

[JFX]

Thanks, compiling windows release of wimlib is now much easier.

To cross-compile Windows ARM64 version extract llvm-mingw-20230320-msvcrt-x86_64.zip
to the root of msys64 folder as llvm-mingw.

In MSYS2 CLANGARM64 enter:

Code: Select all

export PATH=/llvm-mingw/bin:${PATH}

before running:

Code: Select all

tools/make-windows-release --install-msys2-packages

There is an post install error for one of the packages, but it will compile without problems.

[synchronicity]

Thanks! It's too bad that CLANGARM64 doesn't work out of the box, but downloading that separate prebuilt toolchain will work for now.

I've updated the windows-build.sh script (renamed from make-windows-release) to handle downloading the prebuilt toolchain when using CLANGARM64 on x86_64.

So, the new command is:

Code: Select all

tools/windows-build.sh --install-prerequisites

It works in most of the MSYS2 environments: MINGW64, CLANG64, MINGW32, CLANG32, CLANGARM64.

I've also released wimlib v1.14.0-BETA2 which includes (untested) Windows ARM64 binaries. See the downloads page: https://wimlib.net/downloads/index.html

[joveler]

Hmm, I have no clue why this error disappeared, but it compiles fine now.
A quick test in QEMU did also gone well. Will test in next week on real ARM64 hardware.

Finally, I figured out why sometimes wimlib-imagex.exe is built or not built in cross-compiling.
We now have an official Windows ARM64 wimlib build, but documenting this issue would help other developers.

Workaround 1
When you have a functional toolchain in the MSYS2 shell, append the dummy --libdir argument to configure.
When invoking wimlib configure, append --libdir="${SRCDIR}". Any empty directory would do the same.

Code: Select all

$ cd /c/build/wimlib.1.14.3
$ ./configure --libdir=/c/build/wimlib-1.14.3 --host=aarch64-w64-mingw32 --disable-static

Workaround 2
Use an MSYS2 shell that does not have any functional toolchain.
I guess this is the reason why wimlib 1.14.x's official Windows arm64 build works.

Cause
The main culprit was libtool, which was appending a system library path (e.g. -LC:/msys64/ucrt64/lib) which it must not do.
This argument came from libtool libdir value, which denotes the path where .dll/.a/.la files would be located when installed.
That caused x64 libmsvcrt.a, libmingw32.a from the native toolchain to be linked to arm64 wimlib-imagex.exe, sabotaging the link process.

In Linux, the path to find libraries is hard-coded to binary itself. That is why libtool appends --libdir path.
But Windows does not have such 'fixed rpath' notation. Also, we do not 'make install' the wimlib in our case. We collect the .dll and .exe ourselves.
So libdir value is not that necessary. We can safely neutralize that value.

Reference
- Huge thanks to llvm-mingw devs who helped to debug this issue : https://github.com/mstorsjo/llvm-mingw/ ... 1707621414
- My own size-optimized wimlib build script : https://github.com/ied206/ManagedWimLib ... v1.14.x.sh
* It produces libwim-15.dll and statically built wimlib-imagex.exe.
* If libwim-15.dll is not built with a libtool error message, install a native MSYS2 toolchain and retry.

[joveler]

Hi,

I wanted to mention that I've made a number of improvements very recently:

• Eliminated the dependencies on libxml2 and winpthreads, so wimlib is now easier to build for Windows.
• Added dllexport to the declarations in wimlib.h to avoid warnings when building for Windows with clang.
• Updated README.WINDOWS to show how to use MSYS2 (with the MINGW64, CLANG64, MINGW32, CLANG32, or CLANGARM64 environment). This replaces the previous instructions that used Cygwin and only had mingw64 and mingw32.
• Improved the tools/make-windows-release script.

That being said, I haven't yet actually tested CLANGARM64. It seems that build can only be done on a Windows ARM64 system. I will try to find a way to test it, but if anyone else wants to give it a try please go ahead. It *should* be as simple as following the new "building from source" instructions on a Windows ARM64 system, choosing "MSYS2 CLANGARM64" when you get to the step where you choose a MSYS2 environment.

All these changes can be found on the master branch in git (which will eventually become v1.14.0).

Thanks for making wimlib clang compatible!
It would have been much easier if mingw-w64 natively supported ARM64, but for now, we have to rely on LLVM/Clang.

I have personally tested the MSYS2 CLANGARM64 shell on a Windows ARM64 dev machine.
This shell is a hybrid of x64 and arm64 binaries, so it requires ARM64 devices to run.
I have found using the llvm-mingw toolchain on an x64 machine is much easier than using the MSYS2 CLANGARM64 shell, just like how you and JFX concluded.

Official wimlib Windows ARM64 binary (as of v1.14.3) works well on my Win ARM64 dev machine, too.

[zipmagic]

Is there a way to add these flags to the compiler and linker:

-static -static-libgcc -static-libstdc++

To ensure there's no dependencies on the built DLL?

I've seen dependency problems happen on Windows 8.1, for instance.

[synchronicity]

What dependency problems are you referring to, specifically? The ARM64 libwim-15.dll has the same dependencies as the x86_64 one, I believe. It only depends on standard Windows DLLs (ntdll, msvcrt, advapi32, user32, kernel32).

[zipmagic]

So rebuilding it with GCC:

32-bit Windows 8.1:

The program can't start because libgcc_s_dw2-1.dll is missing from your computer. Try reinstalling the program to fix this problem.

64-bit Windows 8.1:

Works fine.

Rebuilding it with CLANG:

32-bit Windows 8.1:

The program can't start because api-ms-win-crt-convert-l1-1-0.dll is missing from your computer. Try reinstalling the program to fix this problem.

64-bit Windows 8.1:

The program can't start because api-ms-win-crt-convert-l1-1-0.dll is missing from your computer. Try reinstalling the program to fix this problem.

For good measure, I also tested both with the last version of WIMLIB used by ZIPmagic and it worked perfectly fine. Since the file lacks version information I can't tell what version it is, but it is probably in the 13.x branch, I had not rebuilt it myself either, and it was probably before the simplified build mechanism was implemented. The file last modified date is December 19th, 2021 - it could be even older than that of course, but certainly not newer.

[zipmagic]

What dependency problems are you referring to, specifically? The ARM64 libwim-15.dll has the same dependencies as the x86_64 one, I believe. It only depends on standard Windows DLLs (ntdll, msvcrt, advapi32, user32, kernel32).

Also to clarify I tested the 32 bit (x86) and 64 bit (amd64) non-aarch64 versions of the DLL that I rebuilt on my own using the latest instructions.

So these dependencies apply to those platforms as noted in my testing results; aarch64 shall have to come a bit after the dust settles with these.

[synchronicity]

Hi,

It turned out that the windows-build.sh script had a bug that made it not use the -static-libgcc compiler flag in the MINGW32 and MINGW64 MSYS2 environments as intended, causing the DLL to depend on libgcc. I've pushed out a commit that fixed this. This did not affect the official i686 and x86_64 binaries from https://wimlib.net/downloads/index.html, as I've still been building those on Linux, not in MSYS2.

I don't have an explanation for why you saw api-ms-win-crt-convert-l1-1-0.dll being required. I'm not seeing that.

I tested the builds on Windows Server 2022, not on Windows 8.1 as you did, though. Maybe that makes a difference.

Please note that it's possible to clone the wimlib repository on GitHub, and then run the GitHub Actions workflow, which builds all the Windows builds reproducibly using the windows-latest GitHub Actions runner which currently uses Windows Server 2022. That can be used as a more-reproducible alternative to doing the build locally.