SHA-1 optimization

Comments, questions, bug reports, etc.
Post Reply
JFX
Posts: 40
Joined: Tue Aug 18, 2015 3:59 pm

SHA-1 optimization

Post by JFX »

While searching for a hardware optimized SHA-1 algorithm for the ARM64 port of wimlib,
I noticed that the implementation of 7-Zip is twice as fast as the current accelerated one of wimlib (for x86_64).

Do you consider implementing SHA-1 code of 7-Zip into wimlib?
synchronicity
Site Admin
Posts: 472
Joined: Sun Aug 02, 2015 10:31 pm

Re: SHA-1 optimization

Post by synchronicity »

I assume you're talking about the Windows binaries? Currently wimlib uses its own SHA-1 code (C or SSSE3-accelerated) on Windows, but uses OpenSSL on other platforms. Probably the performance difference you are seeing comes from the x86 SHA extensions not being used in the Windows binaries. ARM64 Windows binaries must have a similar problem too, as they don't use the ARMv8 Crypto Extensions.

The question is should wimlib add more SHA-1 implementations itself, or should it just always use a library that has fast SHA-1 implementations already, and if so which one.

I don't remember why I didn't just make the Windows binaries statically link to OpenSSL. That might be the best option.
JFX
Posts: 40
Joined: Tue Aug 18, 2015 3:59 pm

Re: SHA-1 optimization

Post by JFX »

synchronicity wrote: Thu Mar 09, 2023 5:47 pm I don't remember why I didn't just make the Windows binaries statically link to OpenSSL. That might be the best option.
That would be great, I didn't managed to compile wimlib with libcryto.
wimlib's current sha1-ssse3.asm is very slow compare to libcrypto-1_1-x64.dll (from VMware Workstation v17) witch is equal to 7zip speed wise.
synchronicity
Site Admin
Posts: 472
Joined: Sun Aug 02, 2015 10:31 pm

Re: SHA-1 optimization

Post by synchronicity »

Statically linking OpenSSL didn't work very well.

I ended up just adding my own SHA-1 code.

It's on the master branch now. I'll upload binaries (probably v1.14.0-BETA1) in a bit; I need to do some more testing and look into some other things that came up recently.
JFX
Posts: 40
Joined: Tue Aug 18, 2015 3:59 pm

Re: SHA-1 optimization

Post by JFX »

Thank you, this makes quite a difference on the windows ARM64 version.
from around 550 MB/s up to 1830 MB/s!
x86_64 doubles in speed and is now similar to 7zip/libcrypto.

Took me a while to notice that you swap the sha1_final() parameters.

For the just released LLVM 16, I had to make a change to sha1.c to compile ARM64 version.

Code: Select all

#if (__clang_major__ >= 16)
	__attribute__((target("crypto,sha2")))
#else
	__attribute__((target("crypto")))
#endif
synchronicity
Site Admin
Posts: 472
Joined: Sun Aug 02, 2015 10:31 pm

Re: SHA-1 optimization

Post by synchronicity »

Thanks for testing and for reporting the build error with clang 16! Clang is potentially going to be fixed to treat the "crypto" target like gcc does, but in the mean time I've pushed out a fix to use "sha2" instead.
Post Reply