Wimlib for re-compression

Comments, questions, bug reports, etc.
Post Reply
GonzaloM
Posts: 2
Joined: Mon Apr 16, 2018 10:43 pm

Wimlib for re-compression

Post by GonzaloM »

Hello people! How are you? First time posting here. I wanted to ask you a question regarding wimlib.
You see, a few fellas and I are working on an open source project called Fairytale. It's an archiver with advanced functionality. We are working on several fronts to make sure it really is state-of-the-art software. One of them is lossless re-compression. We have working code for the deflate algorithm, which is the most common, also lzw, bz2 and a few other.
Now, what I wanted to ask you is how likely is for your code on wimlib to be able to do the same for the algorithms it implements. We would like to use it with your permission to re-compress some types of files. I see from the website that wimlib and Microsoft's software produce compatible streams, i.e. one implementation is able to decompress data from the other. But they don't produce byte-identical output. So, would we be able to reproduce say, an lzx stream from a cab file, using wimlib as a tool, in your opinion? There are some programs that use one library to recompress deflate streams created by othes libraries, with a relatively small overhead that serve to reconstruct them without loss. I don't know about lzx and such, though... Would it be possible, in theory? If so, that would be really great. There is a lot of data compresed by Microsoft algorithms, especially lzx, in the form of installers. Well, you know that better than me :) And below that, there's even more information we are already capable of recompress.

Thank you very much in advance for your help! I hope I explained myself correctly. Please comment on this if you need further details.
Have a great day! - Gonzalo.
synchronicity
Site Admin
Posts: 472
Joined: Sun Aug 02, 2015 10:31 pm

Re: Wimlib for re-compression

Post by synchronicity »

wimlib implements the XPRESS (Huffman variant), LZX, and LZMS compression algorithms. All these compressors provide slightly better compression ratios than their Microsoft equivalents. However, I've only implemented the chunk-based compression that is needed for the WIM file format; there is no "sliding window" support. So the LZX compressor would require modifications to be usable for CAB recompression, since (I think) CAB files use LZX in "sliding window" mode. Though of course, if you actually want to implement LZX CAB recompression, it would be much easier to start with wimlib's LZX compressor than to start from scratch.
GonzaloM
Posts: 2
Joined: Mon Apr 16, 2018 10:43 pm

Re: Wimlib for re-compression

Post by GonzaloM »

Thank you Eric for the insights. That's very helpful.
Post Reply