on the fly hardlinking

[bliblubli]

Where should I have a look if I want to port the old Hardlink functionality back in the latest version ? I would be happy if you can help in this task (like what has changed since that would require changes in the old code, where there some known bugs?)

[synchronicity]

I assume you're taking about creating hard links where there weren't before, since actual hardlinks are already fully supported. Can you explain why you want to do this? How will you deal with programs that do the wrong thing when their files are suddenly hard linked?

[bliblubli]

My use case is that I sometime work on a project, that takes big data files form librairies, but on another computer. I have to make copies of those librairies and because of project pressure, I don't have time to relocate all the path when I'm back to the original librairie. So I have sometime a lot of duplicates and running duplicate cleaner before doing backups is sometime really slow. If I don't clean those duplicates, with wimlib, it will fit on the backup HDD, but will not be able to restore because all those duplicates will be real ones.

Regarding programs, I already harlinked thousands of files and never had a problem. Can you explain me or give an example of problems caused by hardlinking program dlls or such? Anyway, most of the time, I'll use it for images, 3D models, etc...

[synchronicity]

As I said previously, changing a file through one name changes the file as visible through all names. Creating hardlinks on your filesystem as a form of "deduplication" has a huge potential to break things if done systemwide. What if two different programs create a file with identical contents? What if something creates a file that is identical to an operating system file? What if the files had different timestamps and something relies on the timestamps? What if I create two user profiles and they have the same default configuration files? Sounds like a great way to corrupt your system and create security holes.

This feature will not be implemented. If you have files that you want to avoid duplicating then you should store them in a central location and create a symlink to it where needed.

[synchronicity]

By the way: a comparable feature that would actually make sense would be to add support for reflink or "clone" functionality, e.g. as provided by btrfs on Linux. With that you can deduplicate data without breaking things by combining files.

[bliblubli]

Hi Synchronicity,
Wasn't asking you to implement it again, just for advice on porting the code you did on previous versions. By the way, I'm also ok with symlinks if they solve the problem. I also think it's not more dangerous using a beta software than using an option. If you do use one of them, you have to know what you do. And Hardlinks will not be created between users or program/system files, because they will not have the same security descriptors? Of course I'll not base deduplication on the content alone.

[synchronicity]

I can't really help you with the extraction code because it's been heavily rewritten since the time it supported "auto-hardlinking". And for the reasons I've given, I don't think it makes sense to add this feature back. It would have very few, if any, legitimate use cases and would be prone to be misused.

Of course, wimlib is free software so you're free to do whatever you want with the code on your own.

[bliblubli]

Ok, I can understand. Your improvements to this tool are already awesome I'll try to get this part ported. Anyway, their are alternatives like link shell extension which can also do dedup on the fly.