Decrease fragmentation when using ntfs-3G

[bliblubli]

Hi,
When using Ntfs-3G, the partition is heavily fragmented. On huge partition, it leads to broken file system because windows doesn't support more than x (undocumented) fragments per filesystem. Ntfs-3G has an function to preallocate, which should help get contiguous space: ntfsfallocate. It's availaible in all distrib supporting ntfs-3g 2014, so it's pretty safe to implement, without breaking too much compatibility. http://manpages.ubuntu.com/manpages/wil ... ate.8.html

[synchronicity]

Yes, this has been reported before and I am planning to investigate it sometime. The current code is actually already intended to preallocate the space for each file, but perhaps it doesn't work as intended.

[synchronicity]

Hi,

Do you know whether the fragmented files you observed were compressed with NTFS compression?

[bliblubli]

They were not compressed. NTFS-Compression always lead to fragmentation even on windows, so I don't use.

[synchronicity]

Okay. Is there any specific way I can reproduce this problem? When I tried, there was little fragmentation.

[bliblubli]

I used Ubuntu with self-compiled ntfs3g 2016.2.2. Then I used "wimcapture /dev/sda2 ./win7.wim". Then applied it with "wimapply ./win7.wim /dev/sda2". After starting this restored windows, I had 59% fragmentation reported by the windows defragmenter. The same image applied from windows PE (with wimlib also) has 1% fragmentation. The windows partition I saved has 120GB of source codes and programs, with some big binaries and lot of small source files. The partition was always formatted before applying the image. Let me know if you need more informations.

[synchronicity]

I did some tests with applying a Windows 7 image to an NTFS volume (~40000 files and 5 GB of data; volume size ~15 GB).

In the case of nondirectory files, I found that wimlib is correctly requesting that NTFS-3G preallocate clusters. Before writing anything to a file stream, wimlib calls ntfs_attr_truncate_solid() to preallocate all the needed clusters.

Directories are not being preallocated. It might be possible to do so by preallocating space for the INDEX_ALLOCATION attribute. This would need to be tested. However, that isn't the main problem that causes the fragmentation.

I believe the main problem which causes the fragmentation is that after every allocation request, NTFS-3G advances its allocation starting point forward by 4096 clusters (16 MB). This is a large amount, and it means that when creating many files, the allocator will fill the whole volume with extents every 16 MB or less. Then, when it comes time to create a large file, it will necessarily be fragmented, since no extent can be more than 16 MB.

There might be a "hack" possible to override the NTFS-3G behavior. For example, I think an application could change "data1_zone_pos", the allocation starting point, after each request. However, I would prefer to have NTFS-3G itself updated to have better behavior by default. This would take some more research.

I should also note that Windows might be overreporting the amount of fragmentation. In one of my tests, Windows reported the volume was 19% fragmented, when in fact only 39 of 33197 nondirectory files even had multiple extents, and the average number of extents per file was 1.001848, i.e. only 0.1848% worse than optimal. I think the reason Windows reported more fragmentation is that the fragmented files tended to be large, so the "amount of fragmented data" was significant, even if the number of extra extents was not.

[bliblubli]

Thanks for the investigation. Does ntfsfallocate have the same 16MB "buffer" problem (or is ntfs_attr_truncate_solid() the way to call ntfsfallocate)?
If it's really a failed design, I hope it get fixed soon upstream.

[synchronicity]

It's the other way around. ntfsfallocate is another example of an application which uses libntfs-3g and the underlying ntfs_attr_truncate_solid() function and cluster allocator. Technically, I think ntfsfallocate wouldn't encounter this fragmentation problem because it only allows processing one file a time, with the volume being remounted each time. But for the same reason it cannot be used by applications such as wimlib that must process many files.

Filesystem block / cluster allocation is definitely not an easy problem, but I am going to look at how other filesystems do it. I suppose that what might make sense would be to have the gap after each allocation be proportional to the allocation size, rather than be a fixed size.

[bliblubli]

Thanks for the clarification Synchronicity Maybe a quick solution in the meantime would be to first write the bigest files and then the smallest, they would better fit in the gaps without fragmentation and the big files would then be contiguous. Maybe as an option to only use it when beneficial.