wimlib

Ok, so we did a series of tests. Most went smooth, but when time came to again backup the 3TB HDD on an 8TB one (on windows 7 pro sp1), the same problem appeared. Filesize was increasing by small chunks.
Can it be that their is a limit for the allocation or file size and when wimlib/windows api call has to much data, it falls back to another method because it can't allocate 3TB at once?

The 3TB backup finished this time with default options (no option given on the command line)
- speed was 54MB/s mean (versus 130MB/sec in the first 30GB). So there is a slowdown on 7 x64 sp1, but still better than server 2003 which won't finish after many days.
- Memory usage after the backup finished was 4.7GB (versus 1.2GB before), maybe a leak?

I really have no idea. I don't believe wimlib isn't doing anything unusual with the output WIM file. It opens it for writing and writes to it mostly sequentially. Sometimes it seeks slightly backwards and overwrites bytes it recently wrote. I don't think it's an unusual access pattern at all, so if Windows has a problem handling it efficiently, it is probably not the application's fault. And if this is really a problem that only appears on 3 TB+ drives/filesystems, I don't currently have the hardware to reproduce it. Have you tried running other archiving programs to see if they encounter the same problem?

Can you clarify what you mean by the memory usage? Are you saying you observed it at 1.2 GB at one point while writing the WIM file,after the initial directory tree scan, and then you checked later and was 4.7 GB, in the same process? That's surprising to me, because wimlib performs almost no memory allocations during the main read-compress-and-write code which consumes most of the time; almost all needed memory has been preallocated.

Regarding the memory usage, the overall memory usage (displayed in the task manager) is of 1.2GB on startup. It has nothing to do with wimlib (not started at that point). After Wimlib did backup the 3TB HDD and the process is not there anymore (job done) the memory usage (again nothing to do with wimlib as it's not running anymore, it's the total memory usage reported by the task manager) is of 4.7GB. So it's like 3.5GB of memory leak. Part of it is visible in one of the svchost.exe instances which uses 600MB after backup instead of 88MB on startup. It still doesn't explain the 3.5GB difference.
Even if those problem are on Microsoft side, it may be worth reporting them upstream? At least the memory leak certainly appear before 3TB, even if not as high as in our case.
Regarding the fragmentation and attached slowdown, it's really only on server 2003. Win 7, even when increasing file allocation progressively, doesn't fragment the archive.
I'm not sure how wimlib allocates the space, but to prevent progressive allocation and OS-Version-dependent results, maybe this technique http://stackoverflow.com/questions/4552 ... on-windows could help. It allocates one big chunk in one go.
We do our best to help, we know windows can be improved also. Tell us if you need more information.
Regards

Not sure how related that is, but I also tried to backup a 7TB partition with wimlib this week and got the a pretty heavy slowdown after 4TB. CPU usage from wimlib was at 2%... After 5.5TB the speed resumed and it finished pretty quickly.

Would it be possible to have a kind of log to a cvs file with:
filename;backup speed;wimsize
It would allow to see which file and/or which archive size trigger the slowdown.