Huge slowdown when archive size increase

Comments, questions, bug reports, etc.
Gehard
Posts: 18
Joined: Wed Mar 23, 2016 7:52 am

Re: Huge slowdown when archive size increase

Post by Gehard »

Ok, so we did a series of tests. Most went smooth, but when time came to again backup the 3TB HDD on an 8TB one (on windows 7 pro sp1), the same problem appeared. Filesize was increasing by small chunks.
Can it be that their is a limit for the allocation or file size and when wimlib/windows api call has to much data, it falls back to another method because it can't allocate 3TB at once?
Gehard
Posts: 18
Joined: Wed Mar 23, 2016 7:52 am

Re: Huge slowdown when archive size increase

Post by Gehard »

The 3TB backup finished this time with default options (no option given on the command line)
- speed was 54MB/s mean (versus 130MB/sec in the first 30GB). So there is a slowdown on 7 x64 sp1, but still better than server 2003 which won't finish after many days.
- Memory usage after the backup finished was 4.7GB (versus 1.2GB before), maybe a leak?
synchronicity
Site Admin
Posts: 472
Joined: Sun Aug 02, 2015 10:31 pm

Re: Huge slowdown when archive size increase

Post by synchronicity »

I really have no idea. I don't believe wimlib isn't doing anything unusual with the output WIM file. It opens it for writing and writes to it mostly sequentially. Sometimes it seeks slightly backwards and overwrites bytes it recently wrote. I don't think it's an unusual access pattern at all, so if Windows has a problem handling it efficiently, it is probably not the application's fault. And if this is really a problem that only appears on 3 TB+ drives/filesystems, I don't currently have the hardware to reproduce it. Have you tried running other archiving programs to see if they encounter the same problem?

Can you clarify what you mean by the memory usage? Are you saying you observed it at 1.2 GB at one point while writing the WIM file,after the initial directory tree scan, and then you checked later and was 4.7 GB, in the same process? That's surprising to me, because wimlib performs almost no memory allocations during the main read-compress-and-write code which consumes most of the time; almost all needed memory has been preallocated.
Gehard
Posts: 18
Joined: Wed Mar 23, 2016 7:52 am

Re: Huge slowdown when archive size increase

Post by Gehard »

Regarding the memory usage, the overall memory usage (displayed in the task manager) is of 1.2GB on startup. It has nothing to do with wimlib (not started at that point). After Wimlib did backup the 3TB HDD and the process is not there anymore (job done) the memory usage (again nothing to do with wimlib as it's not running anymore, it's the total memory usage reported by the task manager) is of 4.7GB. So it's like 3.5GB of memory leak. Part of it is visible in one of the svchost.exe instances which uses 600MB after backup instead of 88MB on startup. It still doesn't explain the 3.5GB difference.
Even if those problem are on Microsoft side, it may be worth reporting them upstream? At least the memory leak certainly appear before 3TB, even if not as high as in our case.
Regarding the fragmentation and attached slowdown, it's really only on server 2003. Win 7, even when increasing file allocation progressively, doesn't fragment the archive.
I'm not sure how wimlib allocates the space, but to prevent progressive allocation and OS-Version-dependent results, maybe this technique http://stackoverflow.com/questions/4552 ... on-windows could help. It allocates one big chunk in one go.
We do our best to help, we know windows can be improved also. Tell us if you need more information.
Regards
bliblubli
Posts: 88
Joined: Thu Dec 31, 2015 10:45 am

Re: Huge slowdown when archive size increase

Post by bliblubli »

Not sure how related that is, but I also tried to backup a 7TB partition with wimlib this week and got the a pretty heavy slowdown after 4TB. CPU usage from wimlib was at 2%... After 5.5TB the speed resumed and it finished pretty quickly.

Would it be possible to have a kind of log to a cvs file with:
filename;backup speed;wimsize
It would allow to see which file and/or which archive size trigger the slowdown.
Post Reply