rsync and ls -lR slow for directories with many files

[L A Walsh]

On 2020/01/08 08:43, Frank-Ulrich Sommer wrote:
> but rsync did not get faster.
>
> I'm sorry to admit that the ultimate solution does not use Cygwin any more. I'm now using a Windows share and connect to that share from my Linux server with cifs and autofs. rsync then runs on the linux machine and accesses that share (due to --whole-file this should not cause problems).
>
> rsync without any changed files directly after booting both systems (both caches are empty) now takes 91s instead of 42m.
>   
---
    I won't go into the many reasons I know of (which are very 
incomplete compared
to those who actually created cygwin), but in situations like you describe,
I almost never use rsync unless I don't care about time and desperately 
need a
feature it has.  Instead I'll find it's faster to create an uncompressed tar
on windows, copy it to linux and expand it there to work locally with it.

    Any compression/signing/encrypting of data will slow down data 
transfer on
my home network where max CIFS transfer speeds are in the 300-700MB/s range.
>
>
>
>
> Am 5. Januar 2020 22:22:35 MEZ schrieb Stephen John Smoogen <smooge@gmail.com>:
>   
>> On Sat, 4 Jan 2020 at 17:16, <muell@plutonium24.de> wrote:
>>     
>>> I am running rsync on a small linux server to synchronize files in
>>>       
>> one directory and its subdirectories from Windows (using sshd from
>> Cygwin) to this server for backup purposes. The directory contains
>> almost 1 TB of images and videos in about 160k files on a slow disk
>> (Seagate Archive 8TB with SMR) with NTFS.
>>
>> I am not sure if the Linux box has the slow disk or the Windows box
>> has the slow disk.
>>
>>     
>>> Even if there are no changes and whith whole file transfers rsync
>>>       
>> takes about 45 minutes to come to this conclusion.
>>     
>>> I am using the following command line on the linux server:
>>>
>>> rsync -avx --stats --whole-file --no-perms --no-owner --no-group
>>>       
>> <user>@<server>:<source directory> <local destination directory>
>>     
>>> As rsync was only transferring a small number of bytes and gave no
>>>       
>> clue to the cause for being so slow and as rsync should only need
>> filenames, dates and sizes I did a "ls -lR|wc" on both systems. On the
>> linux server this took about 1 minute (only slightly faster magnetic
>> disk, empty read cache at start) and doing the same on cygwin took
>> almost as long as rsync (over 40 minutes). Using Windows Explorer
>> (after a reboot to guarantee that the cache is empty) to get the total
>> number of files and the total size took only a few seconds. Reading all
>> file sizes with Treesize also took less than one minute. As ls -lR
>> needs the same information I would have expected it to take the same
>> time.
>>
>>
>> I would add a bunch of verbose to the rsync to see what it is doing.
>> (I don't recommend sending that to the list as it will be a lot of
>> data.. but maybe an excerpt) I am expecting it is spending a lot of
>> time getting the metadata off of one of the disks and mapping it to
>> Unix permissions then comparing if those items are the same on the
>> other side. Each one of those is going to be a separate action which
>> on a slow drive may be a spinup/get-data/spindown cycle to make it
>> even slower.
>>
>> I would then check to see if perms and metadata on that directory
>> 'look sane' (this is highly dependent on your environment.. if you
>> have an AD server giving out perms it will look different from other
>> things.) If the lookups for mapping metadata permissions is having to
>> ping an AD server or some sort of other network lookup that is going
>> to also slow down things.
>>
>> Sorry I don't have any 'fixes'. I have always found large rsync
>> between Windows and Unix to be slow.
>>
>>     
>>> Runnin "ls -lR" a second time on Cygwin is fast as lightning as it
>>>       
>> only takes less than 30s.
>>     
>>> Is there any way to get ls -lR or better rsync as fast as listing the
>>>       
>> directory with Windows tools?
>>     
>>> Frank
>>>
>>> --
>>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail
>>>       
>> gesendet.
>>     
>>> --
>>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail
>>>       
>> gesendet.
>>     
>>> --
>>> Problem reports:       http://cygwin.com/problems.html
>>> FAQ:                   http://cygwin.com/faq/
>>> Documentation:         http://cygwin.com/docs.html
>>> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>>>
>>>       
>> -- 
>> Stephen J Smoogen.
>>
>> --
>> Problem reports:       http://cygwin.com/problems.html
>> FAQ:                   http://cygwin.com/faq/
>> Documentation:         http://cygwin.com/docs.html
>> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>>     
>
>   

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple