Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance

Mon Sep 6 09:53:00 GMT 2010

Hi,

Abstract: I prepared a patch that improves Cygwin Filesystem performance 
by x4 on Cygwin 1.7.5 (1.7.5 vanilla 530ms --> 1.7.5 patched 120ms). I 
ported the patch to 1.7.7, did tests, and found out that 1.7.7 had a 
very serious 9x (!) performance degradation from 1.7.5 (1.7.5 vanilla 
530ms --> 1.7.7 vanilla 3900ms --> 1.7.7 patched 3500ms), which does 
makes this patch useless until the performance degradation is fixed.

=======================================================================
My patch:
------------

As a cygwin user, I wanted to make cygwin faster, since Cygwin is my 
host for Win32 Windows development platform (make, cvs, grep etc).

I saw that simple file access operations were x10 to x50 times slower 
than on Windows.
And x10 to x50 slower than VMWare linux accessing the drive using HostFS 
(which allows Linux VM to access the Windows C: drive).

So I did some performace test on Cygwin 1.7.5, and found out the biggest 
bottleneck were:

- CreateFile()/CloseHandle() on syscalls that only need file node info 
retrival, not file contents retrival (such as stat()). This can be 
solved by calling Win32 that do not open the File handle itself, rather 
query by name, or opening the directory handle (which is MUCH faster).

- repetitive Win32 Kernel calls on a single syscall (stat() would call 
up to 5 CreateFile/CloseHandle pairs plus another additional 5 to 10 
Win32 APIs).
on stat() system calls, managed to improve the performance of ls approx. 
by 4 times. This can be solved by caching: first time in a syscall the 
API is called the result is stored, so the second time the info is 
needed, it is re-used.

- ACLs: Windows is not a secure system. And its ACL model is strange to 
say the least... Most cygwin users do not use the unix permissions 
system on cygwin to achieve security. All they want is that the unix 
tools will run on Windows. The ACL calls are over 50% (!) of the time 
spent during cygwin filesystem syscalls, including stat. For application 
such as chmod/chown/chgrp etc I added automatic enabling of ACLs. For 
all other applications: the ACLs can be enabled via envirnoment. This 
gives the cygwin end user choice: run everything at x2 the speed with no 
ACLs, or slow - but with ACL support.

- stat inode number and inode link count: getting the inode in Windows 
requires a File handle or Directory handle (not possible to get inode by 
file name). Very few applications actually need a REAL inode number. So 
using the 'get file info by name' apis are used (and of course there is 
an envirnoment if you really want real inode numbers).

- GetVolumeInfo: The C:\ drive does not tend to be changed every 
millisecond! Therefore no reason for every filesystem syscall to call 
it. Caching this further increased the performance.

- File security check: GetSecurityInfo() is implemented in Windows in 
usermode dll (Advapi32.dll). It calls at the end 
NtQuerySecurityObject(). So I implemented a faster version: 
zGetSecurityInfo() which does the same... just faster.

- symbolic link files: Opening a file and reading its contents is an 
expensive operation. All file cygwin operations must check whether the 
file is a symbolic link or not, which is done by opening the file and 
reading its contents and checking whether it has the symlink magic at 
the beginning of this. Since symbolic link must be at least 8 bytes long 
(to include the symlink magic header), and cannot be longer than 
MAX_PATH+magic size  - there is no reason to read the file contents that 
is smaller or bigger than this. Since MOST of the files are outside of 
this range, we save a LOT of CreateFile/ReadFile/CloseHanlde calls on 
every single file operation!

I did all the above work on cygwin 1.7.5. I did a simple test: 'ls 
/bin', where bin holds 3500 files.
Cygwin 1.7.5 vanilla took 530 ms. After all my above improvements, the 
patched version did this in 120 ms. Thats more than 4x improvement! I 
then checked all other applications (grep, make, rsync, cvs, perl code 
etc...): they all had this same 4x improvement!

I think this is a very important patch that should find its way into the 
Cygwin source code, since it helps to dramatically reduce the x10-x50 
filesystem performance gap between Cygwin and native Win32 code.

And once part of standard cygwin, I have many more ideas how the 
filesystem performance can be further improved.

==============================================================================
The checks:

The check is did was 'ls > /dev/null' in /bin directory, which contain 
about 3500 files.
I did all that for version 1.7.5.
When i wanted to merge my changes to version 1.7.7 i noticed a major 
performance degradation.
with 1.7.5 'ls' in /bin took 530ms (with original version), and with 
1.7.7 it took almost 4 seconds!
You can see all the details below.
So, I think someone should have a look, and find out what is causing the 
performance degradation.
I also want to merge my patch to cygwin CVS, so everybody can enjoy it 
(me, and about 10 more people are using a patched cygwin1.dll for about 
4 month without any problems).

/bin$ ls /bin | wc -l
3580

The Win32 comparison (... yes: I know cmd.exe's version of ls collects 
less info)
/bin$ time cmd /c 'dir > nul:'
real    0m0.113s
user    0m0.015s
sys     0m0.015s

_Without my patch:_
/bin$ uname -a
CYGWIN_NT-5.1 yoni 1.7.5(0.225/5/3) 2010-04-12 19:07 i686 Cygwin
/bin$ time ls > /dev/null
real *0m0.530s*
user    0m0.140s
sys     0m0.421s

~$ uname -a
CYGWIN_NT-5.1 yoni 1.7.7(0.230/5/3) 2010-08-31 09:58 i686 Cygwin
/bin$ time ls > /dev/null
real *0m3.949s*
user    0m0.171s
sys     0m0.546s

_With my patch:_
$ uname -a
CYGWIN_NT-5.1 yoni 1.7.5(0.228/5/3) 2010-07-18 14:53 i686 Cygwin
~$ cd /bin/
/bin$ time ls > /dev/null
real *0m0.123s*
user    0m0.030s
sys     0m0.124s

/bin$ uname -a
CYGWIN_NT-5.1 yoni 1.7.7(0.230/5/3) 2010-09-01 17:14 i686 Cygwin
/bin$ time ls > /dev/null
real *0m3.575s*
user    0m0.108s
sys     0m0.374s

Best regards,
Yoni.

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: cygwin_1_7_5.patch
URL: <http://cygwin.com/pipermail/cygwin-patches/attachments/20100906/a99c3a92/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: cygwin_1_7_7.patch
URL: <http://cygwin.com/pipermail/cygwin-patches/attachments/20100906/a99c3a92/attachment-0001.ksh>