Tuning ls, ls -l, find ., find . -ls performance for very large dirs (60000+ files/dir)

[Sam Edge]

On 08/04/2025 19:05, Mark Liam Brown via Cygwin wrote:
> Greetings!
> 
> Are there tuning variables to improve ls, ls -l, find ., find . -ls
> performance for very large dirs?
> 
> If we have a SMB dir with 60000+ entries a simple ls -l can take MANY
> minutes (22+mins), while cmd.exe dir just floods the terminal with
> results immediately.
> 
> Mark

By default, to simulate POSIX using local & remote MS-DOS/Windows-type 
mounts, Cygwin's readdir(), stat() and other filesystem functions have 
to jump through a lot of hoops. For a start, Windows 'Unicode' 
characters have to be converted to the locale, usually UTF-8, being used 
by Cygwin. Then either the Windows (NTFS or SMB) ACLs have to be mapped 
to POSIX file permissions & ACLs and/or the files' extensions have to be 
checked and sometimes the file opened to read the first few bytes to 
determine executable permissions. Windows local & remote SIDs have to be 
converted to POSIX UIDs & GIDs. And Cygwin then has to convert to 
convert the UID/GID passed back from the clients like 'ls' to Windows 
user/group names for display.

As discussed by other replies, sixty thousand entries in a single 
directory is ridiculously poor practice and the best thing would be to 
redesign whatever is creating these directories & their contents.

But if this is not possible, there are mount options that can be used in 
/etc/fstab to reduce the amount of overhead. Try using 'noacl' & 
'notexec' in /etc/fstab and then access files from Cygwin using the 
mount point rather than the UNC path. This disables the advanced ACL/SID 
mapping and reports all files as -rw-r--r--, directories drwxr-x-rx with 
UID of the calling user and GID None. (Of course Windows & SMB will 
still restrict access based on the actual permissions available to the 
caller's access tokens.) If sort order isn't important, use 
'--sort=none' with ls.

//h/s /mnt/h/s smbfs noacl,notexec,nouser 0 0

I doubt this is going to get a 60k entry remote mount directory listing 
down to a reasonable time but it should definitely help.

See https://www.cygwin.com/cygwin-ug-net/using.html#mount-table.

-- 
Sam Edge
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0x8AC2CEBF54528E30.asc
Type: application/pgp-keys
Size: 648 bytes
Desc: OpenPGP public key
URL: <https://cygwin.com/pipermail/cygwin/attachments/20250409/46f2e6da/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 236 bytes
Desc: OpenPGP digital signature
URL: <https://cygwin.com/pipermail/cygwin/attachments/20250409/46f2e6da/attachment.sig>