SLOW ls(1) - cygwin dir lookups with WinNT async requests?

[Aurélien Couderc]

On Mon, Jul 14, 2025 at 10:05 PM Brian Inglis via Cygwin
<cygwin@cygwin.com> wrote:
>
> On 2025-07-14 04:44, Aurélien Couderc via Cygwin wrote:
> > On Sun, Jul 13, 2025 at 11:54 PM Brian Inglis via Cygwin wrote:
> >> On 2025-07-12 10:43, Aurélien Couderc via Cygwin wrote:
> >>> Follow up to an old thread,
> >>> Corinna wrote this:
> >>>> Forth, by not trusting Cygwin to do the right thing and adding a lot
> >>>> of unnecessary code for each single ACL, you slow down ls(1) even more
> >>>> on Cygwin, which already gets a beating for being slow.
> >>>
> >>> Did anyone consider doing the directory lookups piece by piece, as
> >>> async requests?
> >>> Each time read and process a bit of the directory listing, while the
> >>> filesystem requests for the next bits run as Windows async requests?
> >>> That would promote at least parallelism, instead of doing it serially
> >>> like now.
> >>
> >> ISTM the assumption is that any directories accessed are already cached by the
> >> filesystem, whether Linux, Unix, or Windows, so I/O is not required, and even in
> >> Cygwin's case it only has to return the Windows entries converted to
> >> dirent/posix_dent in the directory functions (in a 62KB cache buffer holding 100
> >> entries) for use by coreutils ls(1).
> >
> > OUCH. That assumption is IMHO very wrong.
> > 1. Windows is not good at caching dirs, and it is worse with memory pressure.
> > 2. Not everyone is using SSDs
> > 3. SMB dir lookups can take multiple 1/10s per file/ACL lookup, so if
> > you do not do it in parallel it can take seconds for even small dir
> > listings. Even small async requests could be a big win here,
> > especially for remote filesystems.
>
> Not much will help if you have memory pressure other than adding memory.
> Don't know whether SAMBA can be configured to be an improvement over bare SMB
> under Cygwin?
> Some SAM and AD info is cached if you run cygserver
>
> >> Stat and ACL info require additional calls.
>
> > Right, but my proposal is to do parallel/async lookups. The Windows NT
> > kernel depends heavily on multithreading and parallelism, of which
> > Cygwin uses nothing right now for dir lookups.
>
> Not an option unless it can be cheaply done under winsup/cygwin/fhandler.
> Most utilities are GNU/BSD/Linux ports, so custom mods would have to be
> submitted and accepted upstream, as we often already have enough patches to
> maintain, to get them to build and work cleanly under Cygwin.

I think you misunderstand me. I am suggesting to improve the Cygwin
implementation of opendir(), readdir() and friends to do Win32/WInNT
calls async to speed up dir listings.

Aurélien
-- 
Aurélien Couderc <aurelien.couderc2002@gmail.com>
Big Data/Data mining expert, chess enthusiast