Why does (stat() ?) open files ?

[Ben RUBSON]

On 09 Apr 2018 15:07, Corinna Vinschen wrote:

> On Apr  9 14:12, Ben RUBSON wrote:
>> On 09 Apr 2018 12:52, Corinna Vinschen wrote:
>>
>>> On Apr  9 12:28, Ben RUBSON wrote:
>>>> Hi,
>>>>
>>>> This follows the "Why does readdir() open files ?" discussion we had a  
>>>> few
>>>> days ago.
>>>> Thank you Corinna for your answers and suport there !
>>>>
>>>> So, context is Cygwin, especially rsync, working over a Fuse FS.
>>>> This Fuse FS is assumed to be mounted on `/cygdrive/x` below.
>>>>
>>>> I finally found that readdir() does not open every file.
>>>> `ls /cygdrive/x` does not fire any open() call. Perfect.
>>>>
>>>> However, `ls -l /cygdrive/x` does, every file is opened, with read  
>>>> access.
>>>> As `rsync -an /cygdrive/x /tmp/`, which is a dry-run just grabbing  
>>>> files'
>>>> attributes.
>>>>
>>>> I then went through Cygwin code and found that
>>>> NtCreateFile/NtOpenFile calls
>>>> from symlink_info::check() in path.cc may be the culprits.
>>>> To demonstrate this I added write access to these calls, and found that
>>>> every file was then opened with write access.
>>>>
>>>> Later in this function we have a failback to NtQueryDirectoryFile call.
>>>> I assume (assume only, I may be wrong) this one does not open the
>>>> requested
>>>> file.
>>>
>>> It's nice that you're testing all this, but you should ask *why* Cygwin
>>> does it in the first place.  The reason is that the information one can
>>> gather without opening the file on Windows is insufficient to fill in
>>> all of the stat struct.  The directory info returned by
>>> NtQueryDirectoryFile just isn't, thus it's only a fallback.
>>
>> Corinna, thank you very much for your answer.
>> What info would be missing without opening the file ?
>
> uid, gid, number of links.

If we use "noacl" mount option, I think uid and gid do not really make  
sense and could be forced to some default value ?
Number of links should not be critical too.
Thus gathering files' information without opening them could be possible.
However sounds like a more correct way to handle this is in your answer  
below, though I'm not sure it will be possible.

>> Do you know where the open call could come from, when only using
>> NtQueryDirectoryFile in symlink_info::check() ?
>> (certainly related to the previous question)
>
> The answer here is that your FS should handle the open call differently
> depending on the access mask.  The NtCreateFile call in symlink_info::check
> opens the file with READ_CONTROL | FILE_READ_ATTRIBUTES | FILE_READ_EA
> only.  All these access flags only request *meta* info, not actual data
> from the data stream of the file.  In other words, a Windows open call
> without FILE_READ_DATA/FILE_WRITE_DATA and related flags does not
> actually have to open the file at all in the FS driver.   It only has to
> provide metadata subsequently, an operation which you usally can have at
> much lower cost if the remote FS is running on a POSIX OS:
>
> - NtCreateFile called with only metadata access flags does not have to
>   open the file.
>
> - Make sure to ignore the EaBuffer and EaLength parameter, rather than
>   to return a failure (this avoids YA NtOpenFile call).
>
> - Just call stat/statvfs on the remote file as required to fulfill
>   subsequent NtQueryVolumeInformationFile and NtQueryInformationFile
>   calls.
>
> Does that make sense?

Thank you very much for your detailed answer Corinna !

Yes it does make sense, at least I understand how it should work.
The FS is not really running on a POSIX OS, as it stands on the Windows  
machine using WinFsp FUSE API.
I'll see with WinFsp team if something can be done in this way.

Thank you !

Ben