Cygwin 3.6 possible issue handling compressed .pdb files on SSD?

[brian.inglis@systematicsw.ab.ca]

On 2025-01-15 10:26, Corinna Vinschen via Cygwin wrote:
> On Jan 15 09:12, Brian Inglis via Cygwin wrote:
>> On 2025-01-14 03:13, Roland Mainz via Cygwin wrote:
>>> On Tue, Jan 14, 2025 at 7:19 AM Brian Inglis via Cygwin
>>> <cygwin@cygwin.com> wrote:
>>>> On 2025-01-13 13:10, Roland Mainz via Cygwin wrote:
>>>>> I just hit an endless loop with /usr/bin/cp from "coreutils" version
>>>>> 9.5-1 copying a larger *.pdb file (it seems that only this specific
>>>>> file is affected...) from Visual Studio 19.
>>>>> [...]
>>> I think I found the problem:
>>> The *.pdb file uses NTFS compression:
>>> ---- snip ----
>>> /bin/winfsinfo filebasicinfo "$(cygpath -w
>>> $PWD/../build.vc19/x64/Debug/nfs41_driver.pdb)"
>>> (
>>>          filename='C:\cygwin64\home\roland_mainz\work\msnfs41_uidmapping\ms-nfs41-client-kofemannvacation\build.vc19\x64\Debug\nfs41_driver.pdb'
>>>          CreationTime=133812707624654816
>>>          LastAccessTime=133813220892976366
>>>          LastWriteTime=133812707639811081
>>>          ChangeTime=133812707639811081
>>>          typeset -a FileAttributes=(
>>>                  FILE_ATTRIBUTE_ARCHIVE
>>>                  FILE_ATTRIBUTE_COMPRESSED
>>>          )
>>> )
>>> ---- snip ----
>>>
>>> If I remove the "FILE_ATTRIBUTE_COMPRESSED" flag /bin/cp works without problems.
>>> I think the issues here are:
>>> 1. Coreutils 9.5-1 /bin/cp erroneously assumes that a file is sparse
>>> if the number of blocks is smaller than $((filesize / fs_blocksize)) -
>>> but in this case the file is NOT sparse, just compressed.
>>> 2. The loop to copy a sparse file is faulty because there are no holes
>>> in that file. That itself is IMHO already a bad idea to have a
>>> separate codepath for sparse files, just the normal codepath should
>>> use SEEK_HOLE and just skip those in the destination
>>
>> A possible issue is that Cygwin assumes sparse files on SSD
> 
> No, that's not the problem, because SPARSE handling requires that
> the FILE_ATTRIBUTE_SPARSE_FILE flag of the file is actually set.
> 
> For instance, see lseek w/ SEEK_DATA/SEEK_HOLE, which handles
> non-sparse files as documented by the Linux man page
> https://man7.org/linux/man-pages/man2/lseek.2.html:
> 
>          if (!has_attribute (FILE_ATTRIBUTE_SPARSE_FILE))
>            {
>              /* Default behaviour if sparse files are not supported:
>                 SEEK_DATA: seek to offset
>                 SEEK_HOLE: seek to EOF */
>              fpi.CurrentByteOffset.QuadPart = (whence == SEEK_DATA)
>                                               ? offset
>                                               : fsi.EndOfFile.QuadPart;
>              break;
>            }
> 
>> so we need
>> fhandler/disk_file:fstat_helper to allow cp to handle compressed files
>> normally.
> 
> Say again?  What exactly are you expecting from stat()?  Fibbing about
> the number of blocks taken by a FS-compressed file?

Just a suggestion to work around the issue transparently?

> For the time being, it seems the assumption that #blocks * blocksize <
> filesize is not really correct.
> 
> Actually cp(1) should test if lseek(SEEK_HOLE) returns EOF.  If so,
> the file isn't sparse.
> 
> On Cygwin it could also check with
> 
>    (ioctl (fd, FS_IOC_GETFLAGS, &flags) & FS_SPARSE_FL) != 0
> 
> that the file is sparse, but the lseek test is target-independent.
> 
> On Cygwin and Linux you could also test with
> 
>    (ioctl (fd, FS_IOC_GETFLAGS, &flags) & FS_COMPR_FL) != 0
> 
> that the file is compressed, but there's a problem so far.  The
> Linux flag is called FS_COMPR_FL but the Cygwin flag is called
> FS_COMPRESSED_FL.  THis flag is only supported on btrfs and I'm
> not sure it already existed when I added FS_IOC_GETFLAGS...
> 
> Anyway, I'll change FS_COMPR_FL to FS_COMPRESSED_FL and make
> FS_COMPRESSED_FL an alias for backward compat...

Thanks - I'll submit a bug report and ask what approach they would prefer?

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retrancher  but when there is no more to cut
                                 -- Antoine de Saint-Exupéry