SPARSE files considered harmful - please revert

Max Bowsher maxb@ukf.net
Sun May 18 23:19:00 GMT 2003


Martin Buchholz wrote:
> This patch is a bad idea.
>
> 2003-02-18  Vaclav Haisman  <V.Haisman@sh.cvut.cz>
> * fhandler_disk_file.cc: Include winioctl.h for DeviceIoControl.
> (fhandler_disk_file::open): Set newly created and truncated files as
> sparse on platforms that support it.
>
> As someone on the mailing list asked, "If making every file sparse is
> such a good idea, why isn't it the default?".

Me, I think.

I agree that making *all* files sparse is a bad idea, but let me just say
that I consider the usage of the words "considered harmful" harmful to
effective discussion. It sets the scene for antagonistic flamewars. Ditto
"revert".

Actual test data, as you give below is *good*, though:

> My experience has been that for me, sparse files take up much more
> disk space than non-sparse files, and are also signicantly slower.
>
> I build software.  My build trees have 50000 files, average size 8k.
> When I copied build trees to a Win2000 NTFS disk using Cygwin tools
> (either cp or tar or rsync) the actual space used on the disk (as
> reported by df, not du) quintupled.
>
> Here's what I think is happening.  Sparse files are implemented like
> compressed files, using 16 clusters.  See this web page:
>
>
http://www.storageadmin.com/Articles/Index.cfm?ArticleID=15900&pg=1&show=654
>
> As a result, a non-empty but small sparse file takes up a minimum of
> 16*clustersize bytes on the disk.  My measurements suggest an overhead
> of 32kb per file with a cluster size of 4kb.
>
> Here are some experiments to support my results:
> MKS's commands creates files 5 times smaller than Cygwin commands.
...
> I'm sure if you do the experiments yourself, you will see this for
> yourself.  To reproduce this problem, you need NTFS 5.0 on Windows
> 2000.  Sparse files are a recent NTFS feature.
>
> The patch is obvious, but I'll send it to cygwin-patches anyways.
>
> Without this patch, Cygwin is unusable for me.

May I suggest a middle road? Why not let sparse files be configurable as a
$CYGWIN option? This would allow those users who actually want them to
enable them with minimal effort, but keep them off for most users.

Max.


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/



More information about the Cygwin mailing list