NTFS fragmentation

[Larry Hall (Cygwin)]

Gary R. Van Sickle wrote:
> [cc'ing you per your request]
> 
>> From:  Vladimir Dergachev
>> Sent: Wednesday, August 02, 2006 5:33 PM
>> Subject: NTFS fragmentation
>>
>>
>> Hi all, 
>>
>>        I have encountered a rather puzzling fragmentation 
>> that occurs when writing files using Cygwin. 
>>
>>        What happens is that if one creates a new file and 
>> writes data to it (whether via a command line redirect or 
>> with a Tcl script - have not tried C
>> yet) the file ends up heavily fragmented. 
>>
>>        In contrast, native Windows utilities do not exhibit 
>> this issue.
>>
>>        Someone suggested to me that Windows requires an 
>> expected file length to be passed at the time of open, thus I 
>> searched on Google and found "fsutil" program that allows to 
>> reserve space on the filesystem.
>>
>>        I attached a small Tcl script that, when run, creates 
>> two 30 MB files - one using regular open/write pair (and 
>> which is fragmented into about 300 pieces on my system) and 
>> one using fsutil/open in append mode/seek 0 method.
>>
>>       To see the problem defragment your system, run the test 
>> script and then run analyze and ask to view report. You will 
>> see a.dat at top of the list, while b.dat never appears in 
>> the report. 
>>
>>        Despite the workaround, it is still kinda hard for me 
>> to believe that anyone has designed a filesystem that needs 
>> to know what is the file size going to be - especially for a 
>> single program writing on an almost empty disk. Perhaps there 
>> is some sort of environment variable that I need to set ?
>>
>>         Any suggestions and comments would be greatly 
>> appreciated.        
>> Please CC me - I am not on the list.
>>
>>                            thank you very much
>>
>>                                         Vladimir Dergachev
> 
> I'll try your test case when I get a chance, but my WAG is that you're
> seeing the effects of Cygwin's creation of sparse files by default for any
> file beyond a certain size.  I unfortunately do not recall what that size
> is.  What happens as you change FILE_SIZE and/or BUFFER_SIZE in your script,
> to maybe a small multiple of your cluster size?
> 


This clicked with me as well.  I was thinking first though to try it with
straight C, to remove the possibility of some TCL pollution.  Otherwise a
quick Google of "sparse" for cygwin dot com turns up lots of relevant hits,
utilities, and technical details.  Should be pretty easy to determine if the
resulting files are sparse or not with this info.


-- 
Larry Hall                              http://www.rfk.com
RFK Partners, Inc.                      (508) 893-9779 - RFK Office
216 Dalton Rd.                          (508) 893-9889 - FAX
Holliston, MA 01746

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/