scallywag / cygport not pulling lzip

[Brian Inglis]

On 2021-01-08 12:11, Achim Gratz wrote:
> Brian Inglis writes:
>> Do we know what the frequency weighted difference is on bandwidth of
>> packages actually downloaded?
> 
> Not that I know of, as everything goes through mirrors.  But I happen to
> have a complete Cygwin mirror on disk at the moment plus another one
> that only has the packages for my install and that's a fairly large
> installation, but without a desktop environment:
> 
> 30G     /mnt/mirror/cygwin
> 149G    /mnt/fullmirror/cygwin
> 
> So you can probably assume that only about 20% of the files are
> frequently accessed (likely significantly less since most folks would
> not install the debuginfo or source packages that are included in the
> above figure).
> 
>> I am more concerned with mirror providers (and also the lack of them)
>> especially those with limited resources, and those in marginal
>> locations and circumstances, for whom download time and charges may
>> override other considerations, and perhaps prevent them (or many) from
>> accessing or taking full advantage of available software.
> 
> We could save way more space than that by de-duplicating the noarch
> parts into their own archives as I have already demonstrated before.
> The last time I did that I was cutting out around 30GiB IIRC.
> 
>> I doubt the unarchiving time difference is more than a blip in the
>> total time required to *download* *AND* install any package, greatly
>> outweighed by the download time difference, unless you are on a big
>> pipe to a nearby mirror.
> 
> It is not, with a typical VDSL connection I'd be able to download faster
> than I can install on a more typical machine, I need only about 5MiB/s
> to saturate the filesystem for small files and around 20…40MiB/s for
> large ones (to an NVMe drive, a spinning disk or some of the slower SSD
> can't sustain that).  But that point is somewhat moot since setup will
> always mirror to disk first and that's not easy to change since we read
> the file twice: once for the SHA512 check (which can use up to around
> 300MiB/s input bandwidth somewhat higher in peaks and then the actual
> installation).

Some setup phase stats from my own most recent upgrade of about 130MB of 
downloads, and stats since 2013 (nearly 8 years) since I last cleared setup.log:

$ cyg-setup-phase-times.awk /var/log/setup.log.full
sv 00:04:26 dl 00:01:28 pr 00:00:02 ui 00:00:36 ex 00:03:35 pi 00:07:12
$ cyg-setup-phase-times.awk /var/log/setup.log
sv 04:32:46 dl 02:31:27 pr 00:16:08 ui 00:51:10 ex 06:06:49 pi 00:00:41

phases are:

sv - solve formerly Adding required packages - high times are interaction delays
dl - download
pr - preremove
ui - uninstall
ex - extract
pi - postinstall

so your comments about extracts are validated, taking about 3 times as long as 
downloads even on a currently 2MByte/s medium speed cable modem link to a nearby 
(7.5km direct, 11km drive, 15 hop 10ms round trip) university campus mirror in 
recent years.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]