[RFC] zchunk

Achim Gratz Stromeko@nexgo.de
Sat Jul 3 13:17:04 GMT 2021


I've finally cleaned up my build of zchunk on Cygwin and MingW64 (at the
current version 1.1.16 now).  The latter is losing some exectuables
using POSIX interfaces that are not easily emulated since I am only
interested in having libzck available anyway.

Upstream:
https://github.com/zchunk/zchunk
(further links at the end of that page)

If you want to have a look at the Cygwin packages:
https://cygwin.com/git-cygwin-packages/?p=git/cygwin-packages/playground.git;a=shortlog;h=refs/heads/zchunk
https://ci.appveyor.com/project/cygwin/scallywag/builds/39855288/job/11vtsw0l7p5acmxw
https://ci.appveyor.com/project/cygwin/scallywag/builds/39855288/job/2e0rcx2d9hdw6nm5

The reason I'm looking at zchunk is that it would be helpful with
reducing the amount of initial data that needs to be transferred on an
update with setup.

When using an external dictionary (only needed on the server) on a
slightly re-arranged setup.ini file (so I can create just one chunk per
source package), I'm getting the following results:

original:    3,4M setup64.zst
chunked:     3,8M setup64-rechunk.ini.zck
dictionary:  110K setup64-rechunk.zdict

Delta from 2021-06-26 to 2021-07-03
Would download 141634 of 3960442 bytes
Matched 4074 of 4087 chunks

Delta from 2021-05-29 to 2021-07-03
Would download 198463 of 3960442 bytes
Matched 4032 of 4087 chunks

Delta from 2020-01-02 to 2021-07-03
Would download 1438581 of 3960442 bytes
Matched 2938 of 4087 chunks

So the chunked file is 400k or about 12% larger than the simply
compressed setup.ini (that's the additional header and some loss in
compression ratio due to the shorter blocks the compressor sees).  On
the other hand, each update request would typically only pull between
140k…200k (or about 5% of the compressed setup.ini) if the local copy is
reasonably recent.  Even an update of a system that's over a year old
still only needs about half of the data for getting up-to-date.

Zchunk would fold the download of the missing chunks (the update to the
current version) into a single HTTP request (via ranges) most of the
time after requesting and analysing just the header at the beginning of
the file, so in terms of request rate the change would be negligible
(about one or two additional roundtrips).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada


More information about the Cygwin-apps mailing list