Re: patches to vendor source trees - discussion

[Charles Wilson <cwilson at ece dot gatech dot edu>]

Robert Collins wrote

>>You've stated several times that you're looking only at "baby 
>>steps" -- but 
>>I think baby steps are counterproductive here.  Imagine a 
>>
> 
> I agree in principle. The issues I percieve is that
> 1) setup.exe isn't ready for more than bare bones changes - regardless
> of rpm or dpkg inspiration.


I didn't want setup to do anything but unpack "foo-ver-rel-src.tar.bz2" at 
the "/usr/src" level.  foo-ver-rel-src.tar.bz2 contains the ORIG/foo/* and 
DOC/foo/* SCRIPTS/* structure.  I realize now that I specified full 
'/'rooted paths in my original email; but I didn't really intend that the 
tarballs include the "/usr/src" part.

*PERHAPS* -- if it doesn't already -- setup could record the fact in 
/etc/setup/foo-src.list.gz that "foo-pristine-ver.tar.gz" was put into 
/home/rcollins/src/ORIG/foo/ (or /usr/src/ORIG/foo).  Likewise the other 
primary files in the foo-ver-rel-src.tar.bz2 archive.
  But this really isn't a big deal.


> 2) nothing is documented in a how-to format today - which was my goal.


Mmm...documentation...


>>2) Enforce that after each baby step, new packages and 
>>updated packages 
>>must obey whatever new standard is in place at that time.  
>>this will lead 
>>to a HUGE hodgepodge of src packages, each obeying different 
>>"standards" as 
>>the standard evolves over these baby steps.
>>
> 
> Debian follows this process. They have a standard-document that is
> versioned. All packages state what revision they follow, and the
> automated tools understand the last x versions. Debian has, for example,
> taken a year to sort /usr/doc into the new standard,  but _no breakage_
> occured during that process, and _no pressure_ was on package
> maintainers. Works for me :}.


Well, yeah -- no matter what, you're gonna get this (unless you eject 
packages whose maintainers don't update on your schedule).  My argument is, 
I'd rather get mostly there fast, so that the src package versioning 
distribution looks like this (horiz scale represents "closeness" to the 
final standard.  Version 0 = current, version 1 = the specification we're 
currently designing, version 2 = the next tweak, etc.)

version   0......................1..2.34

The baby step scheme is

version   0..1..2..3..4..5..6..7..8..9..10

I like asymptotic approaches, not linear ones.  (Also, we get "there" with 
fewer iterations, usually...of course this depends on how "baby" your baby 
steps are)


>>out.  Big changes are in the wind, why not this, too?
>>
>>Of course, I'm not saying enforce immediate switchover of all 
>>packages NOW 
>>-- just define the new standard, however radical, and then 
>>switch each 
>>package over when it is updated in the normal course of events.
>>
> 
> I'd love too be able to do that - with with the rpm or deb inspiration,
> but frankly setup ain't ready for that.


Ready for what?  All I'm saying is "maintainers, make your src package look 
like this".  All setup has to do is unpack it.  Under /usr/src (or ~/src if 
you like)


> All setup is ready for is
> * files go into /usr/src
> 
> not even a post install script is run for src files AFAIK. That could be
> trivially changed I think.


this is all we need (not even the postinstall is necessary for my proposal).


>>As much as Chris doesn't like it, there ARE advantages to the 
>>RPM style. 
>>
> 
> I'm not sure on Chris' opinion, but I am a debian convert :}.


Dunno much about .deb; I tend to lump all the automated build systems together.


>>Here's my take (I'll use zlib as my example):
>>
>>/usr/src/cygwin/SRC
>>/usr/src/cygwin/BUILD
>>/usr/src/cygwin/ORIG
>>/usr/src/cygwin/SCRIPTS
>>/usr/src/cygwin/INST
>>/usr/src/cygwin/PACKAGES
>>/usr/src/cygwin/DOCS
>>
> 
> Right. For comparision, IIRC, the debian one looks like (using the fully
> automated process). The / in this list is the dir that the process was
> started from - not necessarily /usr/src. For a arch independent package
> cygwin vendor version 1.3 suffix 2 we get
> 
> /cygwin1.3-2.diff.gz
> /cygwin1.3-2.dsc
> /cygwin1.3.tar.gz
> /cygwin1.3
> /cygwin1.3/debian (added by the patch in the .diff.gz file) - think
> SCRIPTS
> /cygwin1.3/debian/rules (think .spec)
> /cygwin1.3/debian/README.debian .
> /cygwin1.3-2.deb
> /cygwin1.3-2-doc.deb
> /cygwin1.3-2-src.deb


Where does stuff ACTUALLY get built?  in /cygwin1.3/* ?  Ick!  I'd imagine 
your '/' directory (wherever it happens to be) gets awful cluttered with 
oodles of foo2.4* bar4.6* and baz1.3* files and directories...


>>So, zlib-1.1.3-9-src.tar.bz2 contains:
>>/usr/src/cygwin/ORIG/zlib/zlib-113.tgz
>>   (pristine tarball, in whatever format the primary
>>    developers provide it.  No tarball renaming.
>>    /etc/setup/zlib-src.list.gz will take care of
>>    removing it at the proper time.  BTW, *DOES* setup.exe
>>    worry about files installed from -src packages?)
>>
> 
> Nope. IMO it shouldn't. The cygbuild script or whatever we call it, that
> can call setup to grab the source, 


Okay, you're assuming setup has the argv handling stuff already...

> and build and package it for you,
> _that_ can worry about removing the source. For packages like
> kernel-source under linux, setup.exe should remove the tarball, but not
> the extracted source dir. 


Yes!  But I wasn't going that far.  Cygbuild is in the future (say, version 
3 in my scheme above).  Version 1 -- packages have their own specific build 
script.  Later, that build script is converted into a .spec-like file or 
rules-like file for cygbuild to parse.  Or something.

Anyway, "my" proposed -src package only contains:

/cygwin/ORIG/zlib/zlib-113.tgz
/cygwin/ORIG/zlib/zlib-1.1.3-9.patch
/cygwin/ORIG/zlib/zlib-1.1.3-9.postinstall
/cygwin/ORIG/zlib/*
/cygwin/DOCS/zlib/zlib-1.1.3.README

/cygwin/SCRIPTS/zlib-build


setup-uninstall should just remove those files.  Not the BUILD/zlib or INST/zlib or other "created" files


>>/usr/src/cygwin/ORIG/zlib/zlib-1.1.3-9.patch
>>/usr/src/cygwin/ORIG/zlib/zlib-1.1.3-9.postinstall
>>/usr/src/cygwin/ORIG/zlib/*
>>/usr/src/cygwin/DOCS/zlib/zlib-1.1.3.README
>>    maintainers.  So, keep 'em out of the .patch; makes it
>>    easier for the cgywin maintainer to organize the patches
>>    for upstream submission...
>>
> 
> My point on this was that the patch for the end user, and how the
> maintainer manages their patches are orthogonal. A one liner to create a
> new patch at package time can be scripted, and won't add to your
> management overhead.


Not really.  If you make it hard to map the maintainer's method to the 
"official" packaging standard, then you are creating an artificial barrier 
to new ports/maintainers.  Maximum flexibility says: "put your patches here 
(where they won't get lost) and make sure your build script will apply them 
properly".

Your idea makes it difficult for *me* to keep track of all the various 
patch files -- or to explain to a new maintainer.  I'd have to maintain a 
*separate* build script that handled the *separated* patches, but creates a 
src tarball with a unified patch and a build script that uses it. (I'm 
thinking along the lines of "how do I ensure that someone else can take 
over maintaining my 'readline' or 'libtiff' package if I get hit by a bus". 
  With the "all-patches-in-one" method, it would be practically impossible 
for a NEW maintainer to, say, remove the lossless jpeg stuff from the jpeg 
package, but keep the dllization patches.)

I'd like for anybody to be able to take over maintainance of ANY package if 
the original maintainer goes kaflooey, using ONLY the stuff in the -src 
package.

 
>>/usr/src/cygwin/SCRIPTS/zlib-build

>>Why do I want zlib-build to handle so much?  Why do I want so 
>>much stuff in 
>>ORIG, and NOT created within the unpacked src dir by .patch?
>>
>>It's very difficult to add a file to a package, AND munge the 
>>Makefile's to 
>>install it.  Okay, it's not *technically* difficult -- for 
>>non-autotool 
>>packages you just *do* it.  For autotool packages, you have 
>>to add the new 
>>file to the Makefile.am, then re-run automake and autoconf -- 
>>which leads 
>>to a LOT of changes in a LOT of the files == BIG patch.
>>
> 
> -ONLY if your autotools version is different- (or for old autoconfs :[).


Well, that's pretty likely.  Notwithstanding my autotool harmonization 
packages, there are packages out there that use autoconf 2.13, 2.50, and 
2.52 (and soon 2.6 and 3.0).  There probably aren't many packages that 
haven't been updated to 2.13 -- the four year 2.50 development cycle was 
long enough to flush 2.12 out of the pipeline. <g>

automake 1.3 still sees use, and there are six (count 'em, six) subversions 
of 1.4 that have different Makefile.in signatures: 1.4, 1.4p1,... 1.4p5. 
And don't forget 1.5.


> I see your point. I don't agree though: 
> for non-autotools stuff, the patch is easy like you said.
> for autotools stuff, the src has to depend on having the correct
> autotools installed.


Which one is "correct"?  That'll be different for each package (worst case)


> then the patch only alters Makefile.am and configure.in. 'nuff said.


You are assuming that people who want to build from (our) -src packages 
MUST run autoreconf and/or automake themselves, before configuring.  That's 
contrary to the autotools philosophy: distributed packages should be 
buildable even if the builder does NOT have autotools installed.  (That's 
why libtoolize puts the libtool scripts into the target package itself, 
instead of assuming that libtool itself will be installed on the builder's 
machine)


>>But in both cases, these changes are NOT ultimately destined 
>>for upstream 
>>inclusion.  They are cygwin-only, and will never make it into 
>>the "real" 
>>package.  IMO, .patch should be limited to code changes that are 
>>upstream-compatible.  Or at least to those changes that aren't frufru.
>>
> 
> yees, I see what you mean. However you have an environment where
> building the package as it gets installed requires a fixed diretory tree
> - information not in the src dir.


Okay, but so does debian, right?  (Assume "my" version doesn't explicitly 
specify /usr/src/, but starts at /cygwin/<CAPITAL>/*)


> You're also putting the src dir at a fixed level in the tree relative to
> the package data - which might play havoc with nested source trees like
> winsup+newlib. 


Err...only if those are ever downloaded and built independently of each 
other.  I'm assuming they will not be.


> If we make the extra work easy, then is it HARD? Seriously, I'll give
> you a ssh account here, and you can have a play with the debian package
> building environment. Nothing is easier - for the maintainer - IMO.


Well, it's already pretty easy -- for an autotool'ed project -- if you 
understand .am files and .m4 files.  But ordinary porters should NOT have 
to learn the intricacies of m4 scripting just to insure that package foo 
installs its cygwin-specific README file.  Hell, they shouldn't have to 
learn .am files except in extreme circumstances -- yet we're discussing 
mandating that ALL cygwin packages have a cygwin-specific README, AND that 
all porters muck with the autotool config files so that it gets installed 
by "make install" after patch;autoconf;automake?

this is madness (to borrow a phrase).


>>Using a build script (zlib-build) allows us to have multiple patches, 
>>
> 
> So how many build scripts do we need? And how do they determine which
> patches to apply?


At *first* (step 1, but not a baby step <g>), one build script for each 
package.  So, zlib-build.  libpng-build.  automake-build.  etc. (versioning 
aside -- see below)  Later, those get converted into, what?, zlib.cyg, 
libpng.cyg, and automake.cyg.  cygbuild parses them, etc. Just like 
dpkg/rules or rpm/spec.


> One build script for all packages, that I like (as deb and rpm have).


Sure -- cygbuild comes *after* the initial (non-baby) step.


>>applied in the proper order.  We don't need to "revert" to 
>>pristine source, 
>>'cause that's what we're distributing.  AND, this sort of 
>>
> 
> distibruting pristine source I like. Then my single-patch thing adds
> everything we use to the pristine source.
> 
> 
>>My proposal also minimizes the hurdle that new porters must 
>>overcome -- IF 
>>
> 
> Ok. I thought mine did as well... guess not. 


Well, I dunno.  Everybody has their own build style.  Mine pretty much 
sucks (no training).  I bet your proposal works great for your style, but 
sucks for mine -- and my proposal works great for my style, but sucks for 
yours.  :-)


>>Do we want a simple src packaging standard -- that serves as 
>>a disincentive 
>>for new package ports, because it doesn't allow for complex 
>>builds?  "No, 
>>I'm sorry -- you can't submit that package until you can get 
>>it to build 
>>using this src packaging standard." "But I can't -- I need to 
>>wait for this 
>>change to autoconf and that change to binutils -- THEN I can 
>>get it to 
>>build 'simply'"  "Sorry, you must table your package.  Go 
>>work on autoconf 
>>and binutils, then come back"
>>
> 
> Very good point.
>  


I know.  Because that's where I *am*!  I want to update readline, but I 
really can't until after the auto* thing is resolved -- unless I want to do 
to readline what I recently did to ncurses.  Ncurses-5.2-6 builds much more 
cleanly than 5.2-5 did (and uses the auto-import) -- but it took me about a 
week to get it there (and that's for a stable, already ported package!). 
I'd much rather NOT do that with readline -- and go straight to 
"./configure --enable-static --enable-shared").  But first libtool, which 
requires auto*, which (because of various issues) requires the auto* 
harmonization, which is now waiting on the new setup...Sigh.


> Ok here's what I think we are agreeing on:
> 1) The package should be able to be recreated by anyone downloading the
> source
> 2) We should distribute pristine source.
> 3) We need to allow an arbitrarily complex build process decided by the
> maintainer


Yep.


> Your proposal (key points):
> * in /usr/src/Cygwin create a few dirs and place the relevant items for
> each tool in the appropriate dir.


Okay, but let's just say "cygwin/" is stored in the tarball.  Assume 
setup.exe unpacks it in /usr/src -- but the tarball *should* be relocatable.


> * provide a sample script for new packages to learn from
> * Provide the patches needed to *allow the script* to convert the source
> from pristine to cygwin-ready.


Yep.


> My proposal (updated) (key points):
> * include in the directory above the extracted source, a consolidated
> patch that alters the pristine source to be ready for building into a
> package. 
> * The patch should add a README.Cygwin to the $(top_srcdir)/cygwin
> directory at a minimum.
> 
> Now, I believe that they *both* meet the things we agree on above. To
> address your questions about my proposal,
> * how are the requirements for building different packages from one
> source served?
> Thats up to the maintainer. If we? (you?) provide a couple of example
> scripts new maintainer will have a good starting point. The mechanism
> could be as easy as make (build-src| build-libs| build-binary )

                       ^^^^
UUGGHH! You want to muck with the Makefile.am, Makefile.in, Makefile's to 
support building *our* split-up packages directly via make?  I *really* 
don't think that's necessary.  Our packages are just tarballs, really, 
which will only require special tar commands in the build script (or the 
.cyg file as parsed by cygbuild) -- not wholesale "makefile" mucking. My 
personal cygwin-build script does this:

tar cvjf ../mingw-new.tar.bz2 usr/bin/mingwm10.dll \
   usr/include/mingw usr/lib/mingw
tar cvjf ../w32api-new.tar.bz2 usr/include/w32api \
   usr/lib/w32api
tar cvjf ../cgywin-new.tar.bz2 usr --exclude='usr/bin/mingwm10.dll' \
   --exclude='usr/include/mingw' --exclude='usr/lib/mingw' \
   --exclude='usr/include/w32api' --exclude='usr/lib/w32api' \
   --exclude='usr/bin/runtest'

I really want to avoid imposing "learn autotools and .am and .m4" on all 
porters.


> * What about complex packages?
> Well, thats the beauty, by including the scripts within the source tree
> they can't get lost. So it becomes orthogonal to the 'standard'.


Ah, but I'm assuming that *THIS* is the source tree:

cygwin/ORIG/foo.tgz
cygwin/ORIG/foo.patch1
cygwin/ORIG/foo.patch2
cygwin/SCRIPTS/foo-build
cygwin/DOCS/foo.README

You're assuming that the "unpacked" foo/ directory is the source tree.  But 
in either case, the patches won't get lost.


> What I don't like about your idea is:
> 1) fixed directory location. I like to do this stuff in ~/src, not
> /usr/src. Others may have their own preferences.


Misunderstanding.


> 2) We'll end up with ??how many ?? different scripts in that directory
> tree? (remember, as each -suffix will have a different script (think on
> this if you need to) installing the source to three revisions of package
> foo ends up with three foo-build's, three foo-README's etc.


Erm...and what's wrong with that?  debian has how many dsc files for 
different versions of the same package?  I don't see the problem.

(Actually, I hadn't thought too hard about versioning the build scripts. 
Of course, it IS necessary.)

As I see it, the debian method eliminates my:

cgywin/PACKAGES/
cgywin/ORIG/
cgywin/SRC/

directories and just uses the top level (call it cgywin/) for the files 
that would go into them.  It also collapses the cygwin/SRC up to the 
cygwin/ level, and unpacks the tarball there.

the debian scheme doesn't seem to have any provision for building *outside* 
the unpacked source directory (unless it uses some dynmically created dir 
in /var/tmp ?).  ditto the temporary installation directory.  So that's

cygwin/BUILD
cygwin/INST

finally, the

cygwin/SCRIPTS/
cygwin/DOCS/

are kinda replaced by the cygwin/<pkgdir>/debian/ directory (created by the 
patch) coupled with the .dsc file at the cgywin/ level.

do I have that right?  If so, then the primary difference between the 
proposals is

a) splitting things up into multiple toplevel directories vs. everything 
goes in one toplevel directory (not counting the unpacked source archive)

b) script handles applying (possibly) multiple patches; some support files 
NOT created by a patch but are included directly in the primary -src 
archive  vs.  single patch, which creates SOME support files within the 
patchedNunpacked src tree, one other support file (.dsc) is NOT created by 
that unified patch.

I gotta say, this (b) is one area where I think RPM-style wins out over 
DEB...ever look at a linux kernel-src RPM?  *Much* more informative than if 
everything were all in a single patch.  IMO.  <g>

However, in both cases the ultimate goal is to transition to a "process 
description file" that is interpreted by a custom tool (cygbuild? debbuild? 
etc)

--Chuck