Glib2

[Charles Wilson]

[Back on list, just to make sure it's archived.  Max and I have bounced 
some emails back-n-forth privately, so you're coming in on the middle of 
a conversation... See Gen Zhang's original message on 12/4/2002]

Also, this is kinda stream of conciousness...because I end up at the end 
of the message somewhere other than where I thought I was going... <g>

Okay, from Zhang's report:
 > After re-libtoolize-ing the sources (to allow it to produce shared
 > libs) the compilation is fine through glib and will produce
 > libglib-2.0.la, but will complain about undefined references to
 > things like '_g_free' and '_g_log' (which I presume is in
 > libglib-2.0.la). I've attached the output from make (upon a second
 > run, after it going through glib).

The error was not in creating the DLL in the first place(*).  The error was

 > /bin/bash ../libtool --mode=link gcc  -g -O2 -Wall -D_REENTRANT   -o
 > glib-genmarshal.exe  glib-genmarshal.o ../glib/libglib-2.0.la -liconv
 > -lintl -liconv
 > gcc -g -O2 -Wall -D_REENTRANT -o .libs/glib-genmarshal.exe
 > glib-genmarshal.o  ../glib/.libs/libglib-2.0.dll.a -L/usr/lib
 > -L/lib/w32api -luser32 -lkernel32 /usr/lib/libintl.dll.a
 > /usr/lib/libiconv.dll.a -Wl,--rpath -Wl,/usr/local/lib
 > glib-genmarshal.o(.text+0x1807): In function `complete_in_arg':
 > /usr/src/glib-2.0.7/gobject/glib-genmarshal.c:218: undefined reference
 > to `_g_log'
 > ...

(*) unless the DLL itself exported no symbols at all.  But, new libtool 
DOES NOT USE def files anymore.  AT ALL.  (and, you can't even give it 
one as an argument...)

Max Bowsher wrote:

 >>> .def files are passed as -Wl,-retain-symbols-file deffile, not,
 >>> as is correct, the first 'object' file.

Which means this ^^^^^ is wrong.

"-Wl,--retain-symbols-file -Wl,filename" is perfectly okay -- see the ld 
manpage.  However, $export_symbols contains a filename whose contents 
are a list of exports -- like a def file, but not exactly; it's more 
unix-focused.  If you trace the code in ld.exe, you find that 
--retain-symbol-file's argument is handled in a COMPLETELY different 
manner than a def file would be (it mainly acts to prevent the given 
symbols from being stripped; it doesn't "trigger" the necessary hooks to 
export those symbols in a DLL on cygwin/mingw).  So, 
--retain-symmbols-file is okay on other unices, but not really the right 
switch for libtool to pass to ld, on cygwin.

On cygwin, what you end up with is
   "don't strip these symbols"
and
   "there is no def file, so by default, export all symbols(*)"

(*) unless there are symbols that have been specifically decorated with 
__declspec(dllexport), in which case ONLY those decorated symbols are 
exported.  It turns out that THIS is the root of the problem with 
glib+cygwin.

So, on cygwin, this definition of archive_expsym_cmds is probably wrong 
-- but not in the way you thought.  Because **archive_expsym_cmds** is 
NOT USED (much).  [except in glib...sigh...]

Cygwin defines "always_export_symbols" as "no" (unless you use the 
'--export-symbols <FILENAME>' switch when calling libtool), and 
"export_symbols_regex" is empty unless you explicitly call libtool with 
the '--export-symbols-regex <REGEX>' switch.  Given those typical 
definitions, we use "archive_cmds" NOT "archive_expsym_cmds" almost 100% 
of the time.  Only when either '--export-symbols <FILENAME>' or 
'--export-symbols-regex <REGEX>' are specified do we, on cygwin, use 
"archive_expsym_cmds".

And glib does.  It uses '--export-symbols glib.def' (on OS_WIN32, but 
not CYGWIN).  More below.

Further, as far as ld is concerned, the def file doesn't need to be the 
FIRST object file.  That was the case for dlltool (or you could 
explicitly say "--def def.file"), but not for ld once the functionality 
was absorbed.  For ld, the def file just needs to appear as one of the 
arguments to ld.  (see gld_${EMULATION_NAME}_unrecognized_file in 
binutis/ld/emultempl/pe.em)

 > But I'm worried about just assuming that the -export-symbols file is
 > a def file. There was some code in libtool somewhere (recently
 > removed) which searched for EXPORTS to try and determine if it was a
 > def file, and else tries to make one.
 >
 > I'm not sure how export-symbols works on other platforms.

Explained above.  Now, IF someone were to specify --export-symbols-regex 
or --export-symbols on cygwin, then the archive_expsyms_cmd would be 
used -- and we need something like

_LT_AC_TAGVAR(archive_expsym_cmds, $1)='$CC -shared -nostdlib
$predep_objects $libobjs $deplibs $postdep_objects $compiler_flags -o
$output_objdir/$soname $export_symbols
${wl}--out-implib,$lib'

(e.g. no retain-symbols argument, just include $export_symbols as one of 
the "object" files)

There's just one problem: ld tests "unknown objects" filenames to see if 
they end in ".def" or ".DEF".  $export_symbols is (usually) libname.exp. 
  Badness.

So, for this to work as we expect, we must
   (a) change $export_symbols to "libname.def" not "libname.exp".  Old 
versions of libtool (back in dlltool days) did
   cp $export_symbols $output_objdir/$soname-def
(dlltool was less picky about the actual name of the .def file; you'd 
just say "--def foo" and it was happy.  We'd need to
   cp $export_symbols $output_objdir/$soname.def
to insure that the file ended with .def.

   (b) change $export_symbols_cmds so that it prepends "EXPORTS".  Now, 
since always_export_symbols is false by default, the only time 
$export_symbols_cmds ever gets called is when you need to GENERATE an 
exports list -- and since '--export-symbols <FILENAME>' takes an 
argument, you don't need to generate it in that case (it's in 
<FILENAME>).  So, the only time, on cygwin, that $export_symbols_cmds is 
ever called, is when you've done '--export-symbols-regex <REGEX>'  Which 
makes things EVER so much more complicated.  Sigh.

Anyway, perhaps something like $export_symbols_cmds =
   "\$NM \$libobjs \$convenience | \$global_symbol_pipe | \$SED 's/.* //'
   | sort | uniq | \$SED -e '1iEXPORTS' > \$export_symbols"
(note the final $SED command)

However, that still doesn't differentiate between DATA and function 
exports; the .def file, if used, must identify DATA exports.  Which 
means $export_symbols_cmds has be a LOT more complicated:

$NM $libobjs $convenience | \$global_symbol_pipe | \
\$SED -e '/^[BCDG] /s/^\([BCDG]\) \(_[_A-Za-z0-9]*\) 
\([_A-Za-z0-9]*\)/\3 DATA/' | \
\$SED -e '/^[AISTW] /s/^\([AISTW]\) \(_[_A-Za-z0-9]*\) 
\([_A-Za-z0-9]*\)/\3/' | \
sort | uniq | \$SED -e '1iEXPORTS' > \$export_symbols"

Old libtools used to use a specialized program (impgen) to do this, and 
a combination of dlltool calls.  But we want to stay away from that, and 
stick with ld, gcc, and nm.  So, we'll have to work harder...

Worse, if you actually specify --export-symbols=FOO on the libtool 
command line (e.g "use the file 'FOO', which already exists and 
specifies the symbols to export, as $export_symbols") then 
$export_symbols_cmds is NOT called, AND you lose platform 
cross-compatibility in the $export_symbols file.  On unix, the file 
$export_symbols should just contain
   bob
   fred
But on windows, it should contain
   EXPORTS
   bob
   fred DATA
because we're using it (in my "proposal" above) as a def file.

Basically, what you're trying to do is build an .def file on the fly. 
This USED to be very helpful -- before the auto-import, auto-export 
functionality was added to ld.exe itself.  Now, it just messes things up 
on cygwin.  (It might still be useful on other platforms, but not here).

I think this whole concept (on-the-fly def file created by libtool) was 
a kludge, because dlltool NEEDED a def file (during its 'make a dll' 
mode), so libtool created one (by calling dlltool in its 'make a def 
file' mode).  Now, we don't use dlltool anymore, and ld can 
automatically figure out what, and how (DATA or function) to export.  So 
none of this is needed on windows anymore, in general.

The only time you need any of this "specially restrict the export list 
or specifically export a known list of symbols" is when you're doing 
something *special*.  And that requires developer intervention....

In cygwin/mingw, the typical library needs to do one of the following:
   1) export everything (almost always, this is what you want)
   2) export everything EXCEPT ...
   3) export only ...
   4) export everything, BUT you've specifically decorated some symbols 
(like structs) with __declspec(dllexport).  If it weren't for libtool's 
paranoia about passing flags to the linker, you'd just say 
'--export-all' and be done.  But libtool won't let you do that, so life 
is hard.   This is the glib case.

If you want #1, you just let ld do the work.  No need for export_symbols 
or any of that junk.

If you want #2, the *Developer* should specify that in the makefile. 
E.g. libfoo_la_LDFLAGS=.... --exclude-symbols SYMBOL,SYMBOL,... or 
--exclude-libs LIB,LIB,... (to avoid exporting symbols pulled in from 
static convenience libraries).  But that's a *developer* thing; libtool 
shouldn't attempt to do that itself.  It isn't clairvoyant.  Perhaps the 
developer should insure that the Makefile only puts --exclude-symbols 
... when building for cygwin/mingw by using conditionals.  But it's a 
developer thing.

Now, if on unix, a developer used --export-symbols or 
--export-symbols-regex as a means of NOT exposing certain symbols to 
clients, then we need to take special care when porting to windows. 
Create a .def file and add it (conditionally) to the _SOURCES [BUZZ! 
doesn't work], or add --exclude-[lib|symbol] to _LDFLAGS (conditionally) 
[BUZZ! Doesn't work.  Damn, but libtool is picky about the flags it 
passes to the linker], etc.  But this should be RARE.

If you want #3, then the *developer* should specify a .def file as one 
of the library's dependencies (perhaps conditional on cygwin/mingw). End 
of story.

And --export-symbols/--expoprt-symbols-regex should continue to be 
handled on cygwin exactly as it is handled on other unixes: strip or 
don't strip.  It shouldn't affect the "windowsish" exporting process. 
That's the job of --exclude-[lib|symbobs] and .def file, if such 
functionality is needed.

But apparently Tor Lilqvist disagrees:

Now, having said all of that, it appears that this is EXACTLY what the 
glib folks have done.  They explicitly add "-export-symbols glib.def" IF 
and ONLY IF OS_WIN32 (eg. not for cygwin, which has PLATFORM_WIN32 true, 
but OS_WIN32 false).  IMO, that is a *misunderstanding* of what 
--export-symbols does in CURRENT libtool, even if it was correct for OLD 
libtool -- but Tor Lilqvist [the gtk on windows dude] uses a forked 
version of libtool anyway, so...

As it happens, they've managed to outsmart the auto-export code when 
they try this OLD libtool usage with a NEW libtool program.  Auto-export 
ONLY happens if BOTH of the following are true
   1) there is no .def file
   2) there are no explicit __declspec(dllexport) decorations on any 
symbols.

1) is true, because the .def file is only used for OS_WIN32 (not 
cygwin).  However, 2) is false, because there are five symbols that are 
explicitly decorated:
   g_io_watch_funcs  in glib/giowin32.c
   g_timeout_funcs   in glib/gmain.c
   g_idle_funcs      in glib/gmain.c
   g_ascii_table     in glib/gstrfuncs.c
   g_thread_functions_for_glib_use   in glib/gthread.c

These are the only DATA exports from glib that are "complex" (see 'info 
ld' and search on 'auto-import').

So, the "right thing" on cygwin is one of these three things:

include the .def file -- but as explained exhaustively above, you can't 
do it the way Tor does, the (new) "real" libtool doesn't use 
--export-symbols the way he thinks (although the old libtool did).  So, 
you'd have to conditionally include it as a part of glib_la_SOURCES. But 
THAT doesn't work EITHER -- because libtool says "what's this random 
file doing in my argument list, it's not a .lo, it's not .la,  throw it 
out"  So, Tor's solution is the only one that can work -- but it 
requires that $archive_expsyms_cmd be fixed up etc etc... (*)

or

add --export-all-symbols as one of the glib_la_LDFLAGS switches (this 
will re-enable the auto-export feature of ld, even though SOME symbols 
are explicitly decorated).  However, libtool does not pass 
"--export-all-symbols" thru to the linker, so this might require a patch 
to libtool.  Probably not the best way to do this.

or

remove the explicit decorations on those five variables, and rely on the 
  runtime-pseudo-relocs feature of binutils.  This won't work until 
cygwin-1.3.18 is released; it's a very new feature.  And 
--enable-runtime-pseudo-relocs is not the default for ld yet, so you 
need to add that flag to LDFLAGS...but libtool refuses to pass that 
switch to the linker!  So this won't work until (a) the new cygwin 
kernel comes out, and (b) --enable-runtime-pseudo-relocs becomes the 
default on cygwin's ld.  (b) won't happen for quite some time, yet.

-----

So, it appears that that the only way to address this in the short term 
is...restore some form of the old libtool-1.4.x code for 
$archive_expsym_cmd.

Fixup the official libtool so that IF '--export-symbols <FILENAME>' is 
specified, then on cygwin/mingw assume that the file contains a 
.def-format specification, and $archive_expsyms_cmd includes it as "just 
one of the object files" and doesn't use the --retain-symbols-file at 
all.  (On other platforms, go ahead and use --retain-symbols-file, etc). 
  This means that it would be up to the developer to do the following, 
IF he wants to restrict the list of export symbols on both unix and windows:
   1) provide foo.exp which is just a list of symbols
   2) provide foo.def which is a def-format list of symbols, with DATA 
specifiers, etc
   3) conditionally add "--export-symbols foo.exp" on unix, and 
"--export-symbols foo.def" on windows.

This would require all of the changes to $export_symbols_cmds described 
above, too, so that libtool's --export-symbols-regex switch could still 
work (currently, that is hopelessly broken on cygwin).

Now that I've thought about it, I believe that this IS what should 
happen in the libtool codebase.  It seems that some of the "old stuff" 
that was kept around for dlltool and was recently removed, should be 
retained so that the new-and-improved, 
use-auto-import-as-much-as-possible libtool can properly handle corner 
cases like these.

But, that's a separate issue from "how to get glib working on cygwin". 
IF all of those changes were in the current libtool, then yeah -- do it 
that way.  But they are not (yet).  The long-term solution is to remove 
the __declspec decorators on those five symbols when __CYGWIN__, and 
rely on --enable-runtime-pseudo-relocs once it becomes default, and on 
the latest cygwin kernel which should be out in a few days.  (If you 
wanted to try this method out BEFORE --enable-runtime-pseudo-relocs 
becomes the default, the only way to pass that flag thru libtool appears 
to be create a shell script and pretend that it is your compiler:

--gcc--
/path/to/real/gcc -Wl,--enable-runtime-pseudo-relocs $*
-------

Because libtool is REALLY picky about what it passes thru to the linker. 
  Sheesh!

Eventually pseudo-relocs will be the default on cygwin -- and maybe 
mingw -- but not for a number of months, I'm sure.)

Anyway, thanks for bringing up this issue; it appears that the current 
libtool is not quite ready for absolutely every possible use that people 
want...Looks like there are a few things I need to do over Christmas 
break... :-)

--Chuck


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/