Re: Adding a new target to newlib

["J. Johnston" <jjohnstn at redhat dot com>]

KJK::Hyperion wrote:
> 
> Hi all. I'm the main developer of POSIX+, a POSIX compatibility layer for
> Windows NT and ReactOS (<http://www.reactos.com/>). Under suggestions from
> other team members, I've decided to evaluate newlib to provide POSIX+ the
> C runtime. Before investing time into the adaption, I need a general
> understanding of a few concepts. I'll bother you this one time, then rely
> on the documentation. Feel free to RTFM the trivial questions. Let's see:
>   - do I need a POSIX-like shell to build newlib? because I'd have a nasty
>     chicken-egg problem then: I can't have a POSIX shell until POSIX+ is
>     compiled, I can't compile POSIX+ until newlib is compiled, and I can't
>     compile newlib without a shell. Even a future self-hosted version is
>     out of question, because the team has gone to great lengths to minimize
>     the requirements to build ReactOS. A possible advantage is that POSIX+
>     will (likely) have only one public, official version, and the only
>     variants will be architecture-specific builds: does this help? will
>     some old-fashioned makefiles do?

Newlib is designed to be built cross-platform.  That means you can
build it on native Unix platforms (including Cygwin) for any supported target.  
Obviously,  you require a set of cross-tools that run on the native platform and generate
code for the target.  The one exception to this is x86-*-linux which requires 
glibc header files when building so it must currently be built natively.  There are
currently supported cross-tools for x86-elf, x86-coff, and x86-pc-cygwin (pe format).

>   - is newlib thread-safe by default, or only for multithread-aware
>     programs? (e.g. *_r variants of standard functions) I'm trying to
>     follow the Single Unix Specification, and it clearly states that all
>     functions are meant to be reentrant and thread-safe, except when
>     otherwise noted

Newlib is not thread-safe by default but provides reentrant versions of
routines which can be used to create a thread-safe application.  For thread-safety,
a number of lock routines must also be supplied.  A number of newlib routines make calls 
to such lock routines, but the default implementation is an empty stub.  For example,
malloc makes calls to __malloc_lock and __malloc_unlock.  If these are provided, then
a threaded application can safely use malloc.  Such a system is also required to
access the environment list.  Recently, a more generic system has
been added (see include/sys/lock.h and libc/posix/telldir.c).  Eventually, we will likely
switch the malloc and environment locks to use the same mechanism so all one needs to
provide are the basic lock constructs.

Newlib's method of reentrancy is to use a reentrancy structure that is passed to _r versions of the routines.  Normally, the non-reentrant versions of
the functions refer to a default _REENT reentrancy structure that is initialized by the main thread.  A thread-aware application
has two choices.  It can ensure that each thread has its own reentrancy structure and
manually call the _r routines or it can set up dynamic reentrancy swapping.  If
__DYNAMIC_REENT__ is defined in sys/config.h, then references to _REENT will actually
call a routine called __getreent().  You can provide this routine and have it access
thread-specific storage based on the thread id.  When you initialize the thread you 
initialize a new reentrancy structure and make its address accessible via thread-specific
storage.  After that, reentrancy is automatic.

>   - POSIX+ defers simple runtime functions (e.g. memcpy, memcmp, strcpy,
>     etc.) to a system-wide runtime library (NTDLL.DLL). Is it dangerous to
>     mix the two runtimes? does it lead to inconsistent behavior? e.g. some
>     of these functions are interdependent in newlib?

Depends which routines you are referring to.  By definition, a simple runtime function
should have no special internals that other functions depend upon.  As long as the
replacements adhere to the C standard, all should be ok.

>   - is it possible to compile newlib into a Windows DLL, or does it depend
>     on POSIX linking semantics? e.g. symbols need not be qualified with a
>     DLL name?

Check out Cygwin.  Cygwin builds a library DLL using parts of newlib plus a number of
additional files in the winsup component.  You can get details about Cygwin at http://sources.redhat.com/cygwin

>   - to what extent is newlib extensible? Specifically:
>      - can the per-process and per-thread data be customized/extended?

Newlib has a system whereby default header files and functions can be overridden
by files of the same name in the sys and machine directories.  In your case, this would 
be similar to what we do in libc/sys/linux.  For example, you could copy the 
libc/include/sys/reent.h file to your sys/xxxx/sys directory and add
whatever you want to it.  You would of course have to provide the current
functionality for routines that use the reentrancy struct.

>      - does asynchronous I/O depend on POSIX threads, and can it be
>        implemented otherwise without breaking something else?

Vanilla newlib is basically ANSI C plus some UNIX extensions.  It is not POSIX and
is missing some things like asynchronous I/O for example.  You will notice
there is work underway to add many POSIX functions to the linux version of newlib.
Some of this work requires specific linux syscalls and so the files are only added
in the libc/sys/linux directory.  Code that can be used by all platforms is placed
in the shared directories (e.g. libc/stdlib, libc/unix libc/string).  I would suggest
discussing how best to implement Asynchronous I/O on NT on the Cygwin mailing lists.

      - NT has a very good heap manager, can it be used in place of the
>        internal one used (I presume) by newlib?

Yes.  There is a MALLOC_PROVIDED flag that can be set to indicate you are providing
your own malloc routines.  Be aware that you will have to provide a _malloc_r
interface for the use of other parts of newlib.

>      - to the NT Rtl, heaps are just another kind of data structures, so a
>        process can have as many heaps as it sees fit (Microsoft suggests,
>        for example, the use of throwaway per-thread heaps for aggressively
>        multi-threaded programs, like ISAPIs, to minimize the context
>        switches under heavy load). Does some other target have similar
>        semantics, whose API I can "recycle"? if not, are there guidelines
>        for new identifiers/prototypes?
>
> Finally, a "political" question: what about licensing? POSIX+ must be
> under the GPL, and I guess that the various newlib licenses don't conflict
> with it. But what about headers? to allow unlimited commercial use of
> ReactOS system libraries, all of our public headers and DEF files (source
> code for import libraries) are under no license, we don't even claim
> copyright. What about newlib's?
>

Newlib has a number of licenses, but the ones you will encounter all fall
under the BSD-style of license which is less restrictive than the GPL or LGPL.  You
are free to use such code as long as the copyright and license information is
preserved.  The shared header files of newlib do not impose any requirements on 
the application using them.


-- Jeff J.