Testers needed: New passwd/group handling in Cygwin

Corinna Vinschen corinna-cygwin@cygwin.com
Thu Feb 13 14:44:00 GMT 2014


Hi folks,


this week I applied the first incarnation of the new passwd/group
handling code to the Cygwin repository and after fixing a crash which
manifested in Denis Excoffier's network, I think we're at a point
which allows to push this forward.

This is a pretty intrusive change, in need of some serious testing, so
I'd like to ask for volunteers.  The latest 2014-02-13 snapshot from
http://cygwin.com/snapshots/ contains the changes, including the latest
bugfix.

If you're interested in helping, please read on.  It's a long text, but
I feel it's necessary to explain how this works, so you get a grasp of
what to look out for.  In the long run, this is the basis for the
documentation to go into the User's Guide.

----------------------------------------------------------------------
Please note:

If you have questions or comments, please do *NOT* full quote this mail!
Just quote the important text snippet and ask away.

Thanks for considering.
----------------------------------------------------------------------

Did I mention that this mail is *really* long?  Sorry about that, but I
don't see how to explain all the details otherwise.  Fetch yourself some
drink, preferredly some strong coffee or tea.

Oh, and there be footnotes[0].

===========================
So, what is this all about?
===========================

For as long as Cygwin has existed, it has stored user and group
information in /etc/passwd and /etc/group files.  Under the assumption
that these files would never be too large, the first process in a
process tree, as well as every execing process within the tree would
parse them into structures in memory.  Thus every Cygwin process would
contain an expanded copy of the full information from /etc/passwd and
/etc/group.

This approach has a few downsides.  One of them is that the idea to have
always small files is flawed.  Another one is that reading the entire
file is most of the time entirely useless, since most processes only
need information on their own user and the primary group.  Last but not
least, the passwd and group files have to be maintained separately from
the already existing Windows user databases, the local SAM and Active
Directory.

On the other hand, we have to have a mapping between Windows SIDs and
POSIX uid/gid values (for how this works in Cygwin so far, see [1]), so
we rely on some mechanism to convert SIDs to uid/gid values and vice
versa.

Microsoft's "Services for UNIX" (SFU) (which are deprecated since
Windows 8/Server 2012) never used passwd/group files.  Rather, SFU used
a fixed, computational mapping between SIDs and POSIX uid/gid.  It
allows to generate uid/gid values from SIDs and vice versa.  The
mechanism is documented, albeit in a confusing way and spread over
multiple MSDN articles.  For my changes, I took the liberty to clone the
mapping, with a tiny difference for backward compatibility with existing
Cygwin applications.

===================================
Yes, yes, but how does it work now?
===================================

You *are* comfortable with the concept of SIDs and RIDs, right?  If not,
please read [1] again.  I'll wait...  Finished?  Ok, let's start.

Cygwin's new mapping between SIDs and uid/gid values works in two ways.

- Read /etc/passwd and /etc/group files, like before, mainly for
  backward compatibility.

- If no files are present, or if an entry is missing in the files,
  ask Windows.

At least, that's the default behaviour now.  It will be configurable
using a file /etc/nsswitch.conf, but I'm getting ahead of myself.
Let's stick to the default for now.

If files are present, they will be scanned on demand as soon as a
mapping from SIDs to uid/gid or user names is required.  The new
mechanism will never read the entire file into memory, but only scan
for the requested entry and cache this one in memory[2].

If no entry is found, or no passwd or group file was present, Cygwin
will ask the OS.

  Note:  If the first process in a Cygwin process tree determines
         that no passwd or group file is present, no other process
	 in the entire process tree will try to read the files later
	 on.  This is done for self-preservation.  It's kind of bad
	 if the uid or gid of a user changes during the ride.

So if we've drawn a blank reading the files, we're going to ask the OS.
First thing, we ask the local machine for the SID or the user name. 
The OS functions (LookupAccountSid/LookupAccountName[3]) are kind of
intelligent.  They have all the stuff built in to ask for any account
of the local machine, the Active Directory domain of the machine,
the Global Catalog of the forest of the domain, as well as any
trusted domain of our forest for the information.  One OS call and we're
practically done.  Except...

...the calls only return the mapping between SID, account name and the
account's domain.  We don't have a mapping to POSIX uid/gid and we're
missing information on the user's home dir and login shell.  Let's
discuss the SID<=>uid/gid mapping first.

Here's how the SID<=>uid/gid mapping works.  If you're fuzzy on the
existing well-known SIDs, see [4].

- Well-known SIDs in the NT_AUTHORITY domain of the S-1-5-RID type,
  or aliases of the S-1-5-32-RID type are mapped to the uid/gid value
  RID[5].
  
  Examples:

    "SYSTEM" S-1-5-18     <=> uid/gid: 18
    "Users"  S-1-5-32-545 <=> uid/gid: 545

- Other well-known SIDs in the NT_AUTHORITY domain (S-1-5-X-RID):

    S-1-5-X-RID           <=> uid/gid: 0x1000 * X + RID

  Example:

    "NTLM Authentication"
    S-1-5-64-10           <=> uid/gid: 0x4000A == 262154

- Other well-known SIDs:

    S-1-X-Y               <=> uid/gid: 0x10000 + 0x100 * X + Y

  Example:

    "LOCAL" S-1-2-0       <=> uid/gid: 0x10200 == 66048
    "Creator Group"
    S-1-3-1               <=> uid/gid: 0x10301 == 66305

- Logon SIDs:

    The own LogonSid is converted to the fixed uid 0xfff == 4095 and
    named "CurrentSession".
    Any other LogonSid is converted to the fixed uid 0xffe == 4094
    and named "OtherSession".

- Mandatory Labels:

    S-1-16-RID            <=> uid/gid: 0x60000 + RID

  Example:

    "Medium Mandatory Level"
    S-1-16-8192           <=> uid/gid: 0x62000 == 401408

- Accounts from the local machine's user DB (SAM):

    S-1-5-21-X-Y-Z-RID    <=> 0x30000 + RID

  Example:

    "Administrator"
    S-1-5-X-Y-Z-500       <=> uid/gid: 0x301f4 == 197108

- Accounts from the machine's primary domain:
  
    S-1-5-21-X-Y-Z-RID    <=> 0x100000 + RID

  Example:

    "Domain Users"
    S-1-5-X-Y-Z-513       <=> 0x100201 == 1049089

- Accounts from a trusted domain of the machine's primary domain:

    S-1-5-21-X-Y-Z-RID    <=> trustPosixOffset(domain) + RID

  Hang on... "trustPosixOffset"?  Uh, yes.  This value exists in Windows
  domains already since before Active Directory days.  What happens is
  this.  If you create a domain trust between two domains, a
  trustedDomain entry will be added to both databases.  It describes
  how *this* domain trusts the *other* domain.  One attribute of a trust
  is a 32 bit value called "trustPosixOffset"  For each new trust,
  trustPosixOffset will get some automatic value.  In recent AD domain
  implementations, the first trusted domain will get trustPosixOffset
  set to 0x80000000.  Following domains will get lower values.  Also,
  the domain admins are allowed to set the trustPosixOffset value for
  each trusted domain to some arbitrary 32 bit value, thus allowing any
  kind of collisions between the trustPosixOffsets of domains.  Nice,
  isn't it?  Anyway, as the user of this value, we have to *trust*
  the domain admins to set trustPosixOffset to sensible values, or
  to leave it alone.

  So, for the first (or only) trusted domain of your domain, the
  automatic offset is 0x80000000.  An example for a user of that trusted
  domain is:

    S-1-5-X-Y-Z-1234         <=> uid/gid 0x800004d2 == 2147484882

  There's only one problem with this approach.  Assuming you're
  running in the context of a local user on a domain member machine.
  Local users don't have the right to fetch this kind of domain
  information from the DC, they'll get permission denied.  In this
  case Cygwin will fake a, mostly, sensible trustPosixOffset value
  for this session.

- Local accounts from another machine in the network:

  There's no SID<=>uid/gid mapping implemented for this case.  The
  problem is, there's no way to generate a bijective mapping.  There's
  no central place which keeps an analogue value of the trustPosixOffset,
  and there's the additional problem that the LookupAccountSid and
  LookupAccountName functions cannnot resolve the SIDs, unless they know
  the name of the machine this SID comes from.  And even then it will
  probably suffer a "Permission denied" when trying to ask the machine
  for its local account.

  SFU just prints the account RID in this case, Cygwin maps the account
  to the fake accounts "Unknown+User"/"Unknown+Group" with uid/gid -1.

Ok, now we have a semi-bijective mapping between SIDs and POSIX
uid/gid values, but, given that we have potentially users and groups in
different domains having the same name, how do we uniquely differ
between them by name?  Well, we can do that by making their names
unique in a per-machine way.  Dependent on the domain membership of
the account, and dependent of the machine being a domain member or not,
the user and group names will be generated using a domain prefix and
a separator character between domain and account name.  The default
separator character is the plus sign, '+', as in SFU.

- Well-known SIDs will have the separator character prepended:

    "+SYSTEM", "+LOCAL", "+Medium Mandatory Level", ...

- If the machine is no domain member machine, only local accounts
  can be resolved into names, so for ease of use, just the account names
  are used as Cygwin user/group names:

    "corinna", "bigfoot", "None", ...

- If the machine is a domain member machine, all accounts from
  the primary domain of the machine are mapped to Cygwin names
  without domain prefix:

    "corinna", "bigfoot", "Domain Users", ...

  while accounts from other domains are prepended by their domain:

    "DOMAIN1+corinna", "DOMAIN2+bigfoot", "DOMAIN3+Domain Users", ...

- Local machine accounts of a domain member machine get a Cygwin user
  name the same way as accounts from another domain:  The local machine
  name gets prepended:

    "MYMACHINE+corinna", "MYMACHINE+bigfoot", "MYMACHINE+None", ...

- If LookupAccountSid fails, Cygwin checks the accounts against the
  known trusted domains.  If the account is from one of the trusted
  domains, an artificial account name is created.  It consists of
  the domain name, and a special name created from the account RID:

    "MY_DOM+User(1234)", "MY_DOM+Group(5678)"

  Otherwise we know nothing about this SID, so it will be mapped to
  the fake accounts "Unknown+User"/"Unknown+Group" with uid/gid -1.

Any questions?  No?  Ok, let's have a short break and then, next point.

[...5 minutes commercials...]

==========================================
Cygwin user names, home dirs, login shells
==========================================

"That's all very interesting" you think (if you're still awake, that
is), "but passwd entries consist of more than just uids.  What's up
with that?"

Obviously, if you don't maintain passwd and group files, you need
to have a way to maintain the other fields of a passwd entry as well.
Three things come to mind:

- You want to use a Cygwin user name different from your Windows
  user name.

- You want a home dir different from the default /home/$USER.

- You want to specify a different login shell than /bin/sh.

How this is done depends on your account being a domain account or a
local account.  Let's start with the default.  Assuming your Windows
account name is "bigfoot" and your domain is "MY_DOM".  Your default
passwd entry in absence of anything I'll describe below looks like this:

  bigfoot:*:<uid>:<gid>:U-MY_DOM\bigfoot,S-1-5-....:/home/bigfoot:/bin/sh

or, if your account is from a different domain than the primary domain of
the machine:

  MY_DOM+bigfoot:*:<uid>:<gid>:U-MY_DOM\bigfoot,S-1-5-....:/home/bigfoot:/bin/sh

Yes, the default homedir is still /home/bigfoot.

If your account is a domain account:

  Either create an /etc/passwd and/or /etc/group file with entries for
  your account and use that, just as before.

  Or, Cygwin will utilize the posixAccount/posixGroup attributes per RFC
  2307[6].  These attributes are by default available in Active Directory
  since Windows Server 2003 R2.  They are "not set", unless utilized by
  the (deprecated since Server 2012 R2) Active Directory "Server for
  NIS" feature.  The user attributes utilized by Cygwin are:

    uid                If set, will be used as Cygwin user name.
    uidNumber          See next section.
    gecos              Content will be added to the pw_gecos field.
    unixHomeDirectory  If set, will be used as Cygwin home directory.
    loginShell         If set, will be used as Cygwin login shell.

  The group attributes utilized by Cygwin are:

    cn                 If set, will be used as Cygwin group name.
    gidNumber          See next section.

  Apart from power shell scripting or inventing new CLI tools, these
  attributes can be changed using the "Attribute Editor" tab in the user
  properties dialog of the "Active Directory Users and Computers"
  MMC snap-in.  Alternatively, if the "Server for NIS" administration
  feature has been installed, ther will be a "UNIX Attributes" tab which
  contains the required fields, except for the gecos field, which isn't
  really important anyway.  Last resort is "ADSI Edit".

  The primary group of a user is always the Windows primary group set in
  Active Directory and can't be changed.

If your machine is not a domain member machine or your account is a
local account for some reason:

  Either create an /etc/passwd and/or /etc/group file with entries for
  your account and use that, just as before.

  Or enter the information into the "Comment" field of your local user
  entry.  In the "Local Users and Groups" MMC snap-in it's called
  "Description".

  You can utilze this field even if you're running a "home edition" of
  Windows, using the command line.  The "net user" command allows to set
  all values in the SAM, even if the GUI is crippled.

  A Cygwin SAM comment entry looks like this:

    <cygwin key="value" key="value" [...] />

  The supported keys are

    name="value"      Sets the Cygwin user name to value.

    home="value"      Sets the Cygwin home dir to value.

    shell="value"     Sets the Cygwin login shell to value.

    group="value"     Sets the Cygwin primary group of the account
                      to value, provided that the user *is* already
                      a member of that group.  This allows to
                      override the default "None" primary group for
                      local accounts.
		      One nice idea here is, for instance group="Users".
		      This is the *Windows* name of the group, not the
		      Cygwin name, assuming they differ.

    unix="value"      Sets the NFS/Samba uid of the user to the decimal
                      value.  See the next chapter.

  The <cygwin .../> string can start at any point in the comment, but
  you have to follow the rules:

  - It starts with "<cygwin " and ends with "/>".
  - The "cygwin" string and the key names have to be lowercase.
  - No spaces between key and "value", just the equal sign.
  - The value must be placed within double quotes and it must not
    contain a double quote itself.  The double quotes are required
    for the decimal values as well!

  CMD example:

    net use corinna /comment:"<cygwin home=\"/home/foo\"/>"

  Bash example (use single quotes):

    net use corinna /comment:'<cygwin home="/home/foo"/>'

  For changing group comments, use the `net localgroup' command.  The
  supported key/value pair for groups are

    name="value"      Sets the Cygwin group name to value.

    unix="value"      Sets the NFS/Samba gid of the group to the
                      decimal value.  See the next chapter.


=============================
NFS and Samba account mapping
=============================

If you're using Microsoft's NFS client to access UNIX shares, like me,
you might have been bothered by each file on the share seemingly being
yours.  The reason for this is that Microsoft's NFS client does not map
the uid/gid values on the NFS shares to SIDs.  There's no such thing as
a (fake) security descriptor returned to the application.  Rather, via
an undocumented API you can fetch RFC 1813 compatible NFSv3 stat
information from the share[7].  This is what Cygwin is using to show
stat information for files on NFS shares.

The problem is this.  While all other information in this stat record,
like timestamps, or file size etc., can be used by Cygwin, Cygwin had no
way to map the values of the st_uid and st_gid members to a Windows SID.
So it just faked the file owner info and claimed that it's you.

However, SFU has, over time, developed multiple methods to map UNIX
uid/gid values on NFS shares to Windows SIDs.  You'll find the full
documentation of the mapping methods in [8].

Cygwin now utilizes the RFC 2307 mapping for this purpose.  This is most
of the time provided by an AD domain, but it could also be a standalone
LDAP mapping server.  Per RFC 2307, the uid is in the attribute
uidNumber.  For groups, the gid is in the gidNumber attribute.

When Cygwin stat's files on an NFS share, it asks the mapping server via
LDAP in two different ways, depending on the role of the mapping server.

- If the server is an AD domain controller, it asks for an account with
  uidNumber attribute == st_uid field of the stat record returned by
  NFS.  If an account matches, AD returns the Windows SID, so we have an
  immediate mapping from UNIX uid to a Windows SID, if the user account
  has a valid uidNumber attribute.  For groups, the method is the same,
  just that Cygwin asks for a group with gidNumber attribute == st_gid
  field of the stat record.

- If the server is a standalone LDAP mapping server Cygwin asks for
  the same uidNumber/gidNumber attributes, but it can't expect that
  the LDAP server knows anything about Windows SIDs.  Rather, the
  mapping server returns the account name.  Cygwin then asks the DC for
  an account with this name, and if that succeeds, we have a mapping
  between UNIX uid/gid and Windows SIDs.
  
The mapping will be cached for the lifetime of the process, and inherited
by child processes.

And what about Samba?

A fully set up Samba with domain integration is running winbindd to
map Window SIDs to artificially created UNIX uids and gids, and this
mapping is transparent within the domain, so Cygwin doesn't have to do
anything special.

However, setting up winbindd isn't for everybody, and it fails to map
Windows accounts to already existing UNIX users or groups.  In contrast
to NFS, Samba returns security descriptors, but unmapped UNIX accounts
get special SIDs:

- A UNIX user account with uid X is mapped to the Windows SID S-1-22-1-X.

- A UNIX group account with gid X is mapped to SID S-1-22-2-X.

As you can see, even though we have SIDs, they just reflect the actual
uid/gid values on the UNIX box in the RID value.  It's only marginally
different from the NFS method, so... why not just do the same rote as
for NFS?

That's what Cygwin will do.  If it encounters a S-1-22-x-y SID, it
will perform the same RFC 2307 mapping as for NFS shares.

Hang on.  What about home users without any Windows domain or LDAP
server per RFC 2307, but with a Linux machine running Samba?

Not all is lost.  If you're a user on a standalone machine, you
can add this information to your SAM account.  Assuming the uid of
your Linux user account is 505 and the gid of your primary group is,
say, 100, just add the values to your SAM user and group accounts.

The following example assumes you didn't already add something else
to the comment field.

To your user's SAM comment (remember: called "Description" in the GUI),
add:

  <cygwin group="Users" unix="505"/>

To the user's group SAM comment add:

  <cygwin unix="100"/>

This should be sufficient to work on your Samba share and to see
all files owned by your Linux user account as your files.


======================================================================
The /etc/nsswitch.conf file
===========================

Last, but not least, let's talk about a way to configure how the
stuff works on your machine.  On Linux and some other UNIXy OSes,
we have a file called /etc/nsswitch.conf[9].  One part of it is to
specify how the passwd and group entries are generated.  That's
what Cygwin now provides as well.

The /etc/nsswitch.conf file is optional.  If you don't have one,
Cygwin uses sensible defaults.

  Note:  The /etc/nsswitch.conf file is read exactly once by the first
         process of a Cygwin process tree.  If there was no
         /etc/nsswitch.conf file when this first process started, then
         no other process in the running Cygwin process trees will try
         to read the file.  If you create or change /etc/nsswitch.conf,
         make sure to stop and restart all Cygwin processes to pick up
         the change.

So, what mischief can we perform with /etc/nsswitch.conf?  To explain,
lets have a look into an /etc/nsswitch.conf file set up to all default
values:

  # /etc/nsswitch.conf
  passwd: files db
  group:  files db

  db_prefix:    auto
  db_separator: +
  db_cache:     yes

The first line, starting with a hash '#' is a comment.  The hash
sign starts a comment, just as in shell scripts.  Everything up to
the end of the line is ignored.  So this:

  foo:  bar # baz

means, for the entry foo, do bar, ignore everything after the hash sign,
so baz is only a comment.

The other lines define the available settings.  The first word up to a
colon is a keyword.  Note that the colon *must* follow immediately after
the keyword.  This is a valid line:

  foo: bar

This is not valid:

  foo  :  bar

Apart from this restriction, the reminder of the line can have as
may spaces and TABs as you like.  This is a valid line:

        foo:                       bar                baz

Now let's have a look at the available keywords and settings.

The two lines starting with the keywords "passwd" and "group" define
where Cygwin gets its passwd and group information from.  "files" means,
fetch the information from the corresponding file in the /etc directory.
"db" means, fetch the information from the Windows account databases,
the SAM for local accounts, Active Directory for domain account.
Examples:

  passwd: files

Read passwd entries only from /etc/passwd.

  group: db

Read group entries only from SAM/AD.

  group: files # db

Read group entries only from /etc/group ("db" is ignored due to the
preceding hash sign).

  passwd: files db

Read passwd entries from /etc/passwd.  If a user account isn't found,
try to find it in SAM or AD.  This is the default for both, passwd
and group information.

  group: db files

This is a valid entry, but the order will be ignored by Cygwin.  If
both, files and db are specified, Cygwin will always try the files
first, then the db.

The remaining entries define certain aspects of the Windows account
database search.  "db_prefix" determines how the Cygwin user or group
name is created:

  db_prefix: auto

    This is the default.  If your account is from the primary domain of
    your machine, or if your machine is a standalone machine (not a domain
    member), your Cygwin account name is just the Windows account
    name.
    If your account is from another domain, or if you're logged in as
    local user on a domain machine, the Cygwin username will be the
    combination of Windows domainname and username, with the separator
    char in between:

      MY_DOM+username      (foreign domain)
      MACHINE+username     (local account)
    
    Builtin accounts have just the separator char prepended:

      +LOCAL
      +Users

    Unknown accounts on NFS or Samba shares (that is, accounts which
    cannot be mapped to Windows user accounts via RFC 2307) get a
    Cygwin account name consisting of the artificial domains "UNIX_User"
    or "UNIX_Group" and the uid/gid value, for instance:

      UNIX_User+0          (root)
      UNIX_Group+10        (wheel)

  db_prefx: primary

    Like "auto", but primary domain accounts will be prepended by
    the domainname as well.

  db_prefix: always

    All accounts, even the builtin accounts, will have the domain
    name prepended:

      BUILTIN+Users

    As a special case, if the Cygwin account name differs from the
    Windows account name, it will be prepended by the artificial domain
    name "Posix_User" or "Posix_Group" if db_prefix is set to "always":

      Posix_User+cygwin_user_name
      Posix_Group+cygwin_group_name

"db_separator" defines the spearator char used to prepend the domain
name to the user or group name.  The default is '+':

    MY_DOM+username

With "db_separator", you can define any ASCII char except space,
tab, carriage return, line feed, and the colon, as separator char.
Example:

  db_separator: \

    MY_DOM\username

"db_cache" allows to specify if user and group data should be cached in
the process at all or not.  Default is "yes".  "no" is an experimental
feature.  The idea is this.  Assuming you're working in a volatile
domain environment, and you have long-running processes like sshd.
You might want sshd to recognize changes in the existing user entries
as soon as they occur.  Caching user data in sshd *might* not be helpful
in such a scenario.  However, the user data which gets changed a lot
is typically *not* the data required for login purposes, except for
the password, and that doesn't matter in Cygwin's case.  So take this
feature with a grain of salt.

==========
Footnotes:
==========

[0] Not dragons.  Just a test.

[1] http://cygwin.com/cygwin-ug-net/ntsec.html

[2] This may change.  Right now the file is read in 32K chunks, but
    we could easily read the file in 64K chunks and, if we find the
    file is < 64K anyway, just cache the entire bunch, like before.
    Not implemented yet, but something to keep in mind.

[3] http://msdn.microsoft.com/en-us/library/windows/desktop/aa379166%28v=vs.85%29.aspx
    http://msdn.microsoft.com/en-us/library/windows/desktop/aa379159%28v=vs.85%29.aspx

[4] http://support.microsoft.com/kb/243330

[5] This is where Cygwin differs from SFU.  The reason is that we need
    the old uid/gid values for backward compatibility.  There are Cygwin
    packages (cron, for instance) who rely on the fact that the uid of
    SYSTEM is 18.  In SFU, these accounts get mapped like the other
    built in SIDs.

[6] https://tools.ietf.org/html/rfc2307

[7] https://tools.ietf.org/html/rfc1813

[8] http://msdn.microsoft.com/en-us/library/cc980032.aspx

[9] http://linux.die.net/man/5/nsswitch.conf

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://cygwin.com/pipermail/cygwin/attachments/20140213/92882fd0/attachment.sig>


More information about the Cygwin mailing list