Python for Windows reports wrong local time when run under Cygwin on Europe/Moscow TZ
Brian Inglis
Brian.Inglis@SystematicSw.ab.ca
Sat Jun 12 16:34:52 GMT 2021
On 2021-06-11 12:05, Ken Brown via Cygwin wrote:
> On 6/11/2021 1:33 PM, Brian Inglis wrote:
>> On 2021-06-10 13:50, Ken Brown via Cygwin wrote:
>>> On 6/10/2021 2:31 PM, Brian Inglis wrote:
>>>> On 2021-06-10 08:57, Ken Brown via Cygwin wrote:
>>>>> On 6/9/2021 10:36 PM, Brian Inglis wrote:
>>>>>> On 2021-06-09 16:31, Keith Thompson via Cygwin wrote:
>>>>>>> [Sorry if the threading is messed up. I don't subscribe, so I'm
>>>>>>> constructing this message from the web interface. It should at
>>>>>>> least
>>>>>>> show up under the correct subject.]
>>>>>>>
>>>>>>> Brian Inglis wrote:
>>>>>>>> On 2021-06-08 14:03, Mike Kaganski via Cygwin wrote:
>>>>>>>>> On 08.06.2021 16:04, L A Walsh wrote:
>>>>>>>>>> You might ask on a python list if anyone else has experienced
>>>>>>>>>> something similar with python or any other program. I'm
>>>>>>>>>> fairly sure
>>>>>>>>>> that neither MS nor cygwin design their OS with python in mind
>>>>>>>>>> and
>>>>>>>>>> that it is python that is interacting funny when running under
>>>>>>>>>> some
>>>>>>>>>> merge of both. Have you asked the python people about this
>>>>>>>>>> problem?
>>>>>>>>>> What did they suggest?
>>>>>>>>>
>>>>>>>>> FTR: filed https://bugs.python.org/issue44352.
>>>>>>>>
>>>>>>>> See Keith Thompson subthread and my reply with suggested fix:
>>>>>>>>
>>>>>>>> https://cygwin.com/pipermail/cygwin/2021-June/248692.html
>>>>>>>>
>>>>>>>> Windows does not recognize zoneinfo time zone identifiers in TZ
>>>>>>>> only
>>>>>>>> base format POSIX TZ strings with three alphabetic character
>>>>>>>> identifiers:
>>>>>>>>
>>>>>>>> https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/tzset?view=msvc-160
>>>>>>>>
>>>>>>>>
>>>>>>>> That assumes US switch date "rules": for all years up to
>>>>>>>> current, or
>>>>>>>> just DST, and whether pre- or post-2007 is unstated!
>>>>>>>>
>>>>>>>> Otherwise it defaults to regional settings, used by Cygwin to
>>>>>>>> map to
>>>>>>>> zoneinfo time zone identifiers, so if Python for Windows could
>>>>>>>> clear TZ
>>>>>>>> before it is read by MSVCRT, it should DTRT.
>>>>>>>>
>>>>>>>> Windows does not recognize expanded POSIX TZ format strings with <>
>>>>>>>> quoted alphanumeric characters, "-", "+", and start and end
>>>>>>>> dates/times:
>>>>>>>>
>>>>>>>> https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#bottom
>>>>>>>>
>>>>>>>>
>>>>>>>> which make them usable outside of the US.
>>>>>>>
>>>>>>> Summary: IMHO Cygwin should adapt its default TZ setting to work
>>>>>>> with Windows.
>>>>>>>
>>>>>>> The suggestion is to modify Python for Windows so it can deal with
>>>>>>> the TZ format used by Cygwin. I haven't used Python for Windows,
>>>>>>> but
>>>>>>> as far as I know it's unrelated to Cygwin; rather it, like
>>>>>>> Cygwin, is
>>>>>>> intended to work on top of Windows. I'm not convinced it's
>>>>>>> appropriate
>>>>>>> to ask Python for Windows to make a change purely for the sake of
>>>>>>> interoperating with Cygwin, which many PfW users presumably aren't
>>>>>>> even using.
>>>>>>>
>>>>>>> I've run into another application that has problems with Cygwin's
>>>>>>> settings of $TZ. It was a internal test application that isn't
>>>>>>> going to change its timezone handling just for this problem.
>>>>>>>
>>>>>>> The ideal solution would be for Windows to recognize TZ values like
>>>>>>> "America/Los_Angeles", but that's not likely to happen any time
>>>>>>> soon.
>>>>>>>
>>>>>>> My suggestion, since Cygwin is supposed to interoperate with
>>>>>>> Windows,
>>>>>>> is one of the following:
>>>>>>>
>>>>>>> - Cygwin should avoid setting TZ to a value that Windows doesn't
>>>>>>> recognize
>>>>>>> (if I set TZ=PST8PDT, everything seems to work correctly); OR
>>>>>>>
>>>>>>> - Cygwin shouldn't set TZ at all by default. (I've updated my
>>>>>>> $HOME/.bash_profile on Cygwin to unset TZ, and Cygwin commands
>>>>>>> seem
>>>>>>> to work correctly with TZ unset); OR
>>>>>>>
>>>>>>> - Cygwin, when invoking a non-Cygwin executable, should first either
>>>>>>> unset TZ or translate it to a format that Windows will recognize.
>>>>>>> I have no idea how difficult that would be.
>>>>>>
>>>>>> Impossible to set Windows TZ usefully as it obeys unstated US DST
>>>>>> rules (like posixrules, perhaps 2007+?), and may have limits on
>>>>>> hour offset magnitudes.
>>>>>> MS libraries are stuck at POSIX 1996 and C 99 subset
>>>>>> compatibility, but non-standard-conformant including which headers
>>>>>> contain definitions:
>>>>>>
>>>>>> https://docs.microsoft.com/en-us/cpp/c-runtime-library/compatibility?view=msvc-160
>>>>>>
>>>>>>
>>>>>> It may be possible to unset TZ when running non-Cygwin programs
>>>>>> (possibly behind a CYGWIN env var setting e.g. winnotz) by adding
>>>>>> TZ= to conv_envvars, and writing new helper functions
>>>>>> env_tz_to_posix to call tzset and env_tz_to_win32 to remove TZ in:
>>>>>>
>>>>>> https://sourceware.org/git/?p=newlib-cygwin.git;f=winsup/cygwin/environ.cc;a=blob
>>>>>>
>>>>>>
>>>>>> What is the opinion on this from both Windows users and Cygwin
>>>>>> patchers?
>>>>>
>>>>> I'm not convinced it's worth the trouble. I haven't seen anyone
>>>>> argue that it's useful for Cygwin to set TZ, and I have seen an
>>>>> argument that it's harmful:
>>>>>
>>>>> https://cygwin.com/pipermail/cygwin/2017-May/232675.html .
>>>>>
>>>>> So I prefer Keith's second suggestion:
>>>>>
>>>>> >> - Cygwin shouldn't set TZ at all by default.
>>>>
>>>> It does so in default startup scripts
>>>
>>> Right, and I'm agreeing with Bruno (in the message cited above) that
>>> the default startup scripts should stop doing that.
>>>
>>>> to get the correct behaviour from Cygwin DLL and programs,
>>>
>>> Can you be more specific? What goes wrong if TZ is not set? I
>>> haven't seen any POSIX or Linux documentation that says it should be
>>> set, and I've just checked on two different Linux distros that it's
>>> not set by default.
See original message:
https://sourceware.org/pipermail/cygwin/2012-January/199548.html
Looks like the tzdb metadata is not fully populated, including zone
abbreviations, though that could be patched to use <%z> like tzdata
defaults where countries use only a single local time zone and no name
(other than COUNTRY time, and maybe Winter Time or Summer Time, as in
many European countries).
>> I would expect that date, ls, etc. would output UTC, or perhaps PST or
>> EST, depending on tzdata builds of Factory (tz -00/unset) and
>> posixrules (Cygwin PST, Debian EST) and use during system setup and
>> startup, unless /etc/timezone and/or /etc/localtime are set, and used
>> during startup, often by systemd, or login by profiles.
>
> No, you can 'unset TZ' and everything works fine. Try it yourself.
It works incorrectly before 2007 because of DST rule changes.
Cygwin effectively uses expanded POSIX rules with a single pair of
transition dates, applied to all years.
Doesn't work for anything other than current time rules, as Cygwin
doesn't (yet?) use historical "Dynamic" DST.
"Dynamic" DST was probably only added to Vista about 2006 because the US
was changing rules in 2007.
See comments by MS TZ/DST blogger, commentator, and tzdb contributor
Matt Johnson-Pint:
https://stackoverflow.com/questions/19838892/dynamic-dst-historical-data
where he suggest the poster uses tzdb instead of trying to fix Windows
Dynamic DST rules!
[I could generate equivalent Windows Dynamic DST reg entries from tzdb,
but tzdb allows relational holiday calendar rules like "May Mon<25" -
the Monday before the 25th May (Victoria Day, CA which mostly coincides
with Memorial Day, US), and generates up to 400 years of entries, which
I don't think the registry would cope with too well!]
For Windows TZ updates see:
https://support.microsoft.com/kb/914387
Support for rule changes from 2010:
http://support.microsoft.com/gp/cp_dst
As a result, it does not have much in the way of history for times past
(see links above and examples below), unlike the tzdb in tzdata package,
which has full history back to 1970, and in some cases, back to before
standard time (Local Mean Time); if you have the tzcode package
installed, try zdump with your local time zone e.g.:
$ zdump -Vc1800,2008 America/New_York
or use -Vc1800,2022 in other countries with more recent changes.
Many zones in the Middle East have up to four changes each year if DST
would be in effect during Ramadan (it is reverted temporarily during
that month), and there are many recent rule changes in most other
countries, even Europe where countries jumped around annually until they
joined or decided to conform with the EU, similar to Canadian provinces
and Mexican states deciding to conform with the US from 2007, although
some zones in both countries changed their minds after that, and some
dropped DST e.g. Yukon CA in 2020!)
From MS comments about Ramadan not appearing in TZ history, with the
implication that MS have kludged current timekeeping to handle a DST
reversion, but may be unable to handle more than one transition a year,
historical file timestamps from earlier in the year will be displayed
incorrectly.
MS are issuing historical corrections for previous years for Windows 10,
but it looks like earlier Windows versions are getting more outdated:
https://docs.microsoft.com/en-us/archive/blogs/dst2007/
https://techcommunity.microsoft.com/t5/Daylight-Saving-Time-Time-Zone/bg-p/DSTBlog
>> How do you set or get useful local time on your systems?
>
> Cygwin takes care of it. If TZ is unset, Cygwin queries Windows via
> GetTimeZoneInformation. See tzgetwintzi in
> winsup/cygwin/tzcode/localtime_wrapper.c.
I was asking about your Linux/BSD systems.
[Dump of Windows tz info for my tz (from Cygwin regtool as it converts
to decimal but does not dump REG_BINARY keys in full only the first
eight bytes, and Windows reg to dump REG_BINARY keys in full):
$ win-reg-tzi.sh
HKLM/SYSTEM/CurrentControlSet/Control/TimeZoneInformation
Bias (REG_DWORD) = 0x000001a4 (420)
DaylightBias (REG_DWORD) = 0xffffffc4 (4294967236)
DaylightName (REG_SZ) = "@tzres.dll,-191"
DaylightStart (REG_BINARY) = 00 00 03 00 02 00 02 00
DynamicDaylightTimeDisabled (REG_DWORD) = 0x00000000 (0)
RealTimeIsUniversal (REG_DWORD) = 0x00000001 (1)
StandardBias (REG_DWORD) = 0x00000000 (0)
StandardName (REG_SZ) = "@tzres.dll,-192"
StandardStart (REG_BINARY) = 00 00 0b 00 01 00 02 00
TimeZoneKeyName (REG_SZ) = "Mountain Standard Time"
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation
Bias REG_DWORD 0x1a4
DaylightBias REG_DWORD 0xffffffc4
DaylightName REG_SZ @tzres.dll,-191
DaylightStart REG_BINARY 00000300020002000000000000000000
DynamicDaylightTimeDisabled REG_DWORD 0x0
RealTimeIsUniversal REG_DWORD 0x1
StandardBias REG_DWORD 0x0
StandardName REG_SZ @tzres.dll,-192
StandardStart REG_BINARY 00000B00010002000000000000000000
TimeZoneKeyName REG_SZ Mountain Standard Time
HKLM/SOFTWARE/Microsoft/Windows NT/CurrentVersion/Time Zones/Mountain
Standard Time
Dynamic DST\ ()
Display (REG_SZ) = "(UTC-07:00) Mountain Time (US & Canada)"
Dlt (REG_SZ) = "Mountain Summer Time"
MUI_Display (REG_SZ) = "@tzres.dll,-190"
MUI_Dlt (REG_SZ) = "@tzres.dll,-191"
MUI_Std (REG_SZ) = "@tzres.dll,-192"
Std (REG_SZ) = "Mountain Standard Time"
TZI (REG_BINARY) = a4 01 00 00 00 00 00 00
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Time
Zones\Mountain Standard Time
Display REG_SZ (UTC-07:00) Mountain Time (US & Canada)
Dlt REG_SZ Mountain Summer Time
MUI_Display REG_SZ @tzres.dll,-190
MUI_Dlt REG_SZ @tzres.dll,-191
MUI_Std REG_SZ @tzres.dll,-192
Std REG_SZ Mountain Standard Time
TZI REG_BINARY
A401000000000000C4FFFFFF00000B0000000100020000000000000000000300000002000200000000000000
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Time
Zones\Mountain Standard Time\Dynamic DST
2006 REG_BINARY
A401000000000000C4FFFFFF00000A0000000500020000000000000000000400000001000200000000000000
2007 REG_BINARY
A401000000000000C4FFFFFF00000B0000000100020000000000000000000300000002000200000000000000
FirstEntry REG_DWORD 0x7d6
LastEntry REG_DWORD 0x7d7
HKLM/SOFTWARE/Microsoft/Windows NT/CurrentVersion/Time Zones/Mountain
Standard Time/Dynamic DST
2006 (REG_BINARY) = a4 01 00 00 00 00 00 00
2007 (REG_BINARY) = a4 01 00 00 00 00 00 00
FirstEntry (REG_DWORD) = 0x000007d6 (2006)
LastEntry (REG_DWORD) = 0x000007d7 (2007)
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Time
Zones\Mountain Standard Time\Dynamic DST
2006 REG_BINARY
A401000000000000C4FFFFFF00000A0000000500020000000000000000000400000001000200000000000000
2007 REG_BINARY
A401000000000000C4FFFFFF00000B0000000100020000000000000000000300000002000200000000000000
FirstEntry REG_DWORD 0x7d6
LastEntry REG_DWORD 0x7d7
# dump of Morocco Dynamic DST
$ reg query "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Time
Zones\Morocco Standard Time\Dynamic DST" /s
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Time
Zones\Morocco Standard Time\Dynamic DST
2007 REG_BINARY
0000000000000000C4FFFFFF0000000000000000000000000000000000000000000000000000000000000000
2008 REG_BINARY
0000000000000000C4FFFFFF000008000000050017003B003B00E703000005000600050017003B003B00E703
2009 REG_BINARY
0000000000000000C4FFFFFF000008000400030017003B003B00E703000005000000050017003B003B00E703
2010 REG_BINARY
0000000000000000C4FFFFFF000008000600010017003B003B00E703000005000600010017003B003B00E703
2011 REG_BINARY
0000000000000000C4FFFFFF000007000600050017003B003B00E703000004000600010017003B003B00E703
2012 REG_BINARY
0000000000000000C4FFFFFF0000090000000500030000000000000000000400000005000200000000000000
2013 REG_BINARY
0000000000000000C4FFFFFF00000A0000000500030000000000000000000400000005000200000000000000
2014 REG_BINARY
0000000000000000C4FFFFFF00000A0000000500030000000000000000000300000005000200000000000000
2015 REG_BINARY
0000000000000000C4FFFFFF00000A0000000500030000000000000000000300000005000200000000000000
2016 REG_BINARY
0000000000000000C4FFFFFF00000A0000000500030000000000000000000300000005000200000000000000
2017 REG_BINARY
0000000000000000C4FFFFFF00000A0000000500030000000000000000000300000005000200000000000000
FirstEntry REG_DWORD 0x7d7
LastEntry REG_DWORD 0x7ed
2020 REG_BINARY
0000000000000000C4FFFFFF0000040000000300030000000000000000000500000005000200000000000000
2021 REG_BINARY
0000000000000000C4FFFFFF0000040000000200030000000000000000000500000003000200000000000000
2022 REG_BINARY
0000000000000000C4FFFFFF0000030000000500030000000000000000000500000002000200000000000000
2023 REG_BINARY
0000000000000000C4FFFFFF0000030000000300030000000000000000000400000004000200000000000000
2024 REG_BINARY
0000000000000000C4FFFFFF0000030000000200030000000000000000000400000002000200000000000000
2025 REG_BINARY
0000000000000000C4FFFFFF0000020000000500030000000000000000000400000001000200000000000000
2026 REG_BINARY
0000000000000000C4FFFFFF0000020000000300030000000000000000000300000004000200000000000000
2027 REG_BINARY
0000000000000000C4FFFFFF0000020000000100030000000000000000000300000002000200000000000000
2028 REG_BINARY
0000000000000000C4FFFFFF0000010000000400030000000000000000000300000001000200000000000000
2029 REG_BINARY
0000000000000000C4FFFFFF0000010000000200030000000000000000000200000003000200000000000000
2018 REG_BINARY
0000000000000000C4FFFFFF00000A0000000400030000000000000000000300000004000200000000000000
2019 REG_BINARY
0000000000000000C4FFFFFF0000010002000100000000000000000000000600000002000200000000000000
]
--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
More information about the Cygwin
mailing list