"kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

Sun Jun 14 19:08:45 GMT 2020

On 2020-06-14 12:16, David Karr via Cygwin wrote:
> On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis <
> Brian.Inglis@systematicsw.ab.ca> wrote:
> 
>> On 2020-06-14 09:38, David Karr via Cygwin wrote:
>>> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:
>>>> On 14.06.2020 08:12, David Karr wrote:
>>>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
>>>>>>     On 13.06.2020 20:53, David Karr via Cygwin wrote:
>>>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a while,
>>>>>>> to communicate to our in-house k8s clusters. I often use "kubectl
>>>>>>> exec" to open a shell in a container or directly execute a shell
>>>>>>> command.
>>>>>>> This has worked perfectly fine for a long time.
>>>>>>> A couple of days ago, I discovered that all of these attempts were
>>>>>>> failing with "Upgrade request required".  I hadn't upgraded kubectl
>>>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s upgrade,
>>>>>>> but it's entirely possible.
>>>>>>> A colleague of mine has a very similar desktop configuration
>>>>>>> (Windows 10, Cygwin), and he's not seeing this symptom.
>>>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it shows
>>>>>>> the resulting "curl" command that it runs. I tried that resulting
>>>>>>> command, and it results in the same response. I then tried updating
>>>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the
>> entire resulting "kubectl exec" command line and ran
>>>>>>> it in a "cmd" shell.  No problem at all.  No error.
>>>>>>> I know I haven't provided much useful information yet. I wanted to
>>>>>>> get an initial response before I started providing those diagnostics.
>>>>>>> Is there a clear issue here that I'm not aware of?
>>
>>>>> from where is kubectl coming from ?
>>>>> In cygwin I found only a kubectl.py in the ansible package
>>
>>>>>> It's from here:
>>>>>>
>> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
>>
>>>> so it is NOT a cygwin program.
>>>> If the warning is coming about curl, it is likely
>>>> that using from cygwin you are using the cygwin curl
>>>> and from CMD the windows one
>>>> $ which -a curl
>>>> /usr/bin/curl
>>>> /cygdrive/c/WINDOWS/system32/curl
>>>> $ /cygdrive/c/WINDOWS/system32/curl -V
>>>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
>>>> Release-Date: 2017-11-14, security patched: 2019-11-05
>>>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp
>> smtps
>>>> telnet tftp
>>>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
>>>>
>>>> $ /usr/bin/curl -V
>>>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f zlib/1.2.11
>>>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
>>>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
>>>> Release-Date: 2019-09-11
>>>> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps
>>>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
>>>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6
>>>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP
>>>> TrackMemory UnixSockets
>>>> the support Forum https://discuss.kubernetes.io/
>>>> is probably the most indicate place for guidance
>>
>>> I thought it was obvious that it was not working because it was calling
>> the
>>> Cygwin curl. I wouldn't have posted here if that wasn't obvious to me.
>>> And since I'm well aware of the k8s community, I already posted questions
>>> about this in the appropriate place, before I posted here.
>>> What I was hoping to get here was some indication or thoughts on why a
>>> process using Windows curl doesn't have a problem, but does have a
>> problem
>>> when using Cygwin Curl. This isn't likely something that Cygwin curl is
>>> doing "wrong", it's just that it's doing something different.
>>> If it matters, the following is an elided version of the resulting curl
>>> command:
>>>     curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0
>> (windows/amd64)
>>> kubernetes/9e99141" -H "Authorization: Bearer ..." -H
>>> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H
>>> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H
>>> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H
>>> "X-Stream-Protocol-Version: channel.k8s.io" 'https://
>>>
>> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
>>> I can't tell from the logging what request body it sent. It's possible it
>>> didn't send any.
>>
>> We can't tell, not knowing who you are, and from what you are posting,
>> what may
>> or may not be obvious to you, where you are seeing "Upgrade request
>> required",
>> or where that may be coming from.
>> That you need diagnostic help indicates that, what may appear obvious to
>> you,
>> may not be the case, as it is often our assumptions which lead us astray,
>> and we
>> get daily proof that we are imperfect, which is why most of seek to talk
>> over
>> and explain our issues to inanimate objects or colleagues e.g. talk to the
>> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks and a
>> pelican awarded by former projects.)
>>
>> For help with Cygwin, we need to see *whole* commands and all the output
>> between
>> the shell prompts, preferably with context, in this case including PATH
>> under
>> Cygwin and MS Windows and/or program paths typed or invoked.
>>
>> You may retain privacy and security by substituting variables e.g. env
>> vars $VAR
>> to sanitize sensitive information: to avoid problems I use a script to do
>> so
>> that logs are never written containing sensitive content.
>>
> 
> Acknowledged. The "Upgrade request required" is coming from the kubernetes
> server.  At this point, I think I'm going to need to understand exactly
> what that message means (other threads talking about this didn't make that
> clear). Someone in that community did respond to my question, not with an
> answer, but with a query for more information. I'll see what I can find out.

A short Google search turned up that the message is apparently an HTTP 400 Bad
Request response with Upgrade request message, meaning you need to use the
websocket API to talk to a kubernetes pod:

https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required
https://tools.ietf.org/html/rfc6455
https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]