I've been using garden for a year or so at this point with no problem. Today when I try to run garden deploy I receive the following: ✖ providers → Failed processing resolve provider kubernetes (took 0.76 sec). This is what happened: ──────────────────────────────────────────────────────────────────────────────── Unable to connect to Kubernetes cluster. Got error: Request failed with response code 401: Kubernetes API: https://rancher.XXXXXXX.com/version. Message from the API: Unauthorized ──────────────────────────────────────────────────────────────────────────────── Failed resolving one or more providers: - kubernetes We're using a self hosted rancher cluster, the only reference I've seen to this error was regarding using an azure cluster.

Which garden version are you experiencing this with? Did you update it before the issue appeared?

I upgraded to 0.13.20 last week, it was working before and after the upgrade.

I see, so it's probably not related to the upgrade then. I was asking since we updated the kubernetes library in 0.13.20 together with some other changes, so it seemed like a good candidate for a root cause.

I feel like I'm postive that it was working post upgrade, I can attempt to downgrade to 0.13.19 real quick.

That would be good to verify

in progress, the download is going rather slow, I'll let you know in a few minutes. thanks!

Alright, that's good to know. Is there anything special about how kubernetes authenticates with rancher here?

not that I know of, but I'm certainly not an expert here. I'd be happy to do any troubleshooting you need.

Do you know if the auth is happening via a proxy?

sure I'll give it a try here in a little bit. I don't believe we're using a proxy for auth. I'm looking at my kubeconfig, and each server context has a server reference pointed directly to that server. If there's something else I should look at let me know.

@thankful-area-26035 thank you for checking that! Can you send us your Kubeconfig, with the sensitive values redacted?

No results frmo the env / grep. My kubeconfig is attached. The error occurs when I try to deploy to the den1-kube-prod or dal1-kube-prod clusters. I just confirmed that I can run a deploy to my local docker-desktop cluster running 0.13.20. https://cdn.discordapp.com/attachments/1176206279438516254/1176914758805496020/kubeconfig.txt?ex=65709a48&is=655e2548&hm=4dfc4fc9acdc9f6288992ecf8ca5b474fb8488941b7466868fcce237daa49d6b&

We've set up our own rancher but so far have still been unable to reproduce this on our side. @big-spring-14945 had a theory that this might also be related to SNI, which we might be able to confirm with a simple change. Could you try in your cluster config to add the key

tls-server-name: [RANCHER PUBLIC HOST]

and see if it works afterwards with the

edge-bonsai

version? So the config would look something like this

apiVersion: v1
clusters:
- cluster:
    server: https://[RANCHER PUBLIC HOST]/k8s/clusters/c-q7kkb
    tls-server-name: [RANCHER PUBLIC HOST]
  name: dal1-kube-prod

Tim, I upgraded to the latest edge-bonsai and can confirm it fixed my issue. Thank you very much for the help!

This has now been released as 0.13.21 so you won't have to stay on the edge release any longer 🙂