https://garden.io logo
Join Discord
Powered by
Remote Kubernetes for Garden Github Action
# 🌱｜help-and-getting-started
c
I'm experiencing issues with the official Github Action for Garden when attempting to use the remote kubernetes provider. Given an authenticated environment that is already set to the correct context via 
kubectl config set-context
, and a provider in my Garden configuration hard-coded to the same context I am given the error:
Copy code
Could not read cluster from kubeconfig for context arn:aws:eks:us-west-2:XXXXXXXXXX:cluster/my-cluster-name
I am able to run kubectl commands with no issue in the step prior within the same workflow, so I know with certainty that the context is set properly and that there are no authentication / connection issues. Things that I have tried: - Building as-is with no additional options - Providing an explicit $KUBECONFIG that refers explicitly to the location of the kubeconfig - Providing a base64 encoded version of the kube config via the 
kubeconfig
action param Garden action does not seem to pick up on the default kube config by default. Providing an explicit $KUBECONFIG seems to have no discernible effect whatsoever. Providing a base64 encoded version of the kube config results in: 
error: error loading config file <file> line 6: could not find expected ':'
which indicates to me that there may be a bug in the YAML processing. Is there an existing example of the official Garden Action usage with remote Kubernetes clusters rather than Garden Cloud?
b
@clever-policeman-58407 Yes, this is supposed to work and we are using Garden heavily with EKS internally as well, also with the GitHub Action. The YAML loading error sounds interesting; Could you send the YAML file over to us, with the secrets blanked out of course, just so we can see what it's all about. Sometimes there are YAML 1.1 vs 1.2 compatibility issues How do you generate the Kubeconfig, are you using an AWS CLI command for that? Which version of AWS CLI do you use? Thanks for the reply, we'll look into this 🙏
Which Garden version are you using?
I looked at the code a little bit, and it seems like we throw the error "Could not read cluster from kubeconfig for context" when we can't find the cluster in your kubeconfig. How doues the Kubeconfig YAML look like? In my kubeconfig it looks like something like this:
Copy code
# [...]
contexts:
- context:
    cluster: arn:aws:eks:region:123412341234:cluster/cluster-abcdef
    user: arn:aws:eks:region:123412341234:cluster/cluster-abcdef
  name: arn:aws:eks:region:123412341234:cluster/cluster-abcdef
# [...] remaining config with users: and clusters: matching the referenced names
c
I surmised the same, so after double checking that the cluster was present (and it should be, it's auto-generated by 
aws eks update-kubeconfig
) I made the assumption that it must not be seeing the default kubeconfig file at all and decided to provide it directly with the 
kubeconfig
argument. I can verify that the kubeconfig is sane, but even after providing the same file as a base64 encoded string to 
kubeconfig
it returns the aforementioned 
error: error loading config file <file> line 6: could not find expected ':'
which smells like a YAML parsing bug to me
Here's a redacted version of the file:
Copy code
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <blob>
    server: https://FOO.BAR.us-west-2.eks.amazonaws.com
  name: arn:aws:eks:us-west-2:<aws_id>:cluster/foo-bar-cloud
contexts:
- context:
    cluster: arn:aws:eks:us-west-2:<aws_id>:cluster/foo-bar-cloud
    user: arn:aws:eks:us-west-2:<aws_id>:cluster/foo-bar-cloud
  name: arn:aws:eks:us-west-2:<aws_id>:cluster/foo-bar-cloud
current-context: arn:aws:eks:us-west-2:<aws_id>:cluster/foo-bar-cloud
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-west-2:<aws_id>:cluster/foo-bar-cloud
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - us-west-2
      - eks
      - get-token
      - --cluster-name
      - foo-bar-cloud
      - --output
      - json
      command: aws
Can you confirm whether or not the action is supposed to be able to pick up your kubeconfig from the default 
$KUBECONFIG
?
For context, here's the accompanying 
garden.yml
with the irrelevant parts stripped:
Copy code
apiVersion: garden.io/v1
kind: Project
name: foobar-cloud
defaultEnvironment: local
dotIgnoreFile: .gitignore

variables:
  # Kubernetes namespaces do not allow underscores.
  cluster_name: ${local.env.CLUSTER_NAME || "foobar"}
  ci-branch: ${local.env.CI_BRANCH || "ci-demo"}
  ci-run-number: ${local.env.CI_RUN_NUMBER || 1}
  namespace: "${local.env.CI_BRANCH ? kebabCase(local.env.CI_BRANCH) : local.env.CLUSTER_NAMESPACE || kebabCase(local.username)}"
  dev-domain: foobar.dev

environments:
  - name: local
    defaultNamespace: ${var.namespace}
    variables:
      base-domain: localhost
      base-hostname: localhost
      deploy-target: local
  - name: ci
    defaultNamespace: "${var.namespace}"
    variables:
      base-domain: ${var.dev-domain}
      base-hostname: "${var.namespace}.${var.dev-domain}"
      deploy-target: remote
  - name: remote
    defaultNamespace: ${var.namespace}
    variables:
      base-domain: ${var.dev-domain}
      base-hostname: "${var.namespace}.${var.dev-domain}"
      deploy-target: remote
  - name: ephemeral
    defaultNamespace: ${var.namespace}
    variables:
      base-hostname: localhost
      deploy-target: remote

providers:
  - name: kubernetes
    environments: [remote, ci]
    namespace: ${environment.namespace}
    context: arn:aws:eks:us-west-2:<aws_id>:cluster/foo-bar-cloud
I know that the name for the cluster is correct, because I can use the same command that the Github Workflow does and build to the exact same remote cluster when I run that command locally
Update: After sending the YAML for the generated kubeconfig through a YAML formatter and base64 encoding it again I can confirm that the error 
line 6: could not find expected ':'
was indeed a YAML parsing bug. Although the unindented format that AWS returns is technically valid, the same-level indentation of the children under 
clusters
and 
contexts
was causing that error. Now that it parses correctly, however, I'm back to square one with the original error: 
Could not read cluster from kubeconfig for context arn:aws:eks:us-west-2:XXXXXXXXXX:cluster/my-cluster-name
Is there possibly an additional YAML parsing bug where it interprets the multiple 
:
instances in the context name as YAML objects?
I can observe that the Garden action is properly decoding the base64 encoded kubeconfig and outputting it to: 
KUBECONFIG: /home/runner/work/_temp/garden/kubeconfig
If I force the workflow to 
cat
the contents of that file afterwards, I can see that the cluster / context that Garden is failing to read is definitely there.
I think the only reasonable assumption left is that the context name is getting mangled in the parsing step
--- Update: I realized that there was still one unindented line in the kubeconfig and it seems to have worked after indenting and then using the base64 encoded 
kubeconfig
🎉 That confirms again that the root of the issue is in the YAML parsing.
A little problematic, considering that's the stock output of the 
aws eks update-kubeconfig
command. - Latest AWS CLI from Github's 
ubuntu-latest
image. - Garden Action is 
garden-io/garden-action@v1.2
b
@clever-policeman-58407 thank you for reporting this Which version of Garden are you using with the garden-action? The garden-action itself does not parse YAML, that's happening in Garden. I tried reproducing the problem with the YAML you provided in https://discord.com/channels/817392104711651328/1245455859694178334/1245857316658745355 but it did not result in the YAML error you observed. What I did: 1. I created a 
kubeconfig.yml
file containing the YAML text from your message linked above. 2. I created a 
test.garden.yml
file in the same directory, containing
Copy code
apiVersion: garden.io/v1
kind: Project
name: project
defaultEnvironment: local
dotIgnoreFile: .gitignore
environments:
  - name: local
providers:
 - name: kubernetes
   kubeconfig: kubeconfig.yml
   context: arn:aws:eks:us-west-2:<aws_id>:cluster/foo-bar-cloud
3. I ran 
garden validate
(
garden version: 0.13.31
), which failed to authenticate due to the redactions. Do you have additional clues for me that might help to reproduce the problem? Thank you for taking the time to report this problem, hoping we can understand the issue soon ❤️
Side note: I noticed that the action seems to default to a different path than $HOME/kube/config for some reason. I think that's a bug we accidentally introduced in https://github.com/garden-io/garden-action/commit/6e74037afcaaa43257b086eae8da0ea413aa259d – I believe the 
KUBECONFIG
variable should default to whatever the value of 
KUBECONFIG
is before the garden action has been invoked, and we should only set 
KUBECONFIG
to 
${{ runner.temp }}/garden/kubeconfig
if the user added a 
kubeconfig
argument. Or maybe we should remove the base64 functionality. What's your opinion on the action inputs @clever-policeman-58407 ?
c
On the default kubeconfig path: I would agree with the approach you mentioned. I think the natural assumption is: 1. The action will default to the default 
$HOME/.kube/config
2. 
$KUBECONFIG
can override this value 3. 
kubeconfig
overrides both, but only matters if explicitly specified On base64 input: I think in a theoretical sense it seems quite strange to specify an encoded string rather than specifying an input path, because the implication there is "this value is derived from a stored secret" and I don't think most workflows will statically store kubeconfigs. In all the scenarios we currently have for using Garden (dev, test, qa, CI, production) those are generated on the fly from k3s / k3d / EKS and there is no real value in the base64 encode / decode step.
I think a more direct way to test the YAML processing step is to work backwards from an existing configuration and unindent the area I'm talking about
And again in text. Valid YAML, does not work. By default, the EKS toolkit will generate kubeconfigs with valid but unindented blocks like this.
Copy code
yaml
contexts:
- context:
    cluster: foobar
    user: foobar
  name: foobar
Valid YAML, works. Technically unnecessary indentation but fixes the issue.
Copy code
yaml
contexts:
  - context:
      cluster: foobar
      user: foobar
    name: foobar
Garden itself has no problem with either as far as I can tell, we have been using EKS generated kubeconfigs without that indentation for months now outside of the action. The issue seems to arise once the base64 encode / decode step is introduced (or perhaps something specific to the action in particular).
Maybe just to highlight your find here: none of this would have mattered or been discovered if the action was using 
$HOME/.kube/config
so there's that 🤷‍♂️
b
Thanks a lot, I will have a closer look and if I can't reproduce the YAML issue I will at least change the default behaviour of the action to prevent this confusion in the first place.
g
b
Hi, we published Garden-Action v2.0 now. There we removed the kubeconfig option. Does that work better for you @User @User ?
g
hey @big-spring-14945 thank you 🙂 this looks much better. It's working now to start to deploy to the cluster 🙂 now i's not deploying ingress but i think this is another issue (https://github.com/demokratie-live/democracy-api/actions/runs/9494516565/job/26165017325). Currently i just set an environment variable in the github action
Copy code
- name: 🐟 set kubectl context
        run: echo "KUBE_CONTEXT=$(kubectl config current-context)" >> $GITHUB_ENV
and in the project provider use the context if available
Copy code
providers:
  - name: local-kubernetes
    environments: [local, local-prod]
    context: ${local.env.KUBE_CONTEXT || "docker-desktop"}
thank you! 🥳
b
Great, happy to hear that! The new error message looks to me like it's an issue with the node affinity/selector configuration in your Pod or an issue with the Kubernetes cluster. But great to hear that the simplified garden-action works better for you! Thank you for taking the time to report issues like that and making garden better for everyone
g
PreviousNext
Powered by