The kubernetes garden provider has a nice feature to copy secrets in namespaces since garden has the great feature of creating dynamic namespaces. For authentication purposes this allows you to set up service account keys (SAK) which can be loaded in your actions (I'm biased to using GCP here but I think its similar across other clouds). However, the clouds seem to be moving towards using Workload Identity Federation (WIF) over creating and dispersing SAKs. For instance in GCP you can provision an SA, give it IAM bindings, and then create a binding to a WIF role that maps it to kubernets service accounts (KSA). At this point you only need to associate the KSA to your pods for authentication to work in most cases. No copying of SA keys into secrets in CI/kubernetes/local varfiles, no refreshing them, etc. For example in GCP & GKE you can add an IAM binding with this "member":

serviceAccount:my-project.svc.id.goog[my-k8s-namespace/k8s-sa-name]

and the KSA will automatically have the same permissions as the bound SA in GCP. My question is if anyone has experience with getting this to work within garden? The immediate problem is the namespace which is fixed (AFAIK) in the role binding. There are things like Common Expression Language (CEL) and some additional matching roles that might help but I try not to lean on those too much as its pretty complex. In lieu of using some cloud feature I was wondering if there might be a way to have something similar to the

copySecret

option in the

kubernetes

provider for service accounts in Garden.

Hi, yes we have experience with this and updated our in-cluster builders to work with this auth mechanism (Google Workload Identity in GCP and IRSA in AWS) see https://docs.garden.io/kubernetes-plugins/guides/in-cluster-building#using-in-cluster-building-with-google-workload-identity. AWS's IRSA does not have the limitation with the IAM policy being limited to the namespace that you mention. The role binding does not really fix this for GCP afaik, since the issue is that the IAM policy binding in GCP's IAM system needs to be updated with each new

$namespace/$serviceAccount

. So copying the Kubernetes service account to another namespace wouldn't help. To dynamically adjust the GCP IAM policy binding, you could potentially do something like a garden

initScript

that updates it every time there is a new namespace. However that would mean that each developer needs the permissions in GCP to update this policy binding.

I see. The WIF for building sounds nice, but I would need it for other things as well. In my example I'm trying to pull test data from a bucket. It looks like I at one time also experimented with using Pulumi to create the new SA IAM Binding, but that was super complicated and has the same drawbacks you mentioned.

Yeah, it is a weird limitation in GCP and they are apparently working on an improved workload identity for GKE that does not have this limitation see

Its interesting though because its definitely possible to do. I went through setting up WIF OIDC pools and its a feature that you can give dynamic roles from the federated "members". For instance there were patterns you could set up to allow certain users launching CI pipelines, different branches, tag names, etc. So the problem really just lies in the way they've configured this "default" WIF Pool for GKE. Makes me wonder whether you could actually just do it from scratch and pretend GKE is an on-prem cluster.

Interesting. If you try it i'd be super curious 😄

its really painful to set that up though. Unless its truly blocking I'd rather just workaround and let Google add it as a feature.