I’ve recently started leveraging spot instances to optimize workload management on my Kubernetes cluster, which has significantly improved bin packing efficiency. However, I’ve been encountering the following issue:

Failed to pull image "gardendev/k8s-util:0.6.2@sha256:f51e7ce040e2e23bc0eaa7216e4d976f13786d96773ef7b8c8f349e7a63d74e9"

The problem seems to be related to rate limits when pulling the

gardendev/k8s-util image.

This is now happening more since spot instance is trying to place the pods in appropriate nodes more often. I’m looking for a way to specify a custom image hosted in my ECR repository to avoid hitting these rate limits. I’m using Kaniko as the builder, and it already uses its own custom image, which is also hosted on ECR. Does anyone have any suggestions on how I can configure this to resolve the issue and prevent hitting these rate limits? My current provider configuration looks something like this:

providers:
  - name: kubernetes
    context: arn:aws:eks:us-east-1:123456789:cluster/garden
    buildMode: kaniko
    kaniko:
      image: 123456789.dkr.ecr.us-east-1.amazonaws.com/local-mirror:gcr.io-kaniko-project-executor-v1.11.0-debug
      extraFlags:
        - --snapshot-mode=redo
        - --use-new-run
      nodeSelector:
        node/namespace: "build"
      tolerations:
        - key: node
          value: build
      namespace: dara-${local.username}
    deploymentRegistry:
      hostname: 123456789.dkr.ecr.us-west-2.amazonaws.com
      namespace: garden

Also on a side note, I tried following the documentation here for utilImageRegistry setup https://docs.garden.io/reference/providers/kubernetes#complete-yaml-schema but it wasn't helpful as I kept getting error

Error validating provider configuration: key "utilImageRegistryDomain" is not allowed at path [utilImageRegistryDomain]. Available keys: dependencies, environments, buildMode, etc

I am using

garden version: 0.13.13

just fyi

Hello @witty-wall-71317 ,

utilImageRegistryDomain

is the correct field for that. This field was only introduced in the 0.13.43 release see https://github.com/garden-io/garden/releases/tag/0.13.43 . Please upgrade garden to use this flag.