Best way to use garden-util in dynamic CI environments

Hi,

We’re trying to use Garden to build and deploy “preview” environments. Basically we want to follow the principles of Vercel & Netlify, building every change and deploying this to a preview environment to view and test the change.
However, our application stack is not ready for fully serverless deployments and is focused on deployment on a Kubernetes cluster. This is where garden comes into play.

As we’re deploying a lot of environments, we’ve enabled autoscaling (on GKE) which automatically adds and removes nodes from our build and deploy pools. We managed to put build pods on the right nodes (using Kaniko). However, this also deploys a garden-util pod for every environment.

  • This pod is put on the same node as the Kaniko build pods, which causes our build pool (high-cpu VMs) to grow instead of scale down after a build is done.
  • This pod uses 1/4 vCPU and 512MB memory (hardcoded “resources.limit”, and thus also “resources.request” values in Garden), which seems a lot more than needed.

Having +30 environments means several “idle” build pools which make a drastic increase in cost price.

Any suggestions on how we should handle this?

  • For example, a strange but maybe useful workaround could be to just delete the garden-util pod after building. This depends if the util pod helps in caching some data. Could be nice to have some more insight in what this util pod does.

  • Another workaround could be to edit the Kubernetes manifest for these util pods and change the nodeSelector/tolerations together with the resources.

Thomas

Hey Thomas! We actually don’t set resources.request on these pods, so perhaps GKE is using the limit as a default? In that case, we could easily add explicit resource requests to these pods that can be much lower than the limit.

As for what that pod does, we run a few things through it, primarily syncing build contexts for Kaniko builds and querying remote registries. Removing the pods after builds would be a little bit tricky but possible if it matters enough resource wise. And of course, you have the option to do that explicitly yourselves, Garden will always just recreate next time they’re needed.

When you’re using Kaniko, you do also have the option to to set kaniko.tolerations on the kubernetes provider config, and this does get passed to the garden-util pods.