Failed to resolve project configuration.
────────────────────────────────────────────────────────────────────────────────────────────────────
Unable to load Garden project found at /Users/jackhall/code/nucleus: Error: Invalid template string (${command.params.env}.scalecomputi…): Could not find key env under command.params. Available keys: (none).

Several commands including

deploy --sync

are failing due to this usage of template syntax wherein the --env option is not recognized.

Hi all, not sure if this is a bug or just a bad config. I have the following Deploy action that sets up some kubernetes secrets

yaml
kind: Deploy
name: secrets
type: kubernetes
spec:
  manifests:
    $forEach: ${var.secrets}
    $return:
      apiVersion: v1
      kind: Secret
      metadata:
        name: ${item.key}
      type: Opaque
      stringData:
        $merge: ${item.value}
varfiles: 
  - ./secrets.${environment.name}.yaml

I get the following error when I run garden deploy

Error processing config for Deploy action secrets:
Could not find varfile at path './secrets.${environment.name}.yaml'

I have just recently updated to the latest release of garden and this previously worked with Modules. My file structure looks like the following:

secrets/
  garden.yaml <-- file where module / action was defined
  secrets.local.yaml

I notice when i run garden deploy it says the following :

ℹ garden               → Running in Garden environment local.aashelt

which is because I think i have this setup in my environment configuration in my project.yaml

yaml
  - name: local
    defaultNamespace: ${local.username}
    variables:
      defaultHostname: ${project.name}.${local.username}.local.app.garden
      replicas: 1
      cpu: 100m
      mem: 1Gi
      volumeSize: 5Gi
      storageClass: ""
      cluster: ""
      timestamp: ""

so should the environment name resolve to "local"? or "local.aashelt" ? I ended up changing my file to match that and i received the same error, its like its not interpolating the template. Heres the old Module that worked before:

yaml
kind: Module
name: secrets
type: kubernetes
manifests:
  $forEach: ${var.secrets}
  $return:
    apiVersion: v1
    kind: Secret
    metadata:
      name: ${item.key}
    type: Opaque
    stringData:
      $merge: ${item.value}
varfile: secrets.${environment.name}.yaml

Not sure what other information to provide but if theres anything else i need to provide let me know.

Our garden environment is huge, > 80 modules. Recently I started getting these errors (please check the screenshot). Is there an easy way to increase the memory that the CLI uses? Thank you.

Hey all, Trying to understand what is happening under the hood / if we can do something to improve the situation. We have about 240~ garden modules - the actual number of modules is more around a 100, but due to different templates we end up with that number of modules needing to be resolved by Garden on startup. We noticed the graph is taking very long, around 30+ seconds

[2023-06-26T17:24:42.519Z] Resolving 240 modules...

[2023-06-26T17:25:14.026Z] Resolving 240 modules... → Done

This is using the Kubernetes provider. I've look at the logs with debug logs on, and noticed it's doing some FS scans which takes about a second, I assume building the DAG is also very quick.. I wonder if most of this time is spent making Kubernetes API calls? Are those necessary or is there an option here to introduce a flag to skip all of these? Is there anything else I can do to improve this situation? It would be fantastic if this was under 5 seconds. Thanks!

Let's say I have a project level variable structure:

component:
  variable: X

How can I override these variables from the cli flag (

--var

) or from .env varfiles? I tried

--var=component.variable=Z

but that didn't work.

command executed: garden deploy --log-level=silly --logger-type=basic --var targetImage=production --env feature

────────────────────────────────────────────────────────────────────────────────────────────────────
Unable to run docker command: An error occurred while trying to run '/root/.garden/tools/docker/e9744f1ea7f84408/docker/docker' (spawn /root/.garden/tools/docker/e9744f1ea7f84408/docker/docker ENOENT). Please make sure '/root/.garden/tools/docker/e9744f1ea7f84408/docker/docker' is installed and in the $PATH.
────────────────────────────────────────────────────────────────────────────────────────────────────
ℹ build.root-config [debug] → Spawning '/root/.garden/tools/docker/e9744f1ea7f84408/docker/docker manifest inspect xxx.azurecr.io/xxxx/xxx:v-224566a45f' in /opt/atlassian/pipelines/agent/build/.garden/build/root-config
✖ build.root-config    → Failed resolving status for Build type=container name=root-config (took 4.61 sec). Here is the output:

Hi again! I'm fairly confident that this is a bug, or at very least should be documented when moving from 0.12.x to 0.13.x Essentially the problem is this: upgrading from 0.12.x to 0.13.x of garden results in different permissions to be applied to the files synced with garden-util. Example tldr; on version 0.12 we got permissions that mirrored the permissions on the local file system where garden was ran on version 0.13 the permissions were overwritten with 600/700 it seems as though the user and group are being masked to 0 after looking over the code it seems like rsync is nolonger the default? and now garden fully relies on mutagen? We have a frontend image that compiles your typical SPA and loads the content into an nginx:stable-latest docker image, prior to upgrading to version 0.13 everything worked fine. The resulting pod thats created inherits the built frontend with permissions = 600, resulting in NGINX to throw 403's and none of the styles or assets can be loaded. A Simple fix that i did was shell into the pod and chmod the nginx/html directory with the appropriate permissions and that fixed my issue temporarily, however every subsequent build / deploy would cause the issue to reappear I'm open to discussion i've tried various numbers of things, but all of them feel quite hacky, i'm happy to provide some more information or here what other people have done. EDIT: Below is a config i've added to my provider to try and overwrite the default behavior of mutagen.

yaml
    sync:
      defaults:
        exclude:
          - '**/node_modules'
        # owner: 1000  # <- set an integer user ID or a string name
        # group: 1000  # <- set an integer group ID or a string name
        fileMode: 644 # <- set the permission bits (as octals) for synced files
        directoryMode: 755 # <- set the permission bits (as octals) for synced directories

I've still not beable to get this to work with this config

garden version: 0.13.8 rust: 1.70 node:18.16.0-alpine running Linux dev 5.10.102.1-microsoft-standard-WSL2 --- Sorry for the n00b question but I'm at my wit's end. I have a rust backend with a node frontend, both copies of examples from the examples directory from Garden's github. The only thing different is that I've changed the routes so that they should correspond. Here is the repo: https://github.com/matthewaerose/test-of-rust-with-frontend I can reach the backend through its ingress I can reach the frontend through its ingress But when it comes time for the frontend to contact the backend, I get

{
  "error": {
    "cause": {
      "errno": -111,
      "code": "ECONNREFUSED",
      "syscall": "connect",
      "address": "127.0.0.1",
      "port": 80
    }
  },
  "message": "Unable to reach service at http://backend.rust.local.demo.garden"
}

--- I check the svc with

kubectl get svc

and see it doesn't have an ExternalIP.

NAME       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
backend    ClusterIP   10.105.208.217   <none>        80/TCP     21m
frontend   ClusterIP   10.97.105.9      <none>        8080/TCP   16m

--- I execute a curl from ingress pod to backend and it returns

kephalos@dev:~/test-of-rust-with-frontend$ k exec -n garden-system garden-nginx-ingress-nginx-controller-j9gff -- curl -X GET backend.rust.local.demo.garden
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    17  100    17    0     0     55      0 --:--:-- --:--:-- --:--:--    55Hello, Rust 🦀!

Please let me know if there is any additional info I could provide that could be helpful. Thank you!

Hi, For a helm sync operation, I see garden trying to run a kubectl command like below

/Users/ankitmishra/.garden/tools/kubectl/f8a39834529cd1ed/kubectl --context=<context> --namespace=garden-test apply --output=json -f -

I checked the version of kubectl and it is a couple of versions behind the version we run in our cluster

/Users/ankitmishra/.garden/tools/kubectl/f8a39834529cd1ed/kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"25+", GitVersion:"v1.25.11-eks-a5565ad", GitCommit:"df109513ba589adaee235b4e658476275d1fde31", GitTreeState:"clean", BuildDate:"2023-06-16T17:35:00Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.25) exceeds the supported minor version skew of +/-1

This is causing issues with the sync as shown below.

error: error validating "STDIN": error validating data: ValidationError(Deployment.spec): unknown field "topologySpreadConstraints" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false

My question is, can I specify the kubectl version or the kubectl path so that I can work around the issue? An easy fix would be to copy the binary in the garden kubectl destination, but is that the recommended way to proceed?

Has anyone used garden with cloud native buildpacks? Tried running the pack cli directly but getting gradle errors.

Hi all! I'm trying to deploy a helm chart with garden 0.13.7 and it's failing with the following error:

Error: INSTALLATION FAILED: values don't meet the specifications of the schema(s) in the following chart(s):
jupyterhub:
- (root): Additional property .garden is not allowed

So, the error itself is pretty obvious. The jupyterhub helm chart is invalidating any additional properties not included in the chart's schema. The only solution to this that I'm currently coming up with is to fork their helm-chart and edit the schema. That is certainly not an ideal solution and would require a lot of extra work on my end, and I don't have any need to make further edits to that helm chart schema other than to allow additional properties at the root. I've installed 7-8 other charts with Garden and have had no issues. Absolutely in love with it. My question: Is there a way to disable the

--set \.garden.mode=default

argument that is added to the command? The command and output that's being run by garden is:

Command "/home/jessica/.garden/tools/helm/b736014484fd8f0f/linux-amd64/helm --kube-context jess-epx-platform --kubeconfig /home/jessica/GolandProjects/EpxPlatform-BaseDockerfile/garden/terraform/kubeconfig.yaml --namespace jupyterhub install jupyterhub jupyterhub --version 1.2.0 --repo https://jupyterhub.github.io/helm-chart/ --namespace jupyterhub --timeout 300s --values /home/jessica/GolandProjects/EpxPlatform-BaseDockerfile/garden/actions/deploy/platform-hub/helm/base.yaml --values /tmp/8336045834d1f61022757a21faa86dee --set \.garden.mode=default" failed with code 1:

This can be reproduced with the following deploy action:

kind: Deploy
type: helm
name: platformhub-helm
spec:
  releaseName: jupyterhub
  namespace: jupyterhub
  chart:
    repo: https://jupyterhub.github.io/helm-chart/
    name: jupyterhub
    version: "1.2.0"

If anyone has any insight to this, please help! I really really don't want to start maintaining another helm chart.

Hi! I am using docker image build action and it depends on exec deploy action that deploys registry credentials. When I run

garden deploy <my_docker_build_action>

it goes right away to building an image without even checking the status of exec deploy action. As a result image build fails as there are no creds deployed. In 0.12 it worked, but after upgrade to 0.13 dependency resolution seems broken. I've checked everything, but couldn't find anything to solve this issue. My actions are defined as in the following example:

yaml
kind: Deploy
type: exec
name: example-registry-creds
spec:
  deployCommand: [example, deploy, command]
  statusCommand: [example, status, command]
  cleanupCommand: [example, cleanup, command]

yaml
kind: Build
type: container
name: my-docker-image
dependencies:
  - deploy.example-registry-creds
include:
  - Dockerfile

Maybe these logs can be related

Found 0 files in Deploy action example-registry-creds root /path/to/example-registry-creds

Is this a bug? Please ping me if you need additional info. I am using garden

v0.13.9

on Ubuntu 20.04

Hey! I'm trying to use Garden to deploy some images that are built via a separate build process. We use a static tag that is not

latest

, and so I need to be able to configure the pod/deployment imagePullPolicy to be Always. It seems like this is not covered in the docs, and attempting it the key seems to be ignored. Not sure if this is intentional/bug or just a misconfiguration on my side. Thanks for any advice!

👋 I am cruising around Europe in a campervan 🚌. Garden's developers are here to support you while I'm away but please note they are also busy improving your favorite developer tool so response times may be slower than usual!

When running a integration tests (or workflows running them) I can't find a way to get the logs for the services that the tests are hitting. For instance if I just run an HTTP request in my test all I can see is the error status code, and I need to set up the

garden dev

environment and stream the logs to actually see the stack trace on the server side. It would be really awesome if there was a way to specify which services to capture logs for during a test run. I tried doing something like this:

kind: Workflow
name: integration-tests
steps:
  - command: [deploy, --logs, api]
  - command: [test, integration-tests]

But it gives me an error that specifically it can't stream logs using workflows. Is there a solution I'm missing, or is this something that would be a feature request?

The docs on the website list a config which isn't actually compatible (using hub.docker.com results in 404s) and I'm unable to actually get auth to work with standard dockerhub secrets (index.docker.io)

Hi folks! Since recently some of our developers sporadically hit a generic error

Invalid count value

when deploying with local mode.

$ garden deploy --local bex-os

...
Scanning repository at /Users/xxxx/IdeaProjects/bex-os


Invalid count value
Scanning repository at /Users/xxxx/IdeaProjects/bex-umbrella
Scanning repository at /Users/xxxx/IdeaProjects/bex-umbrella/.garden/sources/project/bexapp--accbe12695
Scanning repository at /Users/xxxx/IdeaProjects/bex-umbrella/.garden/sources/project/bom-app--fc03d112c9
Scanning repository at /Users/xxxx/IdeaProjects/bex-umbrella/.garden/sources/project/bom--493b1ac2ed
Scanning repository at /Users/xxxx/IdeaProjects/bex-os


$ echo $?
1

When deploying without the

--local

flag, the problem does not occur. Most of the time, the problem does not occur. However, when it occurs, it keeps occurring until the next day. I'm still trying to reproduce this issue but so far had no luck, since it never occurs on my machine. All our devs are using OSX - I'm on linux. I grepped the garden codebase for the error string and as well only parts of it but couldn't find any sensible matching occurrence. It might come from a `RangeError`: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Errors/Resulting_string_too_large Has anyone seen this behavior before? Any ideas what the problem could be?

Is

garden-system

namespace deprecated and planned to be removed? As I see garden can work without this ns. I want to deploy some custom resources to this namespace and want to know beforehand what will Garden do with it in future.

Hi I am trying to deploy a helm deployment on our Kubernetes cluster from a Jenkins pipeline to run some integration tests. For this, I use an existing chart which I have tested well while running garden from my machine. But when running in Jenkins, I had to create a new cluster admin role with specific permissions (see snippet attached) that I don't have to use while running locally due to my user having complete cluster access. I created this to provide kubectl access to Jenkins. I am getting an error that has my head scratching

ℹ deploy.gdp-garden [verbose] → [kubernetes-plugin] Status of Deployment dp-ci-1691586975313-gdp is "deploying"

⚠ deploy.gdp-garden → Deployment/dp-ci-1691586975313-gdp: Error: failed to generate container "5619a71419b472824c14778e2632e2ed39f0571748f98e54023cf05c87e231cf" spec: failed to generate spec: no command specified

ℹ deploy.gdp-garden [verbose] → [kubernetes-plugin] Deployment/dp-ci-1691586975313-gdp: Error: failed to generate container "5619a71419b472824c14778e2632e2ed39f0571748f98e54023cf05c87e231cf" spec: failed to generate spec: no command specified

A CMD is present inside the Dockerfile to build the container used in this deployment, so our helm chart does not have any command specified for the deployment, and it has always worked in the past (and when I deployed the chart locally using garden). I could add a command to the manifest or helm values to bypass this error. But I am wondering why there is a difference in behaviors. Am I missing some permission in the ClusterRole?

Hi, our pipeline with garden sometimes take over an hour, is there a way to speed things up for the production pipeline?

We have garden setup for local development purposes and we use cluster-buildkit mode for building docker images. But we want to configure kaniko build mode with the same Garden project for running Garden in CI/CD. Currently, I do not see any possibility to define both build modes in one project and choose one for either of cases. Is it even possible? If yes, how can I do it?

No idea if this is a known issue. I can create a bug ticket if it is not. If you have a Run action (a) that depends on a Run action (b) that depends on a Deploy action, and you’ve run all of these actions before, their statuses get cached so Garden doesn’t need to re-run them — all good so far. Then, if you delete and redeploy the Deploy action, I would want that to invalidate the previous run of both actions (b) and (a) but it does not. You can even force Run action (a) to run and it will error out (internally to the contents of the action, not a Garden error) because Garden will skip over (b) as having been run when in reality it has not been run against the new Deploy.

Hello! We're a small team using Garden to manage and deploy a set of AI workloads and services on clusters in different environments, and it's working really well for us. We have until now used a garden build+test+deploy workflow for all our environments (dev, test and prod) using container Build, container Test and kubernetes Deploy actions. However, we want to implement a GitOps approach for our prod environment, in which Garden is only responsible for building images and rendering k8s manifests, after which the manifests should be handed over to a GitOps tool (we're trying out the Pulumi Kubernetes Operator at the moment). But we're not quite sure what is the most natural way to use Garden to just render and output the k8s manifests without deploying them. I can think of a few hacky ways, but since I imagine this might be a reasonably common use case, I'm wondering if I'm missing something and if there is one or more "natural" patterns for such a workflow using Garden.

I'm kind of using Garden as a replacement for manually scripting builds in either CI/CD scripts (i.e. Gitlab yaml) or in my Dockerfiles. I like being able to run the same thing locally and in CI as it makes the whole cycle much faster; and I like writing the script outside of the Dockerfile since its much easier to debug failing builds when its outside of the Dockerfile and I have to deal less with injecting tokens/keys etc. which is a PITA with docker (and different if you use

docker build

locally, and kaniko remotely). My goal is that I want an action to download/clone another repo, run (in effect)

make build

, and then have the outputs be available to other build tasks in my garden graph. I've run into a couple dead-ends so far though: 1. I tried just using "Build exec", which works if you don't sandbox anything which would be the more desirable way of working. Furthermore, this is not not ideal for running in CI and doesn't work in most cases. First, my CI runner is using the

gardendev/garden-gcloud

image which is good for running garden, but not so much for my own toolchain and may actually be incompatible (for instance this image is on alpine). I could make my own image but then it would be a weird hybrid and hard to maintain. Without that concern it would be better just to run exactly the same build container locally and in CI. 2. So I tried running the builds inside of a Pod in the cluster (whether that is local minikube or remote GKE). The problem here is that there is no Build type for running in an exec-like command in a pod (like

kubernertes-pod

for the Run action type). Using Run kubernetes-pod works, but the semantics around passing data between actions is only really set up for Build action types. I can output an artifact but AFAIK there is no good way to get this into the Build container workspace. So I think my feature request would be Build types for kubernetes-pod and kubernetes-exec. Would this be a difficult thing to add?

Trying to run‘garden .13 on a Mac M1 with minikube - I’m getting ‘exec /rsync-server.sh: exec format error’. Any way to get —sync mode to work on Mac silicon? Tia

Hey guys. Is there an equivalent to generateFiles in the Terraform Deploy schema? My use-case is the following: I have Terraform Deploy actions which pull their source files from another repository (specified in source.repository.url). I would like to use a remote Terraform backend for these actions. Garden sadly does not expose a way to configure Terraform state backends. My workaround with Garden modules was using generateFiles to copy a backend.tf file with some templating, into the computed module sources as follows:

generateFiles:
  - sourcePath: ./backend-file-name
    targetPath: ./path/to/module/root
    resolveTemplates: True

Is there a way to achieve the same outcome with Garden Actions? Thanks in advance!

I was trying the quickstart(https://docs.garden.io/basics/quickstart) on bonsai/windows with docker-desktop and running into this issue running deploy. Any ideas here?

Hi All, I'm looking into introducing garden (I really love the idea) and I'm trying to clear in my head how it fits with the way our repos are structured and our code is deployed so I can try get a POC running. We have multiple backend services spread across a few mono-repos (each repo represents a feature of the product, and has a directory per service that makes up that feature), another repo represents our frontend/UI, and a final repo contains the Jenkins jobs for testing post-merge. Each service has a Dockerfile and builds and pushes a helm chart. Our CD repository contains environment configurations. At a surface level this repo is made up of a directory per env. This env directory contains one large values.yaml, and subdirectories per service to be deployed with a yaml file describing the helm chart to deploy. Questions 1. Given the multi-repo nature of the code, are remote sources the right approach or is there something different I have missed? 2. Given all the code repos are of equal importance, where should the root garden project file have its own repo? It seems either: the CD repo, or a similar decoupled repo should store it, OR each dev team has their own garden project in their repo that lists their dependencies as the other repos?

Hello, I'm trying to get garden working using the

kubernetes

provider. Garden version 0.13.12. I have the

local-kubernetes

working just fine but am having trouble getting it set up for remote. I am using this document as my general guide https://docs.garden.io/kubernetes-plugins/remote-k8s/configure-provider. I setup a cluster in Azure AKS and can connect to it using

kubectl

(e.g.

kubectl get nodes

). I deployed contour, cert-manager, letsencrypt and certificate using helm/kubectl such that I have a valid certificate. My DNS points to the cluster (if I install a service using helm I can connect to it at my URL). I have a container registry created in azure and created a imagePullSecret and put it in the cluster using

kubectl create secret docker-registry

My

project.garden.yaml

looks like: Edit - config removed for length. The issue is when I run

garden deploy --env remote

I get the following message:

ℹ garden               → Running in Garden environment remote.default
ℹ providers            → Getting status...
✖ providers            → Failed processing resolve provider kubernetes (took 1.4 sec). Here is the output:

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Error: Client network socket disconnected before secure TLS connection was established
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────


Failed resolving one or more providers:
- kubernetes

See .garden/error.log for detailed error message

There does not appear to be any additional information in the logs. I've tried running with the

--logs

flag which appears to run it in an interactive mode, but I am not sure what to do from there. I am not sure where to go from here. Any help would be appreciated. Thanks, Matt

I'm getting an error when trying to pass some local environment variables to buildargs This works, passing in the actual username and key:

buildArgs: {
    GITHUB_API_USER: username,
    GITHUB_API_KEY: api_key
  }

None of the combinations I'm trying work, eg:

buildArgs:
    - GITHUB_API_USER: ${local.env.GITHUB_API_USER}
    - GITHUB_API_KEY: ${local.env.GITHUB_API_KEY}

this results in the error:

Error validating spec for Build type=container name=policies: key .buildArgs must be of type object. Available keys: localId, publishId, targetStage, buildArgs, extraFlags, dockerfile, hotReload)

I'm not sure how I create an object that can be passed to buildargs using local env vars?