garden-io #🌱｜help-and-getting-started

Channels

📢announcements

optimus-admin

getting-started

👋🏻｜introduce-yourself

💻｜dev

🤖｜ask-kapa

🌱｜help-and-getting-started

Build Failure

blue-lunch-98125

01/12/2023, 3:09 PM

I am attempting to bring my company over from a docker-compose and Jenkins setup to Garden. I am running into a build failure, and can't make sense of what is going on. I get the following output:

Copy code

Command "docker exec -i  crictl images --output=json docker.io/library/pgbouncer:v-67ed3bcc63" failed with code 1:

Seems like maybe it is missing the correct image name? This is running under debian. There's nothing interesting about the module or dockerfile afaict. Any ideas what might be going on here?

Gitlab Runner not finding Garden Artifacts

curved-farmer-35040

01/12/2023, 9:19 PM

Hey Team! Odd question, has anyone had any issues with a GitLab runner not finding artifacts in the .garden folder? Running some unit tests in a container, and I am ssh'ing into my runner and can see the artifacts being copied into that folder as the test is running. Yet, GitLab is complaining it can't find them? Looked around at the GitLab docs, not sure if GitLab is ignoring the .garden folder because it is in .gitignore. I am running parent-child pipelines, and I can see that each garden command is also deleting the .garden folder which could be causing the timing issues. Very vague but I don't have much else to go on! Can see the artifacts myself, but for some reason, my GitLab runner cannot.

cert-manager gives error in Docker-desktop

bright-policeman-43626

01/13/2023, 5:31 PM

Hey there! tried to install cert-manager using docker-desktop (local-kubernetes) but encountered the following issues; apparently this installation of kubernetes don't have the required CRD's?

Copy code

unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "APIService" in version "apiregistration.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ValidatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"

Docker version:

Copy code

docker --version
Docker version 20.10.20, build 9fdeb9c

Also another question, do you guys have experiences by using Garden to generate Custom certificates self-signed? We need TLS but our installations are ephemeral (shared between multiple pc's as everything works locally).

Terraform module displays sensitive outputs in plain text

mammoth-flag-56137

01/16/2023, 3:38 AM

Copy code

ℹ Remaining tasks 0
ℹ Remaining tasks 0

Done! 
Done flushing all events and log entries.
success: true
result:
*snip*
          dependencyResults:
            get-service-status.infrastructure-aws:
              type: get-service-status
              key: get-service-status.infrastructure-aws
              name: infrastructure-aws
              description: >-
                getting status for service 'infrastructure-aws' (from module
                'infrastructure-aws')
              completedAt: '2023-01-16T03:10:10.798Z'
              batchId: 2b6e44f4-32f1-42a1-b55a-98ffeed8d8b0
              output:
                state: ready
                version: v-2570f839d4
                outputs:
                  access_key: x <<<<<< sensitive
                  connect_client_registry: >-
                    x
                  connect_db_password: x <<<<<< sensitive
                  connect_s3_document: >-
                    x
                  connect_s3_export: >-
                    x
                  connect_server_registry: >-
                    x
                  keycloak_admin_password: x <<<<<< sensitive
                  keycloak_db_password: x <<<<<< sensitive
                  keycloak_registry: >-
                    x
                  kms_key_id: >-
                    x
                  one_client_registry: >-
                    x
                  one_db_password: x <<<<<< sensitive
                  one_pricing_registry: >-
                    x
                  one_s3_document: >-
                    x
                  one_s3_export: >-
                    x
                  one_s3_pricing_backup: >-
                    x
                  one_server_registry: >-
                    x
                  pricing_db_password: x <<<<<< sensitive
                  secret_key: x <<<<<< sensitive
                  user_arn: >-
                    x

Sharing images between modules

blue-lunch-98125

01/17/2023, 6:03 PM

Is there a preferred way to share a base image between modules? Does the image have to be pushed to a remote registry as a middle step? What I want to do is something like the following:

Copy code

kind: Module
description: ebtflask
type: container
name: ebtflask
image: efs-server
build:
  timeout: 3600
services:
  - name: ebtflask
    ports:
      - name: http
        containerPort: 5000
    dependencies:
      - kafka-broker
      - pgbouncer
      - redis
    command:
      - nodemon
      - -e
      - py
      - --exec
      - './manage.py runserver --disable_reloader'
      - --config
      - freshebt/nodemon.json
    env:
      EFS_DB_HOST: pgbouncer
      EFS_REDIS_HOST: redis
      OPENSEARCH_HOST: opensearch
exclude:
  - devops/images/**/*
---
kind: Module
description: celery-beat
type: container
name: celery-beat
image: efs-server
build:
  dependencies:
    - name: ebtflask
services:
  - name: celery-beat
    env:
      EFS_DB_HOST: pgbouncer
      EFS_REDIS_HOST: redis
    dependencies:
      - pgbouncer
      - redis
    command:
      - nodemon 
      - -e 
      - py 
      - --exec 
      - 'celery beat -A freshebt.app:celery --pidfile= --schedule /var/lib/celery-beat.db --loglevel=INFO'
      - --watch 
      - freshebt
include: []

Where the second module simply reuses the container image built in the first, but with a different command.

What is Garden using so much upload bandwidth for when running a task?

mammoth-kilobyte-41764

01/17/2023, 11:30 PM

I’ve been working on an internet connection with very little upload bandwidth so I noticed when running a remote Garden task (with the Garden CLI) resulted in maxing out my 1Mbps upload from my home internet connection. What data is garden sending in such high quantities during the execution of the task? Bandwidth stayed pegged the whole time the task was running and the task itself doesn’t send any data so I figured this data upload must all be about communicating with the cluster garden util and checking job status. Is there more to it? It’s not that 1Mbps is all that much data, just curious what it’s being used for.

Using `garden test` with a helm configmap

famous-afternoon-28388

01/18/2023, 9:55 AM

I want to run simple unittests that use a configmap that's defined as a helm template I added this configuration to my garden helm module

Copy code

tests:
  - name: unit
    command: ["pytest", "tests"]

but the relevant file isn't on the pod. Is what I'm looking for possible, or do I need to find some workaround?

JVM remote debugger connection flakey with garden port forwarding

quaint-librarian-55734

01/19/2023, 11:24 PM

We've noticed that, for some reason, JVM remote debugger connections tend to be flakey using the built in garden port-forwarding when connecting a remote debugger to a service in Kubernetes Let's say I have service-x that has a devMode startup command like:

java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -cp /app/app.jar

This starts up the Java server and also listens for incoming remote debugger client connections on port 5005. Then, in our garden.yaml for the service we have configuration like so:

Copy code

services:
  - name: "service-x"
    command:
      - "java"
      - "-cp"
      - "/app/app.jar"
    devMode:
      command:
        - "java"
        - "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"
        - "-cp"
        - "/app/app.jar"
      sync: []
    ports:
      - name: http
        containerPort: 80
        servicePort: 80
        localPort: 8080
      - name: debug-port
        containerPort: 5005
        servicePort: 5005
        localPort: 10000

When launched in devMode, the port forwarding for port 10000 is in place, however multiple attempts to connect the remote debugger result in various forms of error "Unable to open debugger port (localhost:100000): java.io.IOException" and sometimes this includes "handshake failed - connection prematurely closed". If we keep retrying this, eventually it will successfully connect. Once connected, it will usually be able to re-connect again reliably after a disconnect for the duration of the pod's life. Initially - my thought was that there was some sort of lag between when the service was responding on port 80 (meaning the garden health check saw the service as up) and when the remote debugger was able to accept connections. HOWEVER - when I tried directly creating a kubectl port forward using a command like

kubectl port-forward service/service-x 50005:10001

the debugger is immediately and reliably able to connect as soon as the service is deployed.

kubernetes Job dependency

kind-france-89777

01/20/2023, 1:26 AM

anyone doing anything with k8s Jobs in garden? specifically I'm trying to deploy a Job, and then have further services depend on that Job and not deploy until it's completed. unfortunately it looks like garden just waits for the Job to be deployed successfully and then immediately moves on to deploying the services that depend on it.

Stuck after changing terraform backend

rough-kilobyte-44808

01/23/2023, 5:37 PM

I am somehow completely stuck and cannot run any garden commands after switching to an azure backend for my terraform config. I got the message about "Backend configuration block has changed" which said to migrate state, so I did, and state was migrated. But now I get that message for every garden command I run, period. I have blown away the .garden dir, the .terraform dir, everything, and no matter what I do I get that message. It's also worth mentioning that even before this happened, I got the message about terraform state being out of sync on every command, every single time, period, even when the command just previous was a terraform apply... I need help bad, as I am completely shut down right now!

YAML config causes null objects

mammoth-flag-56137

01/25/2023, 3:07 AM

project.garden.yaml

Copy code

one:
  config:
    name: "defaultvalue"
    foo: "bar"

if I have a config for an environment like this clientenv.yaml

Copy code

one:
  config:
    # foo: "bar2"

then when referencing

{$var.one.config.name}

in a kubernetes deployment manifest I get an error

Copy code

errors:
  - detail:
      results:
        task.one-migrate:
          type: task
          description: running task one-migrate in module one-server
          key: task.one-migrate
          name: one-migrate
          error:
            detail:
              err:
                detail:
                  value: null
                  nodePath: []
                  fullPath: var.one.config.name
                  opts:
                    allowPartial: true
                    unescape: true
                    stack: []
                type: configuration
            type: template-string

Is this the intended behaviour that having the

config:

section without anything in it is effectively doing

config: {}

and blanking/nulling any default values?

failed to get relative files

nutritious-kitchen-29367

01/31/2023, 9:01 AM

Hello. I try to build simple container module using Kaniko and get this error: error building image: error building stage: failed to optimize instructions: failed to get files used from context: failed to get relative files: lstat /.garden/context/.npmrc: no such file or directory How can I fix it?

docker-compose profiles

big-zoo-27589

01/31/2023, 7:07 PM

what is the equivalent of docker-compose profiles in garden ?

Run tests on an existing Pod

bored-grass-81679

02/03/2023, 9:11 AM

Hey! I might be missing something, but Garden is currently always creating a new Pod and running the tests on it. I would like to run integration tests on an existing Pod.Is there a way to achieve this? Here is a snippet from my helm module:

Copy code

tests:
  - name: integration
    dependencies: [secrets, ic-be-helm-ci-integration]
    disabled: ${ !var.integration }
    timeout: 600
    command:
    - /bin/sh
    - '-c'
    - 'python manage.py test instance_api/tests/integration --keepdb'
    resource:
      podSelector:
        layer: backend
        type: controller
        app: lex

Running service interactively no longer works as specified in docs

kind-carpet-47197

02/09/2023, 8:57 AM

Hi! I'm new to garden and just set up my first project to try it out. Read the following line in the docs (https://docs.garden.io/using-garden/using-the-cli#running-a-single-ad-hoc-service-and-attaching) . *Running a single ad-hoc service and attaching: `garden run service my-service --interactive # or -i for short`*. However this command fails and the CLI no longer lists

--interactive

as an option for the command

garden run service

. Is there a new way of running a service interactively that hasn't been documented yet?

Invalid template string

chilly-waitress-62592

02/09/2023, 3:37 PM

Somewhat confused as to why I'm receiving an error on this really simple template string. This is being used in the

values

section of a helm module.

"${ environment.name != 'production' ? 20 : -1 }"

I'm assuming it's something silly I'm doing.

service account image pull secret

quick-answer-45507

02/13/2023, 1:49 PM

Am i missing something dumb? i cannot seem to find a way using local-kube to configure this for image pulling...

Copy code

pullSecrets:
  nexus:
    registry: nexus-docker-out.build-tools.domain-production.com
    serviceAccounts:
      - default

Secret management without garden cloud?

careful-evening-79580

02/15/2023, 2:12 PM

Hæhæ! Currently assessing an adoption of garden to streamline a thing or two in my cluster. I've been unable grasp how to pass secrets to container modules after surfing the documentation and hitting a "page not found" (https://docs.garden.io/misc/faq#how-do-i-pass-secrets-to-container-modules) in the FAQ. Has it become a 'cloud only' feature or is it still something that's possible with garden core? I don't have a team of developers to manage(yet🤞 ) so secret management on the cloud level isn't really needed in my use case.

Database migration tasks/container

aloof-lamp-69262

02/15/2023, 4:14 PM

Hello, TLDR: Using a "task" for a database migration container is not a good option as there is no automated way to run the task without having something depend on it explicitly. Using a container dies shortly after the command runs correctly, but garden complains because it dies quickly. I have a service (container) that deploys database X, let's call it "database" I have another service (container) that does the database migration, let's call it "migration" "migration" depends on "database" naturally. If I use task for the migration, I have to run it manually which is not desired, I need this to run always when deployed. If I use it as a container, it works well, but the container dies too fast and garden complains about it, with the "BackOff - Back-off restarting failed container" error. But the container did what it is supposed to do. Is there an option to tell garden not to complain if the container stops too early? Or is there an option to get the task to run automatically? We also had many other use cases where we wanted the task to run automatically, I think this would be a desired feature. Thank you.

Using Garden Workflows

wide-nest-40830

02/20/2023, 7:29 PM

Hi, i have two questions about Garden Workflows: 1. Are Garden workflows idempotent? 2. Can 1 Garden workflow depend on another workflow (so I can sequence the workflows one after another, if I were running certain tests) 3. If my development or testing workflow required me to manipulate my environment up to a certain point, is there a way for me to pin the environment in that exact moment in time and export that state either in a updated stack graph or workflow for local reproduction?

Kubernetes Server-Side Apply

mammoth-flag-56137

02/21/2023, 3:26 AM

Is there a way to get garden to provide

--server-side

kubectl apply

when applying kubenetes manifests? https://kubernetes.io/docs/reference/using-api/server-side-apply/ I'm trying to create a ~1mb configmap with garden and because of limitations with kubectl apply it fails due to being bigger than ~256kb

imagepullSecrets not working?

quick-answer-45507

02/21/2023, 4:37 PM

So i have this current config....

Copy code

providers:
  - name: local-kubernetes
    context: minikube
    environments: [dev]
    defaultHostname: "whenidev.net"
    setupIngressController: false
    deploymentRegistry:
      hostname: nexus-docker-out.build-tools.DOMAIN-production.com
      namespace: default
    imagePullSecrets:
        - name: gitlab
    copySecrets:
        - name: gitlab

but when i try to apply a kube manifest that refrences an image from that repo with

Copy code

imagePullSecrets:
        - name: gitlab

defined i still get failed to pull image no basic auth credentials... any ideas?

split helper function

wonderful-table-85939

02/22/2023, 7:54 PM

should you be able to use array indexes on results of split? ${split("a,c",",")[0]}

Internal Registry

agreeable-grass-30619

02/23/2023, 5:10 PM

Trying to run your quickstart example against an EKS cluster for evaluation. I have an error pulling images for built containers. This is with the default setup. Just targetted at AWS rather than locally.

Copy code

Pod worker-7866cf9ff5-7skfq: Failed - Failed to pull image "127.0.0.1:5000/vote-demo-quickstart/worker:v-e2e51487b4": rpc error: code = 
Unknown desc = failed to pull and unpack image "127.0.0.1:5000/vote-demo-quickstart/worker:v-e2e51487b4": failed to resolve reference 
"127.0.0.1:5000/vote-demo-quickstart/worker:v-e2e51487b4": failed to do request: Head 
"https://127.0.0.1:5000/v2/vote-demo-quickstart/worker/manifests/v-e2e51487b4": http: server gave HTTP response to HTTPS client
Pod worker-7866cf9ff5-7skfq: Failed - Error: ErrImagePull
Pod worker-7866cf9ff5-7skfq: BackOff - Back-off pulling image "127.0.0.1:5000/vote-demo-quickstart/worker:v-e2e51487b4"
Pod worker-7866cf9ff5-7skfq: Failed - Error: ImagePullBackOff

Does the registry require tls? Can I provide it cert-manager issuer or something? It seems you are missing a page referenced in your documentation also. I would have liked to have it. On this page https://docs.garden.io/kubernetes-plugins/remote-k8s/configure-registry This (non) page is referenced https://docs.garden.io/guides/in-cluster-building

helm modules and crds?

tall-greece-98310

03/01/2023, 11:24 AM

Hi, I'm trying to use a helm chart that has a

crds

directory alongside the typical templates directory. It doesn't seem garden propagates this directory into the

.garden/build/

hierarchy, so the crds are hence not installed. Is this intended or a bug? If intended how are you supposed to get CRDs installed? Thanks!

Local docker registry

blue-kite-93685

03/03/2023, 10:48 AM

I'm having some trouble figuring out how garden builds interact with docker registries. I'm trying to develop entirely locally for the moment. Whether I use local-docker or cluster-buildkit, my deploys fail with "Error response from daemon: pull access denied for image, repository does not exist or may require 'docker login'", which doesn't seem right. The images in question do visibly exist as far as my host docker daemon is concerned. What do I need to do for garden to be able to deploy them?

var.* on the environment level

nutritious-kitchen-29367

03/03/2023, 12:36 PM

Hello. I'm insecure is it bug or feature that's why I decided to write here first. I use individual variables set with --var in CLI flags and define it on the environments level in the project file but I have the error. For example: use this command -> garden deploy --var name=username have this line in project file -> defaultNamespace: user-${var.name} get this error -> Invalid template string (user-${var.name}): Could not find key name under var.

Using Google managed cert (ManagedCertificate) rather than Secret

fresh-yak-35965

03/03/2023, 11:52 PM

Hello. I have a ManagedCertificate object that I would prefer to use over the

tlsCertificate.secretRef

that wants a k8s secret. How do I do this?

Dig out a value from a file on disk in a service via a task?

tall-greece-98310

03/06/2023, 11:02 AM

Hi, I was hoping to dig out a value from a file stored on disk in one of my services via a simple

run task

that does a

cat

of the file, then capture this via task output.log and I can then use this as inputs for other tasks etc. It turns out that the task script is run in a separate container, that does not have the required volume mounted, and hence the approach fails. Any tips to accomplish what I want? Any details on how the task container is specified to run? Does it run as a separate container in same pod or something like that?

flaky module resolution: Could not find build dependency foo, configured in module bar

brief-dawn-88958

03/07/2023, 2:02 PM

Hi garden team! I've been helping out raising issues in github earlier but today I'd like to use discord because I feel the problem at hand is super weird and might need some back and forth. In our setup we have around 5 remote sources. When issuing

garden update-remote source --parallel

(via workflow), the relevant log looks like in attached file

garden.log

. So far, so good. However, sometimes -- maybe every 30th run -- the module resolution mysteriously fails. See attached

garden-error.log

. Please note that in the successful run there are 44 modules; in the failed run garden only finds 39: 5 are missing. When retrying the failed pipeline, it normally works. I'd be really happy to eliminate this flakiness in our CI 🙂 Do you have any idea what is going on?

garden garden-error