https://garden.io logo
#🌱|help-and-getting-started
How can I disable garden copying certificate secrets to other namespaces?
# 🌱|help-and-getting-started
b

best-flower-46100

12/06/2023, 3:24 PM
I'm using cert-manager to generate certificates and I use one certificate with wildcards in the default namespace. For a long time I assumed that cert-manager automatically copies over the secret to all the other namespaces. I've had this issue though where certifications would often expire on prod because only the default secret was renewed (the one in the namespace where certificate resource lived), the secrets in other namespaces had to be updated manually and I'd blame cert-manager until I read that it doesn't do the copying - it's garden that does and it seems to only update during deployment. I wanted to take that over and use kubernetes-reflector: https://cert-manager.io/docs/devops-tips/syncing-secrets-across-namespaces/#using-reflector Unfortunately, the secrets copied by garden interfere with that method. I wanted to disable the copying mechanism but it seems not possible. Is there a way to achieve that or make garden update the secrets as soon as they change?
q

quaint-dress-831

12/07/2023, 9:05 AM
Hi @best-flower-46100, according to cert-manager's docs, the ingress-nginx controller Garden deploys by default should support specifying a single certificate for multiple namespaces. See https://cert-manager.io/docs/devops-tips/syncing-secrets-across-namespaces/#serving-a-wildcard-to-ingress-resources-in-different-namespaces-default-ssl-certificate The requirement is
tls.[].secretName
must omitted. Can you clarify if you're using
tls.[].secretName
or not? If you are, can you try without?
b

best-flower-46100

12/07/2023, 9:18 AM
yes i am actually, i'll try it out as soon as i'm done with my current task. thanks!
Encountered an unexpected Garden error. This is likely a bug 🍂 You can help by reporting this on GitHub: https://github.com/garden-io/garden/issues/new?labels=bug,crash&template=CRASH.md&title=Crash%3A%20Cannot%20read%20properties%20of%20undefined%20(reading%20%27name%27) Please attach the following information to the bug report after making sure that the error message does not contain sensitive information: Failed resolving status for Deploy type=container name=platform-ui (took 1.5 sec). This is what happened: TypeError: Cannot read properties of undefined (reading 'name') at getCertificateHostnames (file:///Users/dawidstezycki/Library/Application%20Support/io.garden.garden/1701437467-Is7gZ13.r/rollup/garden.mjs:776044:81) at pickCertificate (file:///Users/dawidstezycki/Library/Application%20Support/io.garden.garden/1701437467-Is7gZ13.r/rollup/garden.mjs:776079:37) at getIngress (file:///Users/dawidstezycki/Library/Application%20Support/io.garden.garden/1701437467-Is7gZ13.r/rollup/garden.mjs:775999:31) at file:///Users/dawidstezycki/Library/Application%20Support/io.garden.garden/1701437467-Is7gZ13.r/rollup/garden.mjs:776020:48 at Array.map () at getIngressesWithCert (file:///Users/dawidstezycki/Library/Application%20Support/io.garden.garden/1701437467-Is7gZ13.r/rollup/garden.mjs:776020:34) at createIngressResources (file:///Users/dawidstezycki/Library/Application%20Support/io.garden.garden/1701437467-Is7gZ13.r/rollup/garden.mjs:775903:32) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async createContainerManifests (file:///Users/dawidstezycki/Library/Application%20Support/io.garden.garden/1701437467-Is7gZ13.r/rollup/garden.mjs:777906:23) at async AsyncFunction.k8sGetContainerDeployStatus (file:///Users/dawidstezycki/Library/Application%20Support/io.garden.garden/1701437467-Is7gZ13.r/rollup/garden.mjs:777750:37)
q

quaint-dress-831

12/07/2023, 11:15 AM
Hi @best-flower-46100 can you report this as a bug using the link provided?
@best-flower-46100 can you also try downgrading to 0.13.19 and re-running your command?
garden self-update 0.13.19
b

best-flower-46100

12/07/2023, 11:23 AM
the bug has been reported, ok let me try downgrading
unfortunately i get the same error :/
q

quaint-dress-831

12/07/2023, 11:25 AM
Ok thanks for giving it a shot! I'll ask our engineers to take a look 👀
b

best-flower-46100

12/07/2023, 11:29 AM
thanks for help 🙏 hopefully I can fix it before christmas so that i don't get a call to fix the certs during a christmas dinner 😄
q

quaint-dress-831

12/07/2023, 11:30 AM
I'm pinging our staff now to see what we can do!
Definitely don't want any Christmas dinners ruined 🎅
cc @astonishing-tomato-18259
b

best-flower-46100

12/08/2023, 2:20 PM
hey guys, is there any progress on that one? i noticed that the reflector thing might work after all, although the annotations seem really weird and messy - i'd have to break some stuff on our dev cluster to properly test it and i don't want to do that if you're close to solving the ingress issue
q

quaint-dress-831

12/08/2023, 2:51 PM
Hi @best-flower-46100 no progress, unfortunately but I can offer my own personal project if you were wanting to use kubed instead of reflector as I do in https://gitlab.hansen.agency/worldofgeese/little_bits_of_buddha/-/blob/main/project.garden.yml?ref_type=heads
If you scroll down there, you can see I just add a label,
kubed
, and that's enough for kubed to sync my secrets
b

best-flower-46100

12/08/2023, 3:18 PM
ok, thanks, i'll look into it
q

quaint-dress-831

12/08/2023, 3:19 PM
I'll be here all week next week as well to support you not having to work over Christmas
f

freezing-pharmacist-34446

12/08/2023, 3:53 PM
Hi @best-flower-46100 are you using the garden provided ingress controller?
The crash happens because garden tries to read the hostname from the certificate which is stored in the secret, but you removed the config that tells garden where to find the secret. This error should be caught and result in a helpful error message though.
b

best-flower-46100

12/08/2023, 4:42 PM
hey! yes, i removed it as I hoped to get the benefit of having a single certificate for all namespaces. If that's causing problems then I guess this way simply isn't supported, right? https://cdn.discordapp.com/attachments/1181979459243495526/1182723963273625620/image.png?ex=6585bc86&is=65734786&hm=5048ac83e384da405ad7cd868c79c48268371c53caf16c65d840cf1fddf1ba29&
f

freezing-pharmacist-34446

12/08/2023, 4:44 PM
The config in garden looks a bit different then directly on the ingress resource. So if you want garden to not copy your secret at all and not reference it in the ingresses, remove the whole entry from the
tlsCertificates
block (or if it is the only entry, the whole block)
Garden will not cleanup any secrets, so if you want to replace them by something that comes from reflector, you will need to delete the garden created ones first. Maybe start that experiment in one dev namespace 🙂
b

best-flower-46100

12/08/2023, 4:49 PM
I would like to use the secrets in the ingress, just not have them be copied by garden. If i remove the entire tlsCertificates then the ingress won't be referencing any secret, right?
The ideal solution would be to point to a secret name but take care of the sync between namespaces myself 😄
f

freezing-pharmacist-34446

12/08/2023, 4:49 PM
Are you using the garden provided ingress controller? Because setting the default certificate would actually be a great usecase that should also allow you to skip copying secrets around alltogether. However that is a value for the nginx ingress controller and we currently don't set it with the garden supplied one.
b

best-flower-46100

12/08/2023, 4:51 PM
yet, I'm using garden provided ingress controller
f

freezing-pharmacist-34446

12/08/2023, 4:51 PM
I think it would be cool to allow doing that via garden and the garden installed ingress controller. However if you need a solution like NOW, i could help you get rid of the garden installed nginx ingress controller and install it via helm instead so you can add custom values like the default-ssl-certificate
Do you have a dev cluster where you could try that out where causing ingress to not work for for a little bit if sth goes wrong at first try doesn't disturb anyone?
Cause i can send you a helm values file, that includes exactly the values garden sets plus the one about the default-ssl-certificate
b

best-flower-46100

12/08/2023, 4:57 PM
Ok, thanks! First, i'll try to figure out if maybe reflector works, even if it's a little messy with annotations. If that doesn't work I will reach out to you, I don't want you to spend time on this without first trying out absolutely all the options myself 😄
I'm signing off for the weekend though, so I'll let you know on monday
f

freezing-pharmacist-34446

12/08/2023, 4:58 PM
Sounds good! I am also signing off for the weekend - have a nice one!
b

best-flower-46100

12/08/2023, 4:59 PM
Have a nice one too, see you! 😄
b

bright-policeman-43626

12/08/2023, 5:01 PM
Are you using the
container
module to deploy this container? Maybe you would benefit from using Helm so you can customize 100% of your resources. As Anna said, if you want to change values in NGINX it would be valuable that you had a separate nginx deployment in the cluster and then in your services use something like
helmet
to template any resources you want in your deployment. This example https://docs.garden.io/garden-seeds/languages/go uses that approach. https://github.com/garden-io/go-seed/blob/main/%7B%7Bcookiecutter.app_name%7D%7D/api.garden.yml