Waiting for k8s resources

Question

Is there a way I can explicitly define how to wait for certain resources within a kubernetes deploy action Bonsai Garden seems to not care about completeness of Jobs and the number of running replicas in statefulsets for example if pods are waiting for a volume I would like to state that my action should be considered complete based on some value in the resource s state if that is possible

calm-family-29323 · Answer

Garden will mark deploy ready or complete once it s ready or complete on the kubernetes side With kubernetes you can describe how to set and monitor your resource s state via liveness readiness and startup probes 1 For more advanced cases that can t be configured with kubernetes you could create a Run action as a dependant of the Deploy action that checks for the status more granularly and completes once desired state is reached If there is a lot of demand for such functionality we might think of implementing it officially into Garden to make it more convenient https kubernetes io docs tasks configure pod container configure liveness readiness startup probes

icy-furniture-17516 · Answer

I still beleive something is not working correctly let me show an example I ve declared a statefulset in my kubernetes yamls and added it to a kubernetes deploy action I intentionally defined a volumemount that has no corresponding volume defined so that the pod won t be able to start and garden happily reports the deployment as done

icy-furniture-17516 · Answer

``` garden deploy Deploy Garden v0 13 Bonsai is a major release with significant changes Please help us improve it by reporting any issues bugs here https go garden io report bonsai Run garden util hide warning 0 13 bonsai to disable this warning ℹ garden Running in Garden environment default default node 84105 ExperimentalWarning The Fetch API is an experimental feature This feature could change at any time Use `garden trace warnings ` to show where the warning was created ℹ cloud dashboard Connected to Cloud Dashboard View logs and command results at https app garden io projects ea10de63 0eb0 472d 8f76 ffe426c0cc24 sessionId=5dfab169 54dd 49eb a29c 2b6ad6f67cf9 userId=72cd8adc fd35 4651 9d9e 1a2b6e15479a ```

icy-furniture-17516 · Answer

``` ℹ providers Getting status providers Cached took 2 6 sec ℹ providers Run with force refresh to force a refresh of provider statuses ℹ graph Resolving actions and modules graph Done took 0 7 sec ℹ deploy kcp cert manager crds missing ℹ deploy kcp outdated ℹ deploy kcp account outdated ℹ deploy kcp cert Already deployed ℹ deploy kcp cert manager Already deployed ℹ build kcpctl Already built ℹ deploy kcp Deploying version v 615bf9f5c5 ℹ deploy ingress nginx Already deployed ℹ deploy kcp Waiting for resources to be ready ℹ deploy cert manager Already deployed ℹ deploy kcp Resources ready deploy kcp Done took 2 6 sec ℹ deploy kcp Ingress http kcp playground garden ℹ deploy kcp account Deploying version v 5f59d745d4 ℹ deploy kcp account Waiting for resources to be ready ℹ deploy kcp account Resources ready deploy kcp account Done took 2 6 sec ℹ deploy kcp cert manager crds Deploying version v ff63574bea ℹ deploy kcp cert manager crds Waiting for resources to be ready ℹ deploy kcp cert manager crds Resources ready deploy kcp cert manager crds Done took 2 6 sec Done ```

icy-furniture-17516 · Answer

``` k get sts NAME READY AGE kcp 0 1 19s kcp cert manager 1 1 4h15m ```

icy-furniture-17516 · Answer

``` k describe sts kcp Name kcp Namespace default CreationTimestamp Tue 23 May 2023 14 45 27 +0200 Selector app=kcp Labels garden io service=kcp Annotations garden io manifest hash 09956a2fe4e9ece9e213b5821ece0e5ad1b0dd81cdfeecef8e604bbed9b0d1ba garden io mode default garden io service kcp Replicas 1 desired | 0 total Update Strategy RollingUpdate Partition 0 Pods Status 0 Running 0 Waiting 0 Succeeded 0 Failed Pod Template Labels app=kcp Init Containers cert bundle Image kcpctl v 79a98dd0ef Port Host Port Command bin sh Args cxe cat etc kcp certificate tls crt etc kcp certificate ca crt > data tls bundle crt || true Environment Mounts data from data rw etc kcp certificate from kcp certificate rw Containers kcp Image ghcr io kcp dev kcp 530d15f Port 6443 TCP Host Port 0 TCP Args start root directory= data external hostname=kcp tls cert file= data tls bundle crt tls private key file= etc kcp certificate tls key client ca file= etc kcp certificate ca crt Environment Mounts data from data rw etc kcp certificate from kcp certificate rw ```

icy-furniture-17516 · Answer

``` Volumes kcp aasdasd Type Secret a volume populated by a Secret SecretName kcp server Optional false data Type PersistentVolumeClaim a reference to a PersistentVolumeClaim in the same namespace ClaimName kcp data ReadOnly false Volume Claims <none> Events Type Reason Age From Message Warning FailedCreate 13s x13 over 33s statefulset controller create Pod kcp 0 in StatefulSet kcp failed error Pod kcp 0 is invalid spec containers 0 volumeMounts 1 name Not found kcp certificate spec initContainers 0 volumeMounts 1 name Not found kcp certificate ```

icy-furniture-17516 · Answer

this is the statefulset status ``` status availableReplicas 0 collisionCount 0 currentRevision kcp ddc5b66b5 observedGeneration 1 replicas 0 updateRevision kcp ddc5b66b5 ```

icy-furniture-17516 · Answer

ftr I used this workaround ``` kind Run description Wait for kcp to start name kcp wait type exec spec command kubectl rollout status watch timeout=120s n $ environment namespace statefulset kcp dependencies deploy kcp ```

icy-furniture-17516 · Answer

and for the job ``` kind Run description Wait for kcp account to be completed name kcp account wait type exec spec command kubectl wait for=condition=complete timeout=60s n $ environment namespace job kcp account dependencies deploy kcp account ```

calm-family-29323 · Answer

Hmm yes this looks like undesired behaviour I ll be on it thanks for bringing it up

calm-family-29323 · Answer

< icy furniture 17516> would you mind sharing a basic repro that would speed us up with this a lot

icy-furniture-17516 · Answer

of course I created a branch that has an sts with a non existent volume to mount https github com pepov kcp playground tree statefulset pods not starting

icy-furniture-17516 · Answer

installation is in the readme but it requires minikube and bonsai and to reproduce just run ``` garden deploy kcp ```

icy-furniture-17516 · Answer

it will pass but the statefulset will be unavailable

icy-furniture-17516 · Answer

https cdn discordapp com attachments 1110512582734917684 1110833967118176286 message txt

icy-furniture-17516 · Answer

it starts in 85 sec for me so I hope this is still easy enough to work with

calm-family-29323 · Answer

thank you

icy-furniture-17516 · Answer

thanks for looking into this I m okay with the workaround for now but I think this would be a very nice thing to have I specifically only care about Jobs Deployments Statefulsets to complete start properly and for that we don t need anything custom I think although I was asking for that initially

icy-furniture-17516 · Answer

I can check whether it worked in 0 12 or not but I have to prepare for this talk tonight

calm-family-29323 · Answer

I ve filed a pr that should fix the issue https github com garden io garden pull 4430 Again thank you for the repro

icy-furniture-17516 · Answer

nice thanks

icy-furniture-17516 · Answer

you don t have handlers for kubernetes Jobs

icy-furniture-17516 · Answer

the source code references the helm libraries which have handler for jobs but I couldn t find it in the garden code

icy-furniture-17516 · Answer

https github com helm helm blob main pkg kube ready go LL225C7 L225C7

calm-family-29323 · Answer

Currently there s no custom wait handler for jobs which basically means we think it s ready immediately https github com garden io garden blob k8s sts state core src plugins kubernetes status status ts L71

icy-furniture-17516 · Answer

referring to this line https github com garden io garden blob 4c84e73597a26e192da1ebbf79ca02816d5a4ec7 core src plugins kubernetes status status ts L71

icy-furniture-17516 · Answer

okay same

icy-furniture-17516 · Answer

can that be a possible opportunity to contribute

icy-furniture-17516 · Answer

first I guess I should just create a feature request and then if it doesn t get priority from your side I could try to give it a shot

icy-furniture-17516 · Answer

although I don t speak typescript but chatgpt surely does

calm-family-29323 · Answer

Sure if you could create an issue that describes the expected behaviour and maybe even includes a repro we could get it done quite quickly

icy-furniture-17516 · Answer

will do thanks