Remote kubernetes garden-docker-registry unbound pvc (v0.12)

Question

Hi all I have a remote kubernetes cluster setup and am trying to deploy my stack against it with garden as I ve been doing locally with minikube but am having some issues I m using this guide https docs garden io v acorn 0 12 kubernetes plugins remote k8s but I skipped step 2 and 3 as I want to build on cluster without an external docker registry and don t think I need a hostname as I ll just use portforwards to access the services I deploy There will only be one garden environment on this cluster Issue I m having is when I run `garden env=remote plugins kubernetes cluster init` I see it creates the `garden system` resources on the cluster but the `garden docker registry` pod is pending Looking into it it appears a PVC its using `garden docker registry` is also pending with the following event ``` 0 4 nodes are available 4 pod has unbound immediate PersistentVolumeClaims preemption 0 4 nodes are available 4 Preemption is not helpful for scheduling ``` I don t see anything in the guide about setting up persistent volumes Do I need to do something get this pvc to work

quaint-dress-831 · Answer

Hi < average painter 87026> what service provider have you chosen for your Kubernetes cluster I d `get` your `pv` `pvc` and `storageclass` and make sure they re available

quaint-dress-831 · Answer

Have you set up an in cluster build mode If so I d check https docs garden io v acorn 0 12 reference providers local kubernetes providers storage registry and make sure your storage is configured correctly to your provider

average-painter-87026 · Answer

< quaint dress 831> Sorry got side tracked with work but finally back around to this I was able to get the in cluster build pods to run by creating a persistent volume on the cluster attached and the `garden system garden docker registry` PVC bound itself to it https cdn discordapp com attachments 1118609692751564860 1123649968100606063 remote garden build pv yaml

average-painter-87026 · Answer

I m using the `cluster buildkit` and the build seems to work but there are issues now when it tries to push the built image to it s local registry The `garden build <service>` logs are showing `500 Internal Server Error` when pushing the image and I see similar error in `garden buildkit` pod I ve attached both The `garden docker registry` pod in `default` namespace is logging this error `filesystem mkdir var lib registry docker permission denied` I ve attached those logs as well It seems like the garden docker registry service isn t able to write to that hostPath persistentVolume in previous post even though it has RWO access modes I ve tried using ` tmp garden build` for the hostPath of the PV but still getting the permission issue so I m not totally sure Do you have any ideas what I should be doing here Thank you https cdn discordapp com attachments 1118609692751564860 1123651419216556122 garden build service log https cdn discordapp com attachments 1118609692751564860 1123651419522736279 garden docker registry log https cdn discordapp com attachments 1118609692751564860 1123651419866681454 garden buildkit log

quaint-dress-831 · Answer

< average painter 87026> can you please just dump exactly what you ve written into a GitHub Issue This looks like a bug we need to look into

quaint-dress-831 · Answer

In the meantime if you re able to switch to another local K8s provider we support like MicroK8s or kind that should unblock you for now

quaint-dress-831 · Answer

More on supported local flavors and instructions for each at https docs garden io kubernetes plugins local k8s install

quaint-dress-831 · Answer

< average painter 87026> I am setting this to resolved since it s been some time since I ve last heard from you If this is still an issue just ping me here