Cluster-buildkit mode not working

Basically garden eventually times out when trying to use buildkit for builds (kaniko works fine). Buildkitd logs are as follows:

time="2022-09-20T19:55:41Z" level=info msg="auto snapshotter: using overlayfs"
time="2022-09-20T19:55:41Z" level=warning msg="using host network as the default"
time="2022-09-20T19:55:41Z" level=info msg="found worker \"80gpbxysig5pa3gu35mecyzqq\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:garden-buildkit-56bf5fdddb-5x9wb org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/386]"
time="2022-09-20T19:55:41Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2022-09-20T19:55:41Z" level=info msg="found 1 workers, default=\"80gpbxysig5pa3gu35mecyzqq\""
time="2022-09-20T19:55:41Z" level=warning msg="currently, only the default worker can be used."
time="2022-09-20T19:55:41Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
time="2022-09-20T19:59:43Z" level=info msg="trying next host - response was http.StatusNotFound"

Not much to go on – note that it takes a good 4 minutes to get the “trying next host” message.

Hi @mowings

I’ve never seen this before and have no idea.
Can you share more information on that with us? e.g. your, any garden log messages? Will garden log anything suspicios with -l5 (“silly” log level)? Garden version? Kubernetes provider / version? Or best would be of course a reproducible example.

If you share portions of the logs here, make sure they don’t accidentally contain any secret information.