Hi, I have remote cluster that is using private registry hosted on the same cluster. I can build and push to that registry local machine no problem. However, when I use either kaniko or cluster-buildkit I get self-signed certificate error.

bash
Unable to query registry for image status: Command "skopeo --command-timeout=30s inspect --raw --authfile ~/.docker/config.json docker://<PRIVATE_REGISTRY>:31570/garden/mongodb:v-778f5ee81b" failed: Failed with exit code 1.

Here are the logs until the error occurred:

time="2024-11-13T03:05:12Z" level=fatal msg="Error parsing image name \"docker://<PRIVATE_REGISTRY>:31570/garden/mongodb:v-778f5ee81b\": pinging container registry <PRIVATE_REGISTRY>:31570: Get \"https://<PRIVATE_REGISTRY>:31570/v2/\": tls: failed to verify certificate: x509: certificate signed by unknown authority"

Certificate is added to both mac os keychain and in ~/.docker/certs.d/ folder. Docker on both local and remote machines can push and pull images from registry, it seem to be problem only related to k8s-util image which runs scopeo.

Hi @flat-dentist-71611 , garden-util as well as cluster-buildkit and kaniko deployed by garden are all running in pods in the remote cluster. We currently don't have a mechanism to tell this pods to accept a self-signed certificate. 1. If you are using a remote kubernetes cluster you could consider using a valid lets-encrypt certificate for your registry. Would that work? Or you could put the registry behind a loadbalancer with a valid certificate or proxy it through another service that provides tls e.g. cloudflare ? 2. If you are using a local kubernetes cluster, there is not really much benefit in using the remote builder solutions, but it is more performant to use the local docker build mechanism. 3. There is an option to tell garden not to use tls for the registry but if it is a remote cluster reachable over the internet that is not really advisable https://docs.garden.io/reference/providers/kubernetes#providers-.deploymentregistry.insecure