Ingresses failing to start on linux

We recently noticed that ingresses are failing to start on linux hosts. One day things were good on 0.12.27, the next the ingresses are failing to start.

Our software versions are:
garden: 0.12.27
minikube: 1.22.0
kubernetes client: 1.20.10
kubernetes server: 1.20.7
docker client: 20.10.8
docker server: 20.10.6

The error we are seeing is:

✖ smtp                      → Deploying version v-4e468c8c1b...

Failed deploying service 'smtp' (from module 'mailhog'). Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Command "/home/cmcculler/.garden/tools/kubectl/11ac0bbd21058c50/kubectl --context=minikube --namespace=pardot apply --prune
--selector service=smtp --output=json -f -" failed with code 1:

Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook
"validate.nginx.ingress.kubernetes.io": Post
"https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1beta1/ingresses?timeout=10s": dial tcp
10.100.168.49:443: connect: connection refused

Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook
"validate.nginx.ingress.kubernetes.io": Post
"https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1beta1/ingresses?timeout=10s": dial tcp
10.100.168.49:443: connect: connection refused

When querying kubectl for the validation webhooks we are definitely seeing them being created:

$ kubectl get validatingwebhookconfigurations
NAME                      WEBHOOKS   AGE
ingress-nginx-admission   1          33m

If we delete them they are recreated when starting garden again. This issue suggests they should not be created but maybe the ingress helm chart is overriding that?

This literally cropped up for us overnight and we’re at a loss as to how to resolve it.

Thanks in advance! Please let me know if you need additional details!

Hi @cmcculler ,

So the issue is that you are using minikube 1.22 and the api version extensions/v1beta1 is unavailable since kubernetes 1.22. You can see that in the warning message that you posted:

Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

Are the ingresses defined in container modules or kubernetes or helm modules?

Best,

Anna

Hi Anna,

The ingresses are all defined on container modules. We are using minikube 1.22 but are setting the kubernetes version to 1.20.7 via the --kubernetes-version minikube flag so the extensions/v1beta1 should still be available to us.

Also, for no reason that I can discern, things are working perfectly fine this morning. When I left off troubleshooting last night I was consistently getting the ingress validation timeout. This morning I started garden up again on a whim and it all Just Worked™.

Hi @cmcculler,

Ah right, sorry I overlooked the kubernetes server version. As for the container modules, we’ll have to look into the ingress api versions going forward.

This pull request disabled admission webhooks for the Garden bundled NGINX ingress controller, but I assume you are using the minikube addon?

I am also baffled, that it just stopped and then started working again. Did the problem return since?

Best,

Anna

yep as of this post everything is still firing up just fine. There is a slight chance this was caused by business technology group experimenting with new iptables rules. They have some … odd … rules on our linux systems that can make things difficult for docker and friends. We have no warning when they deploy updates and no real way to see if anything was deployed. :frowning:

Thanks for working with us on this! we can probably mark this resolved with a cause of "gremlins ¯_(ツ)_/¯ "