We've noticed that, for some reason, JVM remote debugger connections tend to be flakey using the built in garden port-forwarding when connecting a remote debugger to a service in Kubernetes Let's say I have service-x that has a devMode startup command like:

java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -cp /app/app.jar

This starts up the Java server and also listens for incoming remote debugger client connections on port 5005. Then, in our garden.yaml for the service we have configuration like so:

services:
  - name: "service-x"
    command:
      - "java"
      - "-cp"
      - "/app/app.jar"
    devMode:
      command:
        - "java"
        - "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"
        - "-cp"
        - "/app/app.jar"
      sync: []
    ports:
      - name: http
        containerPort: 80
        servicePort: 80
        localPort: 8080
      - name: debug-port
        containerPort: 5005
        servicePort: 5005
        localPort: 10000

When launched in devMode, the port forwarding for port 10000 is in place, however multiple attempts to connect the remote debugger result in various forms of error "Unable to open debugger port (localhost:100000): java.io.IOException" and sometimes this includes "handshake failed - connection prematurely closed". If we keep retrying this, eventually it will successfully connect. Once connected, it will usually be able to re-connect again reliably after a disconnect for the duration of the pod's life. Initially - my thought was that there was some sort of lag between when the service was responding on port 80 (meaning the garden health check saw the service as up) and when the remote debugger was able to accept connections. HOWEVER - when I tried directly creating a kubectl port forward using a command like

kubectl port-forward service/service-x 50005:10001

the debugger is immediately and reliably able to connect as soon as the service is deployed.

@big-spring-14945 do we have any JVM experts on staff to help Brett?

For the time being at least, we've hacked in some additional scripting for our developers so that when they start a debug session we'll disable garden's port forwarding and spin up a kubectl port-forward process in the background instead. Obviously though this is less than ideal

Really interesting, found this issue from 2019, someone having the exact same issue like you: https://github.com/garden-io/garden/issues/967

@quaint-librarian-55734 Hi brett, thanks for reporting this. Can you run garden in silly log level (

-l5

command line option) and compare the kubectl command with the command you're using? Also does the timing of Garden killing / re-starting the kubectl correlate with your disconnect behaviour? I have the feeling that Garden might not set up a port-forward to the service, but to the Pod directly, and thus it must restart it from time to time, but as it's easier for you to reproduce this I would be very happy if you could do a further analysis and point me at the lines of log that correlate with the behaviour that you noticed.

Hi @big-spring-14945 - sorry it's taken me a few days to respond, we've been pretty busy with a rollout and I haven't had much time yet to gather the requested data. I'm hoping to get this pulled together today or tomorrow