JVM remote debugger connection flakey with garden port forwarding


01/19/2023, 11:24 PM
We've noticed that, for some reason, JVM remote debugger connections tend to be flakey using the built in garden port-forwarding when connecting a remote debugger to a service in Kubernetes Let's say I have service-x that has a devMode startup command like:
java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -cp /app/app.jar
This starts up the Java server and also listens for incoming remote debugger client connections on port 5005. Then, in our garden.yaml for the service we have configuration like so:
  - name: "service-x"
      - "java"
      - "-cp"
      - "/app/app.jar"
        - "java"
        - "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"
        - "-cp"
        - "/app/app.jar"
      sync: []
      - name: http
        containerPort: 80
        servicePort: 80
        localPort: 8080
      - name: debug-port
        containerPort: 5005
        servicePort: 5005
        localPort: 10000
When launched in devMode, the port forwarding for port 10000 is in place, however multiple attempts to connect the remote debugger result in various forms of error "Unable to open debugger port (localhost:100000):" and sometimes this includes "handshake failed - connection prematurely closed". If we keep retrying this, eventually it will successfully connect. Once connected, it will usually be able to re-connect again reliably after a disconnect for the duration of the pod's life. Initially - my thought was that there was some sort of lag between when the service was responding on port 80 (meaning the garden health check saw the service as up) and when the remote debugger was able to accept connections. HOWEVER - when I tried directly creating a kubectl port forward using a command like
kubectl port-forward service/service-x 50005:10001
the debugger is immediately and reliably able to connect as soon as the service is deployed.
Additionally - we have Python services that use a similar pattern for starting a remote debug server when started in devMode, however we've never had a problem connecting the remote debuggers to these services. This leads me to believe that it's something related to the jvm remote debugger, but I don't know the internals of how garden port forwarding and/or the jvm debugger network protocol is implemented to even speculate at a cause.


01/23/2023, 1:50 PM
@big-spring-14945 do we have any JVM experts on staff to help Brett?


01/23/2023, 5:02 PM
For the time being at least, we've hacked in some additional scripting for our developers so that when they start a debug session we'll disable garden's port forwarding and spin up a kubectl port-forward process in the background instead. Obviously though this is less than ideal
Also - I'd be curious to know if Garden uses something other that kubectl's mechanism for setting up the port forwards when a service is launched in devMode


01/24/2023, 11:01 PM
Really interesting, found this issue from 2019, someone having the exact same issue like you:
Maybe at this point in time this is legacy code but this is how the issue was "solved" for that specific issue:
Is there anything you can reuse from the configuration used in that old issue?


01/26/2023, 11:32 AM
@quaint-librarian-55734 Hi brett, thanks for reporting this. Can you run garden in silly log level (
command line option) and compare the kubectl command with the command you're using? Also does the timing of Garden killing / re-starting the kubectl correlate with your disconnect behaviour? I have the feeling that Garden might not set up a port-forward to the service, but to the Pod directly, and thus it must restart it from time to time, but as it's easier for you to reproduce this I would be very happy if you could do a further analysis and point me at the lines of log that correlate with the behaviour that you noticed.


01/31/2023, 3:26 PM
Hi @big-spring-14945 - sorry it's taken me a few days to respond, we've been pretty busy with a rollout and I haven't had much time yet to gather the requested data. I'm hoping to get this pulled together today or tomorrow