Garden commands hang on Mac OS X 11.1 after initial "garden deploy"

So I’m looking at using garden to build a consistent development environment with my companies production / staging environments. We’ve gravitated to garden due to it’s support with Helm and it’s helm like structure. I’ve been working on getting a simple PoC setup but am running into a number of problems just getting garden to work reliably. I’m using the recommend setup (Docker for Mac) with Kubernetes enabled (ofc).

When I run the command garden deploy after initializing a project. It runs perfectly the first time outputting.

✔ providers → Preparing environment... → Done ✔ local-kubernetes → Configuring... → Ready ✔ providers → Getting status... → Cached ℹ Run with --force-refresh to force a refresh of provider statuses. ✔ providers → Getting status... → Cached ℹ Run with --force-refresh to force a refresh of provider statuses. ✔ default-backend → Getting build status for v-63466f0842... → Done (took 0 sec) ✔ default-backend → Deploying version v-63466f0842... → Done (took 8.4 sec) ℹ default-backend → Resources ready ✔ ingress-controller → Building version v-82d30a9ab9... → Done (took 5 sec) ✔ ingress-controller → Deploying version v-82d30a9ab9... → Done (took 28.6 sec) ℹ ingress-controller → Resources ready

After this first launch subsequent “deploy” calls cause the “providers → getting status” section to completely hang. If I cancel the command that is doing absolutely nothing I notice that a left over process consuming around 100% cpu of a single core is left hanging around. I’ve tried leaving it alone for 20 - 30 minutes and it does absolutely nothing but just consume resources etc.

I can still access my kubectl environment but garden is just dying very silently doing nothing. I’m not really sure what else to include would appreciate it if someone could give me some advice.

Thanks

Thanks for flagging this @djowinz! I’m trying to see if I can reproduce. Just upgraded to macOS Big Sur to see if the issue relates to that specifically. Will report back.

1 Like

I’m so far failing to reproduce, so we might need more info. That process that’s just spinning at 100% CPU should give us a better hint though.

Can you trigger that to happen, and then run ps -eo args <PID of hanging process>? Knowing what exact process is spinning would help figure this out.

It looks like the entire thing hangs specifically with the UI. For instance now when I do garden create project it hangs and I’m unable to input in anything through the CLI interface. Here is the process that is currently spinning.

dyllenowens 50127 98.7 1.2 5819660 203484 s000 R+ 5:27PM 1:07.54 /usr/local/Cellar/garden-cli/0.12.16/libexec/garden /snapshot/project/tmp/pkg/cli/bin/garden create project

Here is the args output of the above process running ps -eo args 50126
ARGS /usr/local/Cellar/garden-cli/0.12.16/libexec/garden /snapshot/pr

What is interesting is that it’s calling garden twice? Hopefully this is helpful thank you @JonEdvald for the support in this! We really, really want to use this for our local flow.

Hmm that’s interesting. Which terminal are you using? Anything else noteworthy with respect to your OS/environment/etc?

The two processes are because the parent process forks itself to better handle certain issues. It’s plausible that the issue has something to do with that, or potentially something specific in the environment that differs from ours.

And of course, happy to help! This is a critical issue and I’d hate for this to be happening to others as well.

There isn’t anything specific that I can think of that is particularly noteworthy and or worth standing out. Are there things that I should be specifically looking for that might cause issues like this, is there a debug mode for garden that I can enable to get more output?

You could run the command with --log-level=silly and see what comes up at the end. I’ve been trying several different builds and terminals and haven’t come anywhere close, and I’m so far at a loss :stuck_out_tongue:

But I’m here for it, and adding a higher log level would make sense to try next.

Since it appears to happen with even a simple command like garden create project, maybe try that one with the higher log level, instead of e.g. garden deploy (since that would spew out a lot of unrelated logs potentially).

I honestly could not drive to what the root cause was and my environment did have a lot of other tools built onto it. I develop a lot of node.js applications as well as a lot of different applications with different version of node.js. Perhaps there is an issue with a npm package I installed locally and the tool you are using to display the input’s via the CLI. I ended up just completely wiping my machine, “I know”, it’s crazy but I need to get this project moving forward and getting blocked without a reliable replication for the bug sounds like a self created issue.

Thanks @JonEdvald for the assistance here, sad we couldn’t spend more time driving to the root cause of this problem.

That’s the nuclear option right there! But that did the trick then? I suppose it’s possible that it had something to do with a system library or something like that, which might explain why we couldn’t reproduce it… In any case ping back if it comes up again!

Yeah everything is great right now! Really loving Garden I DM’d you @JonEdvald with a personal question but I think I might have found the solution nested in the docs.

Glad that’s sorted! And yea happy to help on that other thread if needed.