https://garden.io logo
#🌱|help-and-getting-started
Getting out of memory errors
a

aloof-lamp-69262

06/26/2023, 2:13 PM
Our garden environment is huge, > 80 modules. Recently I started getting these errors (please check the screenshot). Is there an easy way to increase the memory that the CLI uses? Thank you.

https://cdn.discordapp.com/attachments/1122892487065866303/1122892487304958134/Image_6-26-23_at_10.12_AM.jpeg

Btw, we optimized the include/exclude and we are all set there. I made sure we are ignoring things like node_modules and the likes
b

bright-policeman-43626

06/26/2023, 2:59 PM
This is definitely a new one, do you get this kind of issue while doing actions like
deploy
? Or when are you getting this errors?
a

aloof-lamp-69262

06/26/2023, 3:13 PM
yes, this is on deploy actions
it works a lot and many of the modules are all there
but towards the end it just dies like this.
Is there a command line argument that I can use to increase the memory?
I tried increasing node memory
NODE_OPTIONS=--max-old-space-size=4096
but the CLI just completely stops working, no feedback, no output nothing.
garden scan
works without issues
How can I troubleshoot this?
b

bright-policeman-43626

06/26/2023, 3:17 PM
Are you using Kaniko or some sort of cluster-in-building or everything is being built in Docker side?
I'm asking this because I'm wondering if the out-of-memory are coming from the in-cluster building or your CLI
a

aloof-lamp-69262

06/26/2023, 4:02 PM
We are using a k8 cluster
sorry not sure what kaniko is
a

astonishing-tomato-18259

06/26/2023, 4:17 PM
@aloof-lamp-69262 Looks similar to: https://github.com/garden-io/garden/issues/4638 Did the same garden project work fine with 0.12 without such errors? Could you maybe provide us with more detailed logs by running your garden command with
-l5
(e.g.
garden validate -l5
) and the output of
garden get config
. Perhaps you could share this information privately by adding this info to a private gist and give access? The more information we have, the easier it'd be to investigate it. Thanks.
a

aloof-lamp-69262

06/26/2023, 4:27 PM
Sure, I will try to get this information ready for you soon
I am back tracing what I did in the last few days, trying to get some config that I changed that started causing this, I will keep you posted
b

bright-policeman-43626

06/26/2023, 4:39 PM
Have you added more microservices in the last few days? You just moved to 0.13 right?
a

aloof-lamp-69262

06/26/2023, 7:11 PM
I went back 1 week in my git commits and still getting the same error. I am thinking it has something to do with my garden cache now?
Now we are still on
0.12.31
How can I "refresh" garden or delete it's temporary files? is it under my home profile? .garden?
I am gonna try uninstall/install
Wow that -l5 spits a lot of logs 😄
k

kind-france-89777

06/26/2023, 7:17 PM
It’s called silly log level for a reason 😂
Fwiw, 0.12.31 is an extremely old version, it’s quite possible that your issues have been addressed since the
a

aloof-lamp-69262

06/26/2023, 7:40 PM
I am not sure I can upgrade, we are forced to use specific versions that have been tested and vetted by our company
Ok, I tried to install the latest version manually, it is not backward compatible with our configuration...
@bright-policeman-43626 What is a good way to send you the logs? Can I pm them?
b

bright-policeman-43626

06/27/2023, 2:08 PM
Yes sir, you can PM them to me and @astonishing-tomato-18259 so we can take a look into this with a greater detail!
a

aloof-lamp-69262

06/27/2023, 2:15 PM
Thank you! will do that shortly
b

bright-policeman-43626

07/05/2023, 3:46 PM
Hey @aloof-lamp-69262 we analyzed your silly logs and we couldn't find OOM logs on them, we noticed that there is a service that is
crashloopbackoff
whic might be the cause of your failure. Since you have 80 services it might be hard to detect it but you could try to open another terminal and grep for that string!
kubectl get pods -A |grep Crash
a

aloof-lamp-69262

07/06/2023, 2:22 PM
Got it, ok, I will try to troubleshoot with this. So this crash might be exhausting the memory?
b

bright-policeman-43626

07/06/2023, 2:34 PM
Yes, or Garden is cutting the operation after the default timeout after waiting for all your services to stabilize