https://garden.io logo
Join Discord
Powered by
Getting out of memory errors
# 🌱｜help-and-getting-started
a
Our garden environment is huge, > 80 modules. Recently I started getting these errors (please check the screenshot). Is there an easy way to increase the memory that the CLI uses? Thank you.

https://cdn.discordapp.com/attachments/1122892487065866303/1122892487304958134/Image_6-26-23_at_10.12_AM.jpeg▾

Btw, we optimized the include/exclude and we are all set there. I made sure we are ignoring things like node_modules and the likes
b
This is definitely a new one, do you get this kind of issue while doing actions like 
deploy
? Or when are you getting this errors?
a
yes, this is on deploy actions
it works a lot and many of the modules are all there
but towards the end it just dies like this.
Is there a command line argument that I can use to increase the memory?
I tried increasing node memory 
NODE_OPTIONS=--max-old-space-size=4096
but the CLI just completely stops working, no feedback, no output nothing.
garden scan
works without issues
How can I troubleshoot this?
b
Are you using Kaniko or some sort of cluster-in-building or everything is being built in Docker side?
I'm asking this because I'm wondering if the out-of-memory are coming from the in-cluster building or your CLI
a
We are using a k8 cluster
sorry not sure what kaniko is
a
@aloof-lamp-69262 Looks similar to: https://github.com/garden-io/garden/issues/4638 Did the same garden project work fine with 0.12 without such errors? Could you maybe provide us with more detailed logs by running your garden command with 
-l5
(e.g. 
garden validate -l5
) and the output of 
garden get config
. Perhaps you could share this information privately by adding this info to a private gist and give access? The more information we have, the easier it'd be to investigate it. Thanks.
a
Sure, I will try to get this information ready for you soon
I am back tracing what I did in the last few days, trying to get some config that I changed that started causing this, I will keep you posted
b
Have you added more microservices in the last few days? You just moved to 0.13 right?
a
I went back 1 week in my git commits and still getting the same error. I am thinking it has something to do with my garden cache now?
Now we are still on 
0.12.31
How can I "refresh" garden or delete it's temporary files? is it under my home profile? .garden?
I am gonna try uninstall/install
Wow that -l5 spits a lot of logs 😄
k
It’s called silly log level for a reason 😂
Fwiw, 0.12.31 is an extremely old version, it’s quite possible that your issues have been addressed since the
a
I am not sure I can upgrade, we are forced to use specific versions that have been tested and vetted by our company
Ok, I tried to install the latest version manually, it is not backward compatible with our configuration...
@bright-policeman-43626 What is a good way to send you the logs? Can I pm them?
b
Yes sir, you can PM them to me and @astonishing-tomato-18259 so we can take a look into this with a greater detail!
a
Thank you! will do that shortly
b
Hey @aloof-lamp-69262 we analyzed your silly logs and we couldn't find OOM logs on them, we noticed that there is a service that is 
crashloopbackoff
whic might be the cause of your failure. Since you have 80 services it might be hard to detect it but you could try to open another terminal and grep for that string! 
kubectl get pods -A |grep Crash
a
Got it, ok, I will try to troubleshoot with this. So this crash might be exhausting the memory?
b
Yes, or Garden is cutting the operation after the default timeout after waiting for all your services to stabilize
35 Views
PreviousNext
Powered by