Graph takes 30+ seconds to run

Question

Hey all Trying to understand what is happening under the hood if we can do something to improve the situation We have about 240~ garden modules the actual number of modules is more around a 100 but due to different templates we end up with that number of modules needing to be resolved by Garden on startup We noticed the graph is taking very long around 30+ seconds ``` 2023 06 26T17 24 42 519Z Resolving 240 modules 2023 06 26T17 25 14 026Z Resolving 240 modules Done ``` This is using the Kubernetes provider I ve look at the logs with debug logs on and noticed it s doing some FS scans which takes about a second I assume building the DAG is also very quick I wonder if most of this time is spent making Kubernetes API calls Are those necessary or is there an option here to introduce a flag to skip all of these Is there anything else I can do to improve this situation It would be fantastic if this was under 5 seconds Thanks

quaint-dress-831 · Answer

Hi < future coat 9017> I m escalating this internally

quaint-dress-831 · Answer

Our CEO writes > This is basically a known issue especially in 0 12 x We are doing several things to improve this and some of them are already in 0 13 x One quite important thing to note is that using ` gardenignore` files is broadly more efficient than using `include exclude` in module configs For repos with many files and where possible to use this is strongly encouraged

quaint-dress-831 · Answer

We have a PR open that s likely to help once merged and released https github com garden io garden pull 4642

future-coat-9017 · Answer

< quaint dress 831> Thats awesome do you think we can also backport the fix to 0 12 as well We cannot upgrade to 0 13 yet due to the breaking changes but being able to introduce this improvement with our current setup will be fantastic

quaint-dress-831 · Answer

< brief restaurant 63679> I ll keep you updated We do recommend users migrate to Bonsai eventually

quaint-dress-831 · Answer

But I can promise to advocate for it if it s technically feasible It may be that these improvements are technically locked to Bonsai

quaint-dress-831 · Answer

< future coat 9017> just to note we ve heard graph resolution is up to 6x faster just from switching to Bonsai This is due to changes to Bonsai we can t backport

future-coat-9017 · Answer

Thanks I also see the PR merged so will definitely look into moving to Bonsai

future-coat-9017 · Answer

< quaint dress 831> just tried out Bonsai with `GARDEN GIT SCAN MODE=repo` unfortunately im seeing almost 2x slower performance

calm-family-29323 · Answer

< future coat 9017> Slow resolution times are mostly due to unfavourable file inclusion settings Probably tinkering with your ` gardenignore` files and `include` and `exclude` filters on the modules will increase the speed greatly

brief-dawn-88958 · Answer

< future coat 9017> excludes are not properly working with the new git scan mode see https github com garden io garden issues 4783 Workaround is to use gardenignore files instead

polite-fountain-28010 · Answer

Hey < future coat 9017> we just released a new version that once again has some significant performance improvements when it comes to graph resolution Maybe you could give it a try and report back if that made things faster again for you

future-coat-9017 · Answer

Hey there Compared running `garden validate` with 0 12 48 vs 0 13 13 and the results are about a 5 second improvement from 31s in 0 12 to 25s in 0 13 13 We were hoping to reduce the time to under 10s wondering if we should see a greater improvement than above We ran the command with `GARDEN GIT SCAN MODE=repo` as well

future-coat-9017 · Answer

btw I ve also removed all exclusions and it seems to improved by the same amount ~5 seconds

future-coat-9017 · Answer

my assumption is that most time is simply due to the large number of modules its trying to resolve in our case 260~ not specifically with git scanning it seems include exclude bonsai has minimal effect in our base around ~5 second improvement

polite-fountain-28010 · Answer

`GARDEN GIT SCAN MODE=repo` should be the fastest repo scan mode now for large repos so I d keep using that What OS is this on If you re on MacOS did you make sure you have the ARM build running Also could you share the debug logs I would be interesting to see where it s actually spending its time Previously the graph resolution time was the most expensive part with large projects so we focused on that first

future-coat-9017 · Answer

Good idea making sure I was using ARM reduce time to 22 seconds which is slightly better let me get the logs masked and post it

future-coat-9017 · Answer

See attached https cdn discordapp com attachments 1122951215270072452 1150845237799952394 logs jsonl

future-coat-9017 · Answer

< polite fountain 28010>