https://garden.io logo
#💻|dev
0.13: [Bug]: Deploy cached even though t...
# 💻|dev
b

big-spring-14945

07/12/2023, 1:56 PM
@Anna and me discovered a pretty critical regression in 0.13.7 (was working in 0.13.6): kubernetes actions (and modules) caching does not work properly. If the manifest changes, garden still considers the deploy unchanged and does not correctly deploy the latest version: https://github.com/garden-io/garden/issues/4811 This came to our attention because the garden-platform pipeline stopped updating our internal garden cloud instances
For some weird reason it's one of those things that only happen on the third try. That's probably why it hasn't been catched by the test suites.
Latest discoveries: In step three, it is not important that the manifest changes due to the variable parameter. It also reproduces this bug when changing the value of the postgres-password variable in garden.yml. It even reproduces when simply changing the manifest directly in postgres/garden.yml
@User We are on Discord in #851343720267776000 if you need help understanding this issuee
I would even seriously consider rolling back this change (if it's it) and release 0.13.8 because it's pretty critical. Wdyt @curved-intern-91221 @alert-helicopter-61082 @brief-restaurant-63679 @chilly-gigabyte-83853
b

brief-restaurant-63679

07/12/2023, 2:11 PM
Does the same happen if you use a container action type and change the actual source code, as opposed to Garden config?
Or rather, what if it's a Kubernetes action and you change the manifest, not the config?
b

big-spring-14945

07/12/2023, 2:17 PM
It's happening when only changing the manifest
I did not try it yet with a kubernetes deploy and manifest files. Current repro uses inline manifest
b

brief-restaurant-63679

07/12/2023, 2:24 PM
Ok ok. My first guess is that this is limited to changes to garden.yml config and not other files. Not 100% sure though.
b

big-spring-14945

07/12/2023, 2:33 PM
Trying that now
@brief-restaurant-63679 nope, same result with kubernetes deploy action and manifest files that changed (without garden config changes)
conclusion from the call with @curved-intern-91221 (thanks for jumping in so quickly): Let's decide tomorrow wether or not to include the revert in tomorrow's release of 0.13.8
b

brief-restaurant-63679

07/12/2023, 2:45 PM
Ok ok. Does the revert fix the issue?
b

big-spring-14945

07/12/2023, 2:45 PM
yes, verified that
f

freezing-pharmacist-34446

07/12/2023, 3:03 PM
Just a note for later, but it would be good to do a post-mortem on this one.
c

calm-family-29323

07/13/2023, 7:57 AM
Helm might have the same bug as the logic for both is very similar and partially shared
s

swift-garage-61180

07/13/2023, 11:02 AM
For one thing, it looks like we need more tests around the versioning semantics here.
b

big-spring-14945

07/13/2023, 1:02 PM
Yeah, for some weird reason I remember now the issue was reversed for helm: It always deployed the helm chart, even if it should have been cached
f

freezing-pharmacist-34446

07/14/2023, 1:47 PM
I started a post-mortem in notion. Please add to the timeline and the other points. https://www.notion.so/gardendev/Garden-0-13-7-released-with-major-regressions-around-caching-8578d765fdb24d219577d7d402a15b93 I would love us to take 30 min on Monday to go through the doc and discuss findings.