0.13: [Bug]: Deploy cached even though t...
# 💻｜contributingb
big-spring-14945
07/12/2023, 1:56 PM@Anna and me discovered a pretty critical regression in 0.13.7 (was working in 0.13.6): kubernetes actions (and modules) caching does not work properly. If the manifest changes, garden still considers the deploy unchanged and does not correctly deploy the latest version: https://github.com/garden-io/garden/issues/4811
This came to our attention because the garden-platform pipeline stopped updating our internal garden cloud instances
big-spring-14945
07/12/2023, 1:57 PMFor some weird reason it's one of those things that only happen on the third try. That's probably why it hasn't been catched by the test suites.
big-spring-14945
07/12/2023, 2:01 PMLatest discoveries:
In step three, it is not important that the manifest changes due to the variable parameter. It also reproduces this bug when changing the value of the postgres-password variable in garden.yml.
It even reproduces when simply changing the manifest directly in postgres/garden.yml
big-spring-14945
07/12/2023, 2:02 PM@User We are on Discord in #851343720267776000 if you need help understanding this issuee
big-spring-14945
07/12/2023, 2:09 PMI would even seriously consider rolling back this change (if it's it) and release 0.13.8 because it's pretty critical. Wdyt @curved-intern-91221 @alert-helicopter-61082 @brief-restaurant-63679 @chilly-gigabyte-83853
b
brief-restaurant-63679
07/12/2023, 2:11 PMDoes the same happen if you use a container action type and change the actual source code, as opposed to Garden config?
brief-restaurant-63679
07/12/2023, 2:12 PMOr rather, what if it's a Kubernetes action and you change the manifest, not the config?
b
big-spring-14945
07/12/2023, 2:17 PMIt's happening when only changing the manifest
big-spring-14945
07/12/2023, 2:18 PMI did not try it yet with a kubernetes deploy and manifest files. Current repro uses inline manifest
b
brief-restaurant-63679
07/12/2023, 2:24 PMOk ok. My first guess is that this is limited to changes to garden.yml config and not other files. Not 100% sure though.
b
big-spring-14945
07/12/2023, 2:33 PMTrying that now
big-spring-14945
07/12/2023, 2:41 PM@brief-restaurant-63679 nope, same result with kubernetes deploy action and manifest files that changed (without garden config changes)
big-spring-14945
07/12/2023, 2:44 PMconclusion from the call with @curved-intern-91221 (thanks for jumping in so quickly): Let's decide tomorrow wether or not to include the revert in tomorrow's release of 0.13.8
b
brief-restaurant-63679
07/12/2023, 2:45 PMOk ok. Does the revert fix the issue?
b
big-spring-14945
07/12/2023, 2:45 PMyes, verified that
f
freezing-pharmacist-34446
07/12/2023, 3:03 PMJust a note for later, but it would be good to do a post-mortem on this one.
c
calm-family-29323
07/13/2023, 7:57 AMHelm might have the same bug as the logic for both is very similar and partially shared
s
swift-garage-61180
07/13/2023, 11:02 AMFor one thing, it looks like we need more tests around the versioning semantics here.
b
big-spring-14945
07/13/2023, 1:02 PMYeah, for some weird reason I remember now the issue was reversed for helm: It always deployed the helm chart, even if it should have been cached
f
freezing-pharmacist-34446
07/14/2023, 1:47 PMI started a post-mortem in notion. Please add to the timeline and the other points. https://www.notion.so/gardendev/Garden-0-13-7-released-with-major-regressions-around-caching-8578d765fdb24d219577d7d402a15b93
I would love us to take 30 min on Monday to go through the doc and discuss findings.
8 Views