https://garden.io logo
Powered by
#💻｜dev
integ test ci imagepull err thread
c

calm-family-29323

04/05/2023, 10:44 AM
I got a silly level log of the incident here: There are a few things that are of interest to me
Although the status resolves as ready it still builds the build a few lines down the line:
Copy code
ℹ build.missing-sh [debug]  → Image missing-sh:v-bda89cfaa0 already exists
ℹ build.missing-sh          → Build type=container name=missing-sh (from module missing-sh) already complete, nothing to do.
...
ℹ build.missing-sh [silly]  → Processing node build.missing-sh:process
ℹ build.missing-sh [silly]  → Executing node build.missing-sh:process
ℹ build.missing-sh [silly]  → Completing node build.missing-sh:process. aborted=false, error=null
@alert-helicopter-61082 any ideas?
a

alert-helicopter-61082

04/05/2023, 1:52 PM
Those log lines don't mean that the build is happening, just that the "process" node in the graph is being executed, which will promptly return if the status was previously resolved as ready. If an actual build was happening, you'd see a lot more logs to that effect.
The actual issue does seem to be simply that the image isn't available to the cluster, despite the status check indicating that it is. So the first thing I'd like to know is whether the image is there after the test fails.
c

calm-family-29323

04/05/2023, 2:44 PM
I made some progress > I'd like to know is whether the image is there after the test fails It is not. Re-running the same test again creates the image. The log from the ci has the line 
Image missing-sh:v-bda89cfaa0 already exists
which comes from here [1]. So the 
containerHelpers.imageExistsLocally
that's called above returns an identifier, but when I run 
docker images missing-sh:v-bda89cfaa0 -q
manually after the first test run it returns nothing. The 
containerHelpers.imageExistsLocally
call should this line [2], but if you look at the logs from earlier it's nowhere to be found, so something goes wrong along the way there. It is logged on a second attempt and also when I run locally. I think this is where the problem lies. 1. https://github.com/garden-io/garden/blob/0.13-integ-imagepull/core/src/plugins/container/build.ts#L30 2. https://github.com/garden-io/garden/blob/0.13-integ-imagepull/core/src/util/ext-tools.ts#L195
a

alert-helicopter-61082

04/05/2023, 4:15 PM
Ok nice. That must be the issue then. Think you can debug that further?
c

calm-family-29323

04/06/2023, 2:26 AM
I have high hopes I can figure this out today
Figured it out: https://github.com/garden-io/garden/pull/4048 It was a mock leaking out of it's scope (written by me \: ) Here's a green ci [1] But after rebasing I'm getting timeouts again [2] (I don't yet know what they are as it's taking too long and I want to go to sleep, but it shouldn't bee too hard) 1. https://app.circleci.com/pipelines/github/garden-io/garden/16394/workflows/96276cbc-cf7a-4a9b-82c5-d354ea23ecdb 2. https://app.circleci.com/pipelines/github/garden-io/garden/16395/workflows/2ca90aa4-4253-4f7a-b87e-5800b0080b01
Figured out the timeouts as well. They are caused by tests where an action is expected to fail and 
throwOnError
is set to 
true
for the task processing [1]. Later on when the action is resolved [2] the solver gets stuck in an infinite loop. Can be reproduced easily with tests like this [3] @alert-helicopter-61082 can you have a look at this? If this is fixed we'd be really close to a green integ suite cc @swift-garage-61180 1. https://github.com/garden-io/garden/blob/0.13/core/test/integ/src/plugins/kubernetes/container/container.ts#L165 2. https://github.com/garden-io/garden/blob/0.13/core/test/integ/src/plugins/kubernetes/container/container.ts#L171 3. https://github.com/garden-io/garden/blob/0.13/core/test/integ/src/plugins/kubernetes/container/container.ts#L150
@alert-helicopter-61082 ping. I'd love to get the integ tests passing but I can't quite figure out the solver thing ^
a

alert-helicopter-61082

04/12/2023, 3:30 PM
I may be able to look at this tomorrow
But that's as much of a commitment as I can give, unfortunately 😬
c

calm-family-29323

05/02/2023, 1:43 PM
@alert-helicopter-61082 if you have time could you take a look. Then I could take a look at the integ tests
a

alert-helicopter-61082

05/02/2023, 1:49 PM
Not sure when I will tbh. The broken test suites now seem to just be kind not working... That looks to be completely separate.
c

calm-family-29323

05/02/2023, 2:28 PM
Could we have an introduction session on the solver, then I would feel more comfortable digging in on my own
a

alert-helicopter-61082

05/02/2023, 5:45 PM
We can do that, not sure it has anything to do with the solver directly though
It does, turns out
or... hmm. we'll see, this is tricky
Yeah, solver it was. I worked it out.
More issues remain, but the solver issue seems to be solved
That testdouble issue (
td.reset()
not working) seems to be a lot of what's wrong in CI. So that's fun.
Very puzzling that this only happens in CI... Trying a simple version upgrade on the lib to start.
Powered by