Is there a way to stream logs from Test Pods?

Hi!

First off thank you for building such a great tool! I feel enlightened after playing with it. I’m trying to see if we can replace our CI scripts with garden, and I noticed that when the test Pods fail I just get a vague undefined error. I saw there’s a garden logs command, but looks like that’s only for getting logs from services.

For example, I just have this really simple lint test:

tests:
  - name: lint
    args:
      - yarn
      - lint

When it fails I only see this output from garden:

✖ backend-test              → Running lint tests → Failed! (took 33.4 sec)
 
Failed running lint tests in module backend-test. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Error - undefined
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1 test task(s) failed!

But the logs from the Pod tells me it’s an OOM error. The Pod goes away quite quickly too, so it’s hard to manually see logs from there. Any help is appreciated!

Thanks,
Anna

Hi Anna and thanks for the kind words!

Currently you can’t stream logs from test pods using just the CLI (it’s only available in the Cloud/Enterprise version).

But Garden should tell you if it’s an OOM error so I wonder why you’re not seeing that. Would you mind sharing the relevant pod logs? We check for exit code 137 and whether the exit reason is OOMKilled, so I wonder what we’re missing.

And in the meantime, you can use a tool called stern to stream the logs. It’s easy to install.

The following command streams all logs from a given namespace:

stern -n <your-namespace> . --since 1m

With stern running, you can run the test command again and you should see a live stream of logs from the test pod once it’s up.

Sorry I lied, it wasn’t a 137 but the error went away after I increased the memory limit so I thought it’s OOM.

Here’s the actual logs from the pod:


yarn run v1.22.5
$ yarn lint:base ./src
$ eslint --quiet --ext .js,.ts ./src
<--- Last few GCs --->
[40:0x33d9f20[]    23292 ms: Mark-sweep 1024.7 (1035.3) -> 1024.0 (1029.1) MB, 392.4 / 0.0 ms  (+ 236.0 ms in 81 steps since start of marking, biggest step 63.0 ms, walltime since start of marking 797 ms) (average mu = 0.263, current mu = 0.227) allocation[40:0x33d9f20[]    24096 ms: Mark-sweep 1025.7 (1029.1) -> 1022.5 (1027.6) MB, 611.5 / 0.0 ms  (+ 175.8 ms in 28 steps since start of marking, biggest step 59.2 ms, walltime since start of marking 804 ms) (average mu = 0.149, current mu = 0.021) allocation
<--- JS stacktrace --->
==== JS stack trace =========================================
    0: ExitFrame [pc: 0x140dc19[]
Security context: 0x2e63dc6808d1 <JSObject>
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
    1: parsePrimaryExpression(aka parsePrimaryExpression) [0x2ad4b2779ab1[] [/retool_backend/node_modules/typescript/lib/typescript.js:~32850] [pc=0x1630fef56e8e](this=0x1468d68804b1 <undefined>)
    2: parseUpdateExpression(aka parseUpdateExpression) [0x2ad4b2779371[] [/retool_backend/node_modules/typescript/lib/typescript.js:~32266] [pc=0x1630fef779a8](this=0x146...
 1: 0xa1a640 node::Abort() [/usr/local/bin/node]
 2: 0xa1aa4c node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0xb9a62e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0xb9a9a9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0xd57c25  [/usr/local/bin/node]
 6: 0xd582b6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/usr/local/bin/node]
 7: 0xd64b75 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 8: 0xd65a25 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
 9: 0xd684dc v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
10: 0xd2eefb v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/bin/node]
11: 0x10714ce v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
12: 0x140dc19  [/usr/local/bin/node]
Aborted (core dumped)
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 134.
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

(btw I tried replying but looks like my post got hidden by the system)

Ok that would explain that then. Garden should show pod logs up until the failure though, we need to look into why that’s not happening.