Kubernetes example failing

I’m looking at using garden with our microservices project and was having some troubles with the tutorial. I already have a full k8s setup running in Google Cloud and all the microservices have a kubernetes yaml file so it would seem that the kubernetes module type is exactly what I’d want to use. So I went and created a project file with garden create project and then went into the sub-folder of one of the microservices and ran garden create module and selected the kubernetes option. I pointed it at my existing yaml file and then ran garden dev or garden deploy and both just returned that there was no modules to run, and it aborted.

I assumed I must be missing something in my setup so I grabbed the example project (garden/examples/kubernetes-module at master · garden-io/garden · GitHub) and gave that a whirl. This time it found the modules but immediately failed with:

√ providers                 → Getting status... → Cached
   i Run with --force-refresh to force a refresh of provider statuses.
√ redis                     → Syncing module sources (1 file)... → Done (took 0.3 sec)

Failed getting status for service 'postgres' (from module 'postgres'). Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
StatusCodeError from Kubernetes API - 404 - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server could not find the requested
resource","reason":"NotFound","details":{},"code":404}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Failed getting status for service 'redis' (from module 'redis'). Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
StatusCodeError from Kubernetes API - 404 - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server could not find the requested
resource","reason":"NotFound","details":{},"code":404}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
√ redis                     → Building version v-b5c2ab6a0a... → Done (took 0 sec)
√ postgres                  → Building version v-a3aca15fc7... → Done (took 0 sec)

 Garden dashboard running at http://localhost:9777

 Waiting for code changes...

So I can’t really figure out what’s wrong with my project if the examples don’t even work… What am I missing?

1 Like

Hi @rfuhrer!

We’re very happy to have the first question on our new community forum :tada: (And apologies for the late reply, we’re still ramping up after the holidays).

Regarding the example project, I suspect the issue is that it’s configured for local Kubernetes (i.e. the local-kubernetes provider). The docs for the example could be clearer on this.

You should be able to fix this by setting the provider to just kubernetes and providing your context. See for example this demo project for reference.

You’ll find more information on configuring remote Kubernetes in our docs.

As for the original issue, would you mind sharing an excerpt of your configuration? That should help us get to the bottom of this.

I’m trying to run the projects locally (Docker for Windows installed on WSL2 with Kubernetes option enabled) so wouldn’t local-kubernetes be what I want? I wouldn’t know what the “context” of my local k8s would be if you do need kubernetes as the provider. The way I read the example/demo projects is that I should be able to just pull the repo and run garden dev if I have k8s installed. I have no problem running other demo projects if they use containers. Just kubernetes style ones seem to fail.

For my actual project I have the following setup:
Project Folder: project.garden.yml

kind: Project
name: alberta
environments:
  - name: default
providers:
  - name: local-kubernetes

Microservice in a sub-folder /dataservice/garden.yml:

kind: Module
type: kubernetes
name: dataservice
files: [./manifests/deployment.yml]

that deployment.yml:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: alberta
  labels:
    app: dataservice-v6
  name: dataservice-v6-deployment
...

I think this looks like a bug on our side actually… The error message indicates that an error is thrown during the status check for the resources, but a 404 is of course to be expected from the k8s API when the resource hasn’t been deployed. And your configs look just fine at a glance.

I’m gonna quickly give this a spin, see if I can reproduce. If not, I’ll first be a bit puzzled tbh, and then follow up again to see about working this out.

I can reproduce this, so this appears to be a regression. I’m on it.

1 Like

I found three problems:

  1. The kubernetes-modules example doesn’t work with Kubernetes 1.19 because the manifests use apps/v1beta2 for the StatefulSets. So I’ll update that.
  2. The error handling for when a resource type can’t be found was busted, and gave this misleading message. I’ll post a quick PR to fix that.
  3. It appears the manifests in the kubernetes-modules example don’t work even with the updated apiVersion, so I figure we just need to update all those manifests. It anyway makes more sense to have simpler examples, as opposed to using postgres and redis.

Now, that doesn’t explain why your own project fails. Can you share the error message that comes up with your own project?

It just seems like the scan for garden.yml’s fails or something. This is all I get:

Good afternoon! Let's get your environment wired up...

✔ providers                 → Getting status... → Done
No enabled modules found in project.
Aborting...


 Garden dashboard running at http://localhost:9777

roryf MINGW64 /c/git/Alberta (master)
$

I was just trying to get one microservice working before jumping on the others, so maybe the issue is that there’s 10 other folders in git/Alberta that don’t have garden.ymls?

That is odd… This indicates the garden.yml for the module can’t be found during the scan, but I wonder why that might be. Does it work if you hoist it up to the same directory as the project config and name it something like dataservice.garden.yml? (and change the path to the manifest file along the way)

That got it working, but it seems like there’s an issue with sub-folders in the k8s files listing:

dataservice.garden.yml (right beside project.garden.yml now)

kind: Module
type: kubernetes
name: dataservice
files: [./dataservice/manifests/deployment.yml]

Console output:

Good afternoon! Let's get your environment wired up...

✔ providers                 → Getting status... → Cached
   ℹ Run with --force-refresh to force a refresh of provider statuses.
✖ dataservice               → Building version v-047e9fab98...

Failed building dataservice. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ENOENT: no such file or directory, open 'C:\git\Alberta\.garden\build\dataservice\dataservice\manifests\deployment.yml'
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

 Garden dashboard running at http://localhost:9777

 Waiting for code changes...

dataservice\dataservice seems pretty suspect

Interesting mystery here… The paths actually look alright (the first dataservice in dataservice\dataservice\... is the module name) but it looks like Garden just can’t see or scan through the dataservice directory.

Any chance there are some permission issues at play? Or symlinks/links perhaps? It might well still be a problem in Garden, to be clear, but I’m trying to narrow it down.

I also notice roryf MINGW64 /c/git/Alberta (master) in your shell. Can you elaborate on your setup a bit? We’ve tested ourselves on PowerShell and the new Windows Terminal, but I wouldn’t rule out some compatibility issue if it’s a setup we haven’t actively tested on.

I was just running git bash inside a VSCode terminal window. I get the same result in PowerShell as well. Potentially this is something with our deployment.yml? This project was setup to be directly deployed to GKE awhile back and I’m not sure how friendly it is with local k8s (hence the interest in garden.io :slight_smile: ). The only thing I did was just hardcode all the secrets to be some dummy dev values since I didn’t want to have to setup 20 secrets just to do some research on garden. I’ll see if I can get a more hello-world style application running. Still not sure why the demo project finds the sub-folder ymls no problem, but mine doesn’t work. That part is odd. I kind of expected to hit some hurdle like where I’m at now right off the bat due to our k8s config.

Yeah I’m still a little perplexed by this. Perhaps narrowing down a bit further: Does Garden find the module if you have the garden.yml in the dataservice folder, but change it into a type: container module?

Another thing to try is to create a fresh directory and put just the garden.yml and the deployment.yaml in there, see if that works.

It does certainly look like a file scanning issue, somehow. If it were a problem with the actual K8s manifest it’d, well, manifest differently :grimacing:

Hmm… I swear this used to work, but maybe I’m misremembering…

Cleaned up the k8s module and did this:

PS C:\git\Alberta\dataservice> garden create module
Create new module

? Select a module type: container
? Set the module name: dataservice

-> Created new module config in garden.yml

i We recommend reviewing the generated config, uncommenting fields that you'd like to configure, and cleaning up any
commented fields that you don't need to use.

For more information about container modules, please check out https://docs.garden.io/reference/module-types/container,
and the container provider docs at https://docs.garden.io/reference/providers/container. For general information about
Garden configuration files, take a look at https://docs.garden.io/using-garden/configuration-overview.

resulting garden.yml

kind: Module
type: container
name: dataservice
dockerfile: Dockerfile

result:

No enabled modules found in project.

renaming to dataservice.garden.yml didn’t change anything. Moving it up a directory beside the project.garden.yml and then updating the path dockerfile: dataservice/Dockerfile gets it running, but then outputs this:

Good afternoon! Let's get your environment wired up...

√ providers                 → Getting status... → Cached
   i Run with --force-refresh to force a refresh of provider statuses.
× dataservice               → Building dataservice:v-4ac18c6a2d...

Failed building dataservice. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Unable to run docker command: Command "C:\Users\roryf\.garden\tools\docker\4db49a8e19f00055\docker\docker.exe build -t
dataservice:v-4ac18c6a2d --build-arg GARDEN_MODULE_VERSION=v-4ac18c6a2d --file
C:\git\Alberta\.garden\build\dataservice\dataservice\Dockerfile C:\git\Alberta\.garden\build\dataservice" failed with
code 1:

unable to prepare context: unable to evaluate symlinks in Dockerfile path: CreateFile
C:\git\Alberta\.garden\build\dataservice\dataservice: The system cannot find the file specified.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Ah, so the dataservice directory is symlinked? That could indeed explain the matter. Using symlinks like that does work on *nix systems but it’s possible that it doesn’t on Windows. At least we’ve never tried it, and I do know there are some different semantics at play there.

Perhaps try this one and see how that shakes out:

Another thing to try is to create a fresh directory and put just the garden.yml and the deployment.yaml in there, see if that works.

If that works we at least know the cause, and can evaluate what best to do from there.

I don’t think I’m using symlinks. I had an Alberta folder that had a bunch of git repos inside it (we haven’t switched to a monorepo style yet) and I created all those folders just by running git clone. Garden doesn’t work if there’s no git repo so I couldn’t run in git/Alberta until I ran git init, but that’s all I did to make this folder any different than a normal windows folder.

I created a fresh garden-test folder in c:\git and copied the project yml, made a subfolder called dataservice and copied dataservice.garden.yml over inside that. It does seem to scan that directory properly, but I get the same

× dataservice               → Building dataservice:v-c09a885a00...

Failed building dataservice. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Unable to run docker command: Command "C:\Users\roryf\.garden\tools\docker\4db49a8e19f00055\docker\docker.exe build -t
dataservice:v-c09a885a00 --build-arg GARDEN_MODULE_VERSION=v-c09a885a00 --file
C:\git\garden-test\.garden\build\dataservice\Dockerfile C:\git\garden-test\.garden\build\dataservice" failed with code
1:

unable to prepare context: unable to evaluate symlinks in Dockerfile path: CreateFile
C:\git\garden-test\.garden\build\dataservice\Dockerfile: The system cannot find the file specified.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Is that symlinks maybe a red herring? Seems like it’s just saying it can’t find this dockerfile (which doesn’t exist at all in the new folder)

Aah, I did not think to ask if those were git directories. The symlink thing was indeed a red herring.

So, Garden uses git to scan for files, and a git scan will not find files in nested git directories. However, Garden will recurse into git submodules, so what you need to do there is make the dataservice (and the others you want to add) repo a git submodule in the project root git repo you initialized.

There is an alternative, which is to use remote sources. There are pros and cons to consider, between using submodules, project sources and module sources. I think I’d suggest choosing between what you started on (i.e. using submodules), and using module sources.

If you go with the latter, you basically have all your Garden configs in a repo separate to your other repos and all the source code, and point to the individual repos using the repositoryUrl field on each module config.

It all depends on how you prefer to work and how you want to use Garden. I’m a monorepo fan myself (especially with Garden, which tends to make that approach much easier), but I quite understand that it’s a matter of taste :slight_smile:

Do let me know if I can help out with that. But at least to start, you can make your git cloned repos submodules in the project repo and your modules should work nicely.

Ahh ok that makes sense. Sorry, I assumed Garden was using a file scan and not a git scan for the yaml files. I thought it was insane that such a core feature as “scanning for modules” was broken without anyone noticing :slight_smile: Maybe a quick note in the docs about this being how it all works will prevent someone in the future from having the same issue as me and wasting all your time :laughing:

Copied everything into a clean folder and ran git init then garden dev. Now we’re getting somewhere!

Good evening! Let's get your environment wired up...

√ providers                 → Getting status... → Cached
   i Run with --force-refresh to force a refresh of provider statuses.
√ dataservice               → Syncing module sources (1 file)... → Done (took 0.4 sec)
√ dataservice               → Building version v-f286959a97... → Done (took 0 sec)
| dataservice               → Deploying version v-f286959a97...
   ‼ dataservice               → Deployment/etsdataservice-v6-deployment: Error: InvalidImageName

This is what kind of errors I was expecting to hit. Thanks for all the help!

1 Like

Ah I’m glad we worked this out, I was getting worried we had some nasty bug in some specific context or something.

Maybe a quick note in the docs about this being how it all works will prevent someone in the future from having the same issue as me and wasting all your time :laughing:

Very good point! I’ll see about finding a good spot to mention this. I’m actually more worried about users wasting their own time as opposed to mine (especially if it’s multiple people, and that could well be the case), but the point stands either way :stuck_out_tongue:

Anyway, glad to be of help. By all means post a new thread if something else comes along the way, feedback, questions, issues etc., it’s all ultimately useful for us.

1 Like