https://garden.io logo
#🌱|help-and-getting-started
Database migration tasks/container
# 🌱|help-and-getting-started
a

aloof-lamp-69262

02/15/2023, 4:14 PM
Hello, TLDR: Using a "task" for a database migration container is not a good option as there is no automated way to run the task without having something depend on it explicitly. Using a container dies shortly after the command runs correctly, but garden complains because it dies quickly. I have a service (container) that deploys database X, let's call it "database" I have another service (container) that does the database migration, let's call it "migration" "migration" depends on "database" naturally. If I use task for the migration, I have to run it manually which is not desired, I need this to run always when deployed. If I use it as a container, it works well, but the container dies too fast and garden complains about it, with the "BackOff - Back-off restarting failed container" error. But the container did what it is supposed to do. Is there an option to tell garden not to complain if the container stops too early? Or is there an option to get the task to run automatically? We also had many other use cases where we wanted the task to run automatically, I think this would be a desired feature. Thank you.
b

bright-policeman-43626

02/15/2023, 4:40 PM
Hello @aloof-lamp-69262 thanks a lot for posting this. Just one question, are you adding dependants to your tasks by any chance? According to the docs
Copy code
"Tasks that have dependents (i.e. something that depends on them) are run automatically by Garden. For example, if a service depends on a task being run before it is deployed, then Garden will automatically run that task before deploying the service. Other tasks will need to be run manually."
So maybe following this example might help? https://docs.garden.io/using-garden/tasks#database-migration
a

aloof-lamp-69262

02/15/2023, 4:44 PM
No, in this case, we don't have anything that can depend on this task really, it is just something that runs after the database starts. Alternatively if a task can have an option to run "after" a parent, and not "before" that might help too, but I imagine it might become confusing. To make the parent depend on it, but it runs after the parent...
I did read that part about the example that you posted yes. But it doesn't solve our use case, the main reason being we want the task to start automatically
b

bright-policeman-43626

02/15/2023, 4:55 PM
Oh, I see what you mean now. A little 🐔 and the 🥚 issue. Is this
migration
service a Node application that runs
npm migrate:up
or something like that? Or it's just a set of scripts that you are running after the database is already up and running?
a

aloof-lamp-69262

02/15/2023, 4:56 PM
It's a custom set of scripts yes. not node specific
And yes, it's a good way to put it 🙂 🐔 and 🥚
I am thinking usually database migration scripts would go always after the db engine starts, so perhaps this is the prevailing use case?
Btw, Garden.io is really great. We love it ❤️
We have an ugly hack, we are letting the container that does the migration sleep for around 2 minutes before it dies off quickly... This is keeping garden happy and not reporting that 1 task has failed. Just FYI
b

bright-policeman-43626

02/15/2023, 5:08 PM
Oh I'm glad you guys are enjoying Garden 🌸 . That workaround it's good but definitely not ideal as makes you waste 2 precious minutes everytime you deploy it. Let me try to experiment a little bit with a custom postgres container in my local trying to inject some scripts and see if I can reproduce your scenario.
a

aloof-lamp-69262

02/15/2023, 5:18 PM
Thanks! appreciate it.
yeah, just a small script that does anything should reproduce that. Perhaps "echo" or similar
Btw, you guys should add the garden.io icon to the build kite emojis 🙂 https://github.com/buildkite/emojis#contributing-new-emoji
b

bright-policeman-43626

02/15/2023, 5:24 PM
This is actually a neat idea, I love emojis @quaint-dress-831
This is really interesting, I actually got hit by the same problem you are having @aloof-lamp-69262 I'm uploading a tiny repository to submit a feature request to Garden.
a

aloof-lamp-69262

02/15/2023, 6:32 PM
Cool 🙂
So just to be clear, how are you gonna address this? or not decided yet
b

bright-policeman-43626

02/15/2023, 6:33 PM
The first try works fine but subsequent just fails, what's happening is that Kubernetes obviously tries to keep the pod alive after finishing the task, because these are not init containers or jobs that are supposed to finish either successfully or not.
a

aloof-lamp-69262

02/15/2023, 6:33 PM
Right
so perhaps tag it with some property to be a job?
b

bright-policeman-43626

02/15/2023, 6:34 PM
Basically going to scalate this one with the OSS Developers to see if they have any other recommendation about what could we use to run those migrations automatically. And also create a Github Issue to document the need of a Garden Job or some sort of module like that.
a

aloof-lamp-69262

02/15/2023, 6:34 PM
Nice, can you link the issue on github so I can follow that please, when it's created.
b

bright-policeman-43626

02/15/2023, 6:35 PM
Yeah, for now the only solution that I can think of is creating a tiny helm-chart that has the postgresql database and an init container that runs the migration everytime there's a deploy
a

aloof-lamp-69262

02/15/2023, 6:35 PM
you mean instead of using "container" as module type, use "helm-chart"?
b

bright-policeman-43626

02/15/2023, 6:36 PM
Correct, but I believe it might be too much work to do the helm chart for that. Let's see if @quaint-dress-831 / @swift-garage-61180 could help us debug this one!
a

aloof-lamp-69262

02/15/2023, 6:38 PM
Cool.
b

bright-policeman-43626

02/15/2023, 7:32 PM
Sent you a couple of examples by DM so you can take a look, I'll be submitting this as an issue with the dependencies objects, as the setup task should run successfully and automatically if the dependencies are well set between services/tasks.
a

aloof-lamp-69262

02/15/2023, 10:56 PM
Thank you
b

bright-policeman-43626

02/16/2023, 6:38 AM
@quaint-dress-831 documented this issue in this repository: https://github.com/ShankyJS/garden-migrations-postgres Hopefully you can take a look as I tried two different options to provide a good workaround for this issue but wasn't able to find a good solution. (Feel free to run both examples, each of them have a tiny README to guide through the thought process I followed to implement them).
q

quaint-dress-831

02/16/2023, 7:35 AM
We could definitely open this as an issue. We also would benefit from badges so people can show they're building with Garden 💪
Ok so I've run into this issue too as will almost anyone trying to do these kinds of early init database tasks using the
container
modules. Fundamentally, the container module stands in an awkward position of really easy to get going with (and that's why it has so many fans) but not really full-powered enough for anything but fairly rudimentary deployments.
Garden more or less expects that as people mature inside the tooling that the
container
module relegates to an image-builder for the
kubernetes
or
helm
modules.
Using either of those "unlocks" the ability to have the service depend on its own tasks.
You can see what I mean here in the first two modules of my module "stack": https://github.com/worldofgeese/flask-cockroachdb-devcontainer/blob/gardenify/modules.garden.yml
We should support Jobs but we don't. The closest open issue we have for that request is at https://github.com/garden-io/garden/issues/2713
Thank you @bright-policeman-43626 for creating these examples. I think they excellently illustrate where Garden could improve both in messaging and direction.
b

bright-policeman-43626

02/16/2023, 7:46 AM
If we could support jobs in the container modules the approach I documented in my repository (using-modules folder) would work really easily, definitely a nice to have for future versions. I think @aloof-lamp-69262 was able to solve the issues by adding a sleep to prevent the container from dying, but definitely a better solution would be leveraging a helm/kubernetes container instead
q

quaint-dress-831

02/16/2023, 7:48 AM
Once the core team ships our next-gen release, I think it's absolutely something worth addressing providing it's still an issue in 0.13
It's a full re-architecture and changes the pattern. I'm not as familiar with it as I'd like to be but @swift-garage-61180 could illuminate if it will still be an issue.
a

aloof-lamp-69262

02/16/2023, 5:57 PM
Thanks for that, for now the 2 minute sleep is preventing the ill effects, but it's good to know that other options such as kubernetes or helm exists, in case we want to try them out as an alternative.
18 Views