https://garden.io logo
Join Discord
Powered by
This time the Garden team needs your help: Are you...
# 🌱｜help-and-getting-started
f
Hey all, thanks for reading this. We are currently thinking about multi-repo workflows and how well Garden integrates with them. For this we'd love to know: 1. How often in a week do you need to write a feature or fix a bug across multiple repositories? 2. How does the process look? How complicated or painful it is? 3. How do you test that your changes do not impact other repos and work well together? 4.How do you release changes to multiple repos? 5. Does Garden help with any of the above or does it stand in your way? Feel free pick one or multiple questions, we are curious about your experiences!
o
We have a pretty monorepo focus, but we do have multiple repos that are connected. We have an "open source" core monorepo with most of the business code, then a repo for packaging that and combining with other components, and then we have a sort of staging/dog fooding instance that consumes the helm chart.
So we very often (weekly and hopefully more frequently going forward) need to propagate changes through this chain. It was painful at first, but I've set up good pre-release build/release workflow in each that runs Gitlab CI/CD to autogenerate prerelease tags and trigger downstream pipelines. For the downstream repos the pipelines may be triggered by upstream but they are responsible for fetching the source code via git. This is done with an exec Build action in garden. For testing we can't test the full integration of downstreams before committing to main branch and triggering downstream CI, but our upstream stable configuration always points to a stable upstream tag. While still being able to product 'latest' builds. So any commit to upstream main will run the test suites downstream and you can see issues at that point and correct them. We use garden in each pipeline individually to do the testing (and fetching "vendored" code as mentioned above). But it is self contained in each repo. The build/testing pipelines only interact through CI triggers and VCS tags.
b
Oh, I have a use case for this one with one of my clients. Basically they use Garden as a local-development tool and CI only. The development is fully microservice oriented, so repositories and logic is distributed among different repositories, we have a usage case where sometimes services have "dependencies", so let's say we have: - service_1_api - service_2_api - service_3_frontend If I want to develop service_1_api I consume service_2_api using the remoteSources from Garden, Garden manages this pretty gracefully, variables cascade from different projects to the one I'm developing which is nice. If I develop 
service_3_frontend
I need to consume both 
1_api and 2_api
Garden enables us to do this seamessly with remoteSources. Also we have a bunch of shared modules for databases, for example we have some modules for 
Redis
, 
MongoDB
, 
Postgres
that are shared across different projects and we use them only when needed. So far Garden has worked pretty nicely for this use case. Now answering the questions: - How often in a week do you need to write a feature or fix a bug across multiple repositories? Every day we do this kind of changes across multiple repositories, however microservices are managed by different teams so sometimes we fix the dependencies version to a version we know it works and pretty much maintain that for a long time until we need to bump it for local development purposes. - How does the process look? How complicated or painful it is? We are not using Garden for the release process itself as we are using other deployment mechanism that are a bit more robust. - How do you test that your changes do not impact other repos and work well together? Unit testing & E2E testing as Garden Tasks, we test the whole workflow in CI as well. - How do you release changes to multiple repos? PR's to different repos and get them merged. - Does Garden help with any of the above or does it stand in your way? It's definitely a valuable tool to manage the multi-dependencies process
f
Thank you both so much for answering! I am curious is there something that you feel like garden could do to facilitate these workflows even more?
o
One painful piece to write was tagging and moving containers between different registries. I had to write pods/etc. to run skopeo manually. It would have made it a lot more streamlined to just have that integrated to Garden. I.e. I do a build, test build, tag kind of workflow, plus other tagging related to releases. The current 
garden publish
did not work very well or at all for this purpose.
For the code vendoring steps the build isolation features can sometimes be challenging. I've written issues about this before.
Other than that I'll have to try out the remote source feature, although I don't have too much that could be shared at this time. Probably as teams grow its useful to have one team that builds CI components that they use. This is a whole other "rabbit hole" though to solve in general. Most of the duplication for me still is at the level of my CI runner (e.g. Gitlab CI/CD) and "task" runners (e.g. the Makefile or build system) rather than garden itself.
To flip this around a little bit though, one solution to poly-repo is to monorepo 🙂 for which I think there might be some improvements to garden. I'm still kind of formulating this but having "modules" for actions would be helpful since scaling action names and variable definitions/resolution across a very large repo would get unwieldy. For instance I have multiple garden spec files like 
project.garden.yaml
, 
services.garden.yaml
, 
db.garden.yaml
, 
app.garden.yaml
, 
tests.garden.yaml
, etc... These form natural "modules". Currently for defining constant valued variables (not CLI or varfiles) they either all go at the top-level 
project.garden.yaml
or in the individual action (or in a static varfile, which is very unintuitive and cannot interact with the top-level variables well then). In a lot of cases I have variables that are useful for a couple of actions but not all of them. So I either hardcode them in a few actions, which can make refactoring pretty painful and time consuming. E.g. you change the name of volume and need to propagate that across a few actions. Or I put them in the top-level variables which becomes super bloated (hundreds of lines long) and the names become super long (or worse ambiguous) since you basically have to write in your own fully-qualified module names e.g. 
test_data_storage_size
.
Without knowing exactly I would worry about the same thing with the remote sources which would add even more naming restrictions that aren't easily obvious without looking at the names of all the remote actions (or variables etc.). So basically a module system would probably be what I would want to see for scaling as the number of actions grow. Would also allow 3rd party packages. I've asked/written about the future of the templating system and a lot of templating systems already have such a module system AFAIU, e.g. jsonnet. That said I do enjoy having multiple files in the same "module" at a smaller scale so when I run 
garden test integration-test
I don't worry which file it is in like, 
garden test tests.integration-test
which you might have to add if actions were given modules.
2 Views
PreviousNext
Powered by