https://garden.io logo
#🌱|help-and-getting-started
Request to pull specific tools with garden util
# 🌱|help-and-getting-started
s

sparse-easter-31125

02/14/2024, 6:03 AM
The current options for garden util fetch-tools is only an '--all' function which pulls every single tool, even ones which are not necessary. Can we have a feature for this that allows us to only pull a specific tool or set of tools?
s

swift-garage-61180

02/15/2024, 12:57 PM
Hi @sparse-easter-31125! Garden automatically fetches the tools needed by the plugins being used, so this is usually not an issue in practice. Is there a specific use-case you have in mind where this would help?
s

sparse-easter-31125

02/16/2024, 12:39 AM
Hi @swift-garage-61180, I'm trying to build a deployment container which does not have any garden projects in it but it will be used for the garden project when it is built through CI. The idea is just to be preemptive in case the links in which garden is referencing to pull those tools from goes down or becomes inaccessible due to changes in the future. During the container build process, it appears that the only way for me to do this is to fetch all the tools and then remove the ones which are not necessary for my project. If I were to fetch all the tools, it would bloat the image by a significant amount. Because of this, I'm wondering if it's possible to get specific tools rather than the whole toolbox.
m

mammoth-flag-56137

02/16/2024, 5:52 AM
Yeah we have solved this with work arounds involving multi-stage docker builds but the intention here is to pack everything we need including the tools gardens need into a docker image
we arent going to the extreme of completely locking off all internet access from our ci deploy jobs but if we wanted to we could, having the ability to prefetch only the specific tools we need makes building the docker image easier
fetching all tools blows the image out to more than a gig because of openjdk and pulumi
b

bright-policeman-43626

02/16/2024, 6:19 AM
@mammoth-flag-56137 I'm working on a Docker image that have both
kind
and
garden
so in CI I can create the kind cluster on the flight and deploy with
garden
did you got this error at any point of your journey here?
Copy code
Could not find rsync binary. Please make sure rsync (version 3.1.0 or later) is installed and on your PATH. More about garden installation and requirements can be found in our documentation at https://docs.garden.io/v/acorn-0.12/guides/installation
`
when doing the deployment?
Forget it I think I forgot the prerequisites (rsync and stuff)
m

mammoth-flag-56137

02/16/2024, 6:22 AM
this is what Ek made
Copy code
FROM alpine:3.18 as terraform
ARG TARGETARCH=amd64
ARG TERRAFORM_VER=""
RUN apk add --no-cache bash jq curl git openssh gcompat
WORKDIR /tmp
RUN curl -sLo terraform.zip "https://releases.hashicorp.com/terraform/${TERRAFORM_VER}/terraform_${TERRAFORM_VER}_linux_${TARGETARCH}.zip" && unzip terraform.zip && rm terraform.zip && mv ./terraform /usr/local/bin/terraform && terraform --version
WORKDIR /
RUN curl -o /usr/bin/gitlab-terraform https://gitlab.com/gitlab-org/terraform-images/-/blob/master/src/bin/gitlab-terraform.sh
RUN chmod +x /usr/bin/gitlab-terraform
ENTRYPOINT []

FROM terraform as aws-cli-v2
ARG AWSCLI_VER=""
RUN apk add --no-cache aws-cli=$AWSCLI_VER
RUN apk add --no-cache curl docker-cli git openssl rsync ca-certificates tar gzip openssh-client libstdc++ python3 py3-pip libc6-compat py3-openssl libffi gnupg groff py3-crcmod
RUN apk add --no-cache --virtual .pipeline-deps readline linux-pam && apk add bash sudo shadow && apk del .pipeline-deps
ENTRYPOINT []
WORKDIR /

FROM aws-cli-v2 as garden-fetch-tools
ENV USER=root
ENV HOME=/root
ENTRYPOINT []
RUN curl -sL https://get.garden.io/install.sh | bash
ENV PATH=$PATH:$HOME/.garden/bin
RUN garden util fetch-tools --all
RUN rm -rf /root/.garden/tools/openjdk*
RUN rm -rf /root/.garden/tools/pulumi*
RUN rm -rf /root/.garden/tools/otel-collector*
RUN rm -rf /root/.garden/tools/octant*
RUN rm -rf /root/.garden/tools/gradle*
RUN rm -rf /root/.garden/tools/conftest*
RUN rm -rf /root/.garden/tools/maven*
RUN rm -rf /root/.garden/tools/mavend*
RUN rm -rf /root/.garden/tools/hadolint*

FROM aws-cli-v2 as garden
ARG GARDEN_VER=""
ENV USER=root
ENV HOME=/root
ENTRYPOINT []
RUN curl -sL https://get.garden.io/install.sh | bash
ENV PATH=$PATH:$HOME/.garden/bin
RUN garden self-update "$GARDEN_VER"
COPY --from=garden-fetch-tools /root/.garden/tools/ /root/.garden/tools/
WORKDIR /
b

bright-policeman-43626

02/16/2024, 6:22 AM
Oh, I see your magic sauce there to delete the tools you don't want. Nice.
m

mammoth-flag-56137

02/16/2024, 6:23 AM
thats the work around yeah
b

bright-policeman-43626

02/16/2024, 6:23 AM
I'm doing this:
Copy code
FROM gcr.io/cloud-builders/docker
RUN add-apt-repository ppa:longsleep/golang-backports && \
    apt-get update && \
    apt-get install -y golang-go && \
    apt-get install -y rsync && \
    apt-get install -y gcc

RUN GO111MODULE="on" go install sigs.k8s.io/kind@v0.22.0
RUN curl -Lo /usr/local/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && \
              chmod +x /usr/local/bin/kubectl

RUN curl -sL https://get.garden.io/install.sh | bash
ENV PATH="/root/.garden/bin:${PATH}"
RUN garden self-update 0.12.61

ENV PATH /root/go/bin:$PATH
ENTRYPOINT ["/bin/bash"]
I just want that the container gets ready so when I run cloudbuild CI I can just create the kind cluster and deploy with Garden. Already got the kind cluster working, fully ephemeral
they are reasonably sized
b

bright-policeman-43626

02/16/2024, 6:25 AM
Nice multistage build there tbh. I might implement that in mine so I just copy the binaries at the end
m

mammoth-flag-56137

02/16/2024, 6:25 AM
COPY --from= is a nice hack to deal with huge layers
b

bright-policeman-43626

02/16/2024, 6:27 AM
You are actually gonna use this Dockerfile to deploy to AWS right? that's why you are the AWS CLI and stuff I guess.
That's cool, my use case is purely CI so deploy, run tests and turn off the lights
m

mammoth-flag-56137

02/16/2024, 6:30 AM
one of the steps in our deploy is pushing images from gitlab to aws ECR which is where it needs awscli
we use terraform through garden to create ecr s3 etc then push images, then deploy with garden
b

bright-policeman-43626

02/16/2024, 6:31 AM
That's sweet. So Garden E2E basically
m

mammoth-flag-56137

02/16/2024, 6:31 AM
yeah
once you trigger the pipeline everything is automated
b

bright-policeman-43626

02/16/2024, 6:32 AM
How are you handling the versioning? I guess you do a garden publish with some sort of semantic or do you run from release branches?
m

mammoth-flag-56137

02/16/2024, 6:32 AM
so now with our docker image that has garden it also goes faster because it doesnt need to setup garden/awscli every time for every separate job in the pipeline
b

bright-policeman-43626

02/16/2024, 6:33 AM
Yeah. Base images are KEY.
m

mammoth-flag-56137

02/16/2024, 6:33 AM
we use envs in gitlab and garden and manifests that let you target specific container tags
each pipeline will deploy to only the env you tell it to not to every env
b

bright-policeman-43626

02/16/2024, 6:36 AM
This is awesome, y'all are running everything with Garden free tier version right?
m

mammoth-flag-56137

02/16/2024, 6:39 AM
yeah
terraform state stored in gitlab