https://garden.io logo
#🌱|help-and-getting-started
0.13.X garden-util does not pass in correct permissions into the build context and resulting pods
# 🌱|help-and-getting-started
w

wide-flag-41451

07/05/2023, 7:55 PM
Hi again! I'm fairly confident that this is a bug, or at very least should be documented when moving from 0.12.x to 0.13.x Essentially the problem is this: upgrading from 0.12.x to 0.13.x of garden results in different permissions to be applied to the files synced with garden-util. Example tldr; on version 0.12 we got permissions that mirrored the permissions on the local file system where garden was ran on version 0.13 the permissions were overwritten with 600/700 it seems as though the user and group are being masked to 0 after looking over the code it seems like rsync is nolonger the default? and now garden fully relies on mutagen? We have a frontend image that compiles your typical SPA and loads the content into an nginx:stable-latest docker image, prior to upgrading to version 0.13 everything worked fine. The resulting pod thats created inherits the built frontend with permissions = 600, resulting in NGINX to throw 403's and none of the styles or assets can be loaded. A Simple fix that i did was shell into the pod and chmod the nginx/html directory with the appropriate permissions and that fixed my issue temporarily, however every subsequent build / deploy would cause the issue to reappear I'm open to discussion i've tried various numbers of things, but all of them feel quite hacky, i'm happy to provide some more information or here what other people have done. EDIT: Below is a config i've added to my provider to try and overwrite the default behavior of mutagen.
Copy code
yaml
    sync:
      defaults:
        exclude:
          - '**/node_modules'
        # owner: 1000  # <- set an integer user ID or a string name
        # group: 1000  # <- set an integer group ID or a string name
        fileMode: 644 # <- set the permission bits (as octals) for synced files
        directoryMode: 755 # <- set the permission bits (as octals) for synced directories
I've still not beable to get this to work with this config
I've tried with both 0.13.3 and 0.13.6 ~
q

quaint-dress-831

07/06/2023, 12:59 PM
Hello again! Did changing
fileMode
and
directoryMode
as you demonstrated fix the issue?
w

wide-flag-41451

07/06/2023, 1:11 PM
no
q

quaint-dress-831

07/06/2023, 1:20 PM
Ok, that's not good! So files get set remotely as 600/700 even when you explicitly set?
That is not correct behavior if so
Looks like your keys might be wrong for Bonsai
Copy code
yaml
        defaultOwner: 1000  # <- set an integer user ID or a string name
        defaultGroup: 1000  # <- set an integer group ID or a string name
        defaultFileMode: 0666  # <- set the permission bits (as octals) for synced files
        defaultDirectoryMode: 0777  # <- set the permission b
w

wide-flag-41451

07/06/2023, 1:27 PM
well
in the advance example on that link u provided its exactly what i have
so what you provided is for a deploy action but we are defining it on the provider
I dont want to use a deploy action
er rather i dont want to define this on a deploya ction
q

quaint-dress-831

07/06/2023, 1:31 PM
Yes, you're right, we may be constructing defaultOwner in this case since that's what Mutagen is expecting https://mutagen.io/documentation/synchronization/permissions
You're putting this in a provider and not an action?
w

wide-flag-41451

07/06/2023, 1:33 PM
yes
q

quaint-dress-831

07/06/2023, 1:33 PM
I didn't know we supported Code Sync on the provider level
w

wide-flag-41451

07/06/2023, 1:34 PM
lol
well, previously the default was rsync
which was great
q

quaint-dress-831

07/06/2023, 1:34 PM
Wait wait I think I see the issue here
Excludes are set at the provider level
File permissions are not
w

wide-flag-41451

07/06/2023, 1:36 PM
what am i looking for on this page
isn't this for essentially dev deployment where changes on the host machine will propogate to the pod?
i dont want that for a production deploy
q

quaint-dress-831

07/06/2023, 1:37 PM
If I check both the provider and action documentation permissions are supported at both levels
w

wide-flag-41451

07/06/2023, 1:37 PM
i just want my static files with the correct permissions so my NGINX server can serve the files
q

quaint-dress-831

07/06/2023, 1:39 PM
What command are you running in production?
w

wide-flag-41451

07/06/2023, 1:39 PM
garden deploy
q

quaint-dress-831

07/06/2023, 1:39 PM
Right, permissions will not sync in that case
w

wide-flag-41451

07/06/2023, 1:39 PM
then........... isn't this a breaking change?
q

quaint-dress-831

07/06/2023, 1:40 PM
Sync is only for inner loop development
w

wide-flag-41451

07/06/2023, 1:40 PM
anyone who made static files and didn't chmod them in their docker container will have the wrong permissions
q

quaint-dress-831

07/06/2023, 1:41 PM
If you don't run deploy with --sync the entire sync field of your provider won't resolve
An application that's running in production will have files permissioned at the Dockerfile level
w

wide-flag-41451

07/06/2023, 1:44 PM
yeah but by default the files that get copied into the dockerfile take the permissions from the host machine's context. sure we could chmod it in the docker container but we feel that we shouldn't have to insecure our frontend pod for what essentially is an undocumented breaking change.
i'm just saying that this behavior is not the same as the default in 12.x
i'm gonna try the sync flag
on the deploy
q

quaint-dress-831

07/06/2023, 1:45 PM
"Insecure your frontend pod" not sure what this means
You should always be changing file permissions in your Dockerfile for prod
Sync is only for inner loop development. It was never intended for production deploys
Sync keeps Mutagen up and running so it can sync your work live as you code
w

wide-flag-41451

07/06/2023, 1:46 PM
ok but rysnc was used previously not mutagen
not unless you specified it
in 12.x
can we just go back to rsync?
i've literally never had to chmod static files loaded into an a nginx pod for them to get the correct permissions to serve.
q

quaint-dress-831

07/06/2023, 1:48 PM
Best practice is to do like so
Copy code
Dockerfile
COPY --chown=65332:0 --chmod=775 tests tests

# Copy the source code into the container.
COPY --chown=65332:0 --chmod=775 src/telegram_bot_service_worldofgeese telegram_bot_service_worldofgeese/
w

wide-flag-41451

07/06/2023, 1:49 PM
none of our files are executable
its just css/js/html
why would you be copying tests into a container
also wouldn't this also affect our staging deployments?
q

quaint-dress-831

07/06/2023, 1:50 PM
In this example, because I'm using our
kubernetes-exec
type of Test
w

wide-flag-41451

07/06/2023, 1:50 PM
then this literally is not the same
if this was considered "best practice" then why isn't it in the official NGINX/ docker documentation? because its not best practice.
q

quaint-dress-831

07/06/2023, 1:51 PM
You should be using multi-stage Dockerfiles for the various types of dev/stage/prod containers you're working with, not use third-party tools to shim desired permissions in
Sync was never intended as a tool for deployment. We're sorry if that was unclear
w

wide-flag-41451

07/06/2023, 1:53 PM
i'm not trying to use it as a tool for deployment literally upgraded to 13.x and now my prod builds do not work. this isn't me using sync as a cludge, i just want the default behavior in 12.x to work in 13.x
the issue is in garden-util the permissions are wrong
q

quaint-dress-831

07/06/2023, 1:54 PM
I'm telling you that permissions should be set inside a multi-stage Dockerfile or independent Dockerfiles for each of your environments with permissions set inside
w

wide-flag-41451

07/06/2023, 1:54 PM
no
literally in my 8 years of doing this i've never had to do that once
q

quaint-dress-831

07/06/2023, 1:56 PM
I'm not going to argue with your experience. But I can't continue to assist you using a tool in a way that doesn't conform with what we support. And unfortunately, in Bonsai, this is what we support
w

wide-flag-41451

07/06/2023, 1:57 PM
then you should document a breaking change.
thanks for the help.
s

silly-accountant-45583

07/17/2023, 7:13 AM
@quaint-dress-831 Hi! I am having the same issue as described here after upgrading from
v0.12
to
v0.13
. I think some miscommunication/confusion happened in the above discussion, so I want to just clarify what is really happening in hope to resolve the issue. As I see when running container build action, garden copies project files to
.garden/build/
directory and from there to
util
container inside
garden-buildkit
pod (I am using this build mode). In
v0.12
files were copied as is, with the same unchanged permissions. But in
v0.13
on the stage of copying files from local machine to container in the remote cluster (
.garden/build/
->
util
container inside
garden-buildkit
pod) permissions were modified to
0600
for files and
0700
for directories. These are default permissions used by Mutagen when synchronizing files - https://mutagen.io/documentation/synchronization/permissions#permissions. This is not configurable from garden at all, so this is actually breaking change (or bug) when upgrading from
v0.12
to
v0.13
. Services built (through in-cluster building) and deployed by Garden
v0.12
were working and functioning properly, but not with
v0.13
anymore. I managed to solve issue with wrong permissions by using following Mutagen config at `~/.mutagen.yml`:
Copy code
yaml
sync:
  defaults:
    permissions:
      defaultFileMode: 0644
      defaultDirectoryMode: 0755
However, I think, it can't be considered as a proper solution, because permissions are hardcoded for all docker images and are not defined within Garden project folder (not portable). Ideally, permissions must be copied over with files to
util
container in
garden-buildkit
pod as is. Alternatively, we can define permission for each docker image in corresponding
Dockerfile
, but in this case I would like you to confirm that current behavior won't change in future. Thank you in advance and please let me know if you need any additional info.
q

quaint-dress-831

07/17/2023, 9:22 AM
@silly-accountant-45583, I may have misunderstood then, apologies, and thank you for clarifying! I brought this to our core team minutes ago and we've identified this as fixed by https://github.com/garden-io/garden/pull/4824. Please keep your eyes on that PR; we're working on shipping this fix as a priority!
b

big-spring-14945

08/03/2023, 10:36 AM
Hi @silly-accountant-45583 @wide-flag-41451 I'm really sorry that it took so long for us to indentify and fix this problem. I'm happy to tell you that we now released Garden 0.13.10 that solves this problem: https://github.com/garden-io/garden/releases/tag/0.13.10 The release notes also contain instructions how to solve the problem that images might be cached with the wrong permissions: > If you are using in-cluster build modes like kaniko and buildkit, images that have been built with Garden versions prior to this release might contain files with too restrictive permissions. To remediate this issue, you need to follow these steps: > > 1. Update all Garden clients to 0.13.10 or later (
garden self-update
) > 2. Rebuild all docker images (
garden build -f
) to make sure that permissions are okay in all cached images.
I totally agree with your earlier statements @wide-flag-41451 , adding permissions to the
COPY
directives can merely be a workaround, and even if it may be a good practice to reduce permissions as much as possible, the in-cluster building feature in Garden should stick to the well-known defaults of Docker and Git and use 644 and 755 default permissions for files and directories, unless specified otherwise.
w

wide-flag-41451

08/03/2023, 12:58 PM
Thanks for fixing it. Sorry for the misunderstanding.
b

big-spring-14945

08/03/2023, 1:02 PM
No need to be sorry, thank you so much for reporting this issue! I hope it helps 🥳
31 Views