Hi again! I'm fairly confident that this is a bug, or at very least should be documented when moving from 0.12.x to 0.13.x Essentially the problem is this: upgrading from 0.12.x to 0.13.x of garden results in different permissions to be applied to the files synced with garden-util. Example tldr; on version 0.12 we got permissions that mirrored the permissions on the local file system where garden was ran on version 0.13 the permissions were overwritten with 600/700 it seems as though the user and group are being masked to 0 after looking over the code it seems like rsync is nolonger the default? and now garden fully relies on mutagen? We have a frontend image that compiles your typical SPA and loads the content into an nginx:stable-latest docker image, prior to upgrading to version 0.13 everything worked fine. The resulting pod thats created inherits the built frontend with permissions = 600, resulting in NGINX to throw 403's and none of the styles or assets can be loaded. A Simple fix that i did was shell into the pod and chmod the nginx/html directory with the appropriate permissions and that fixed my issue temporarily, however every subsequent build / deploy would cause the issue to reappear I'm open to discussion i've tried various numbers of things, but all of them feel quite hacky, i'm happy to provide some more information or here what other people have done. EDIT: Below is a config i've added to my provider to try and overwrite the default behavior of mutagen.

yaml
    sync:
      defaults:
        exclude:
          - '**/node_modules'
        # owner: 1000  # <- set an integer user ID or a string name
        # group: 1000  # <- set an integer group ID or a string name
        fileMode: 644 # <- set the permission bits (as octals) for synced files
        directoryMode: 755 # <- set the permission bits (as octals) for synced directories

I've still not beable to get this to work with this config

Hello again! Did changing

fileMode

and

directoryMode

as you demonstrated fix the issue?

no

Ok, that's not good! So files get set remotely as 600/700 even when you explicitly set?

well

Yes, you're right, we may be constructing defaultOwner in this case since that's what Mutagen is expecting https://mutagen.io/documentation/synchronization/permissions

yes

I didn't know we supported Code Sync on the provider level

lol

Wait wait I think I see the issue here

what am i looking for on this page

If I check both the provider and action documentation permissions are supported at both levels

i just want my static files with the correct permissions so my NGINX server can serve the files

What command are you running in production?

garden deploy

Right, permissions will not sync in that case

then........... isn't this a breaking change?

Sync is only for inner loop development

anyone who made static files and didn't chmod them in their docker container will have the wrong permissions

If you don't run deploy with --sync the entire sync field of your provider won't resolve

yeah but by default the files that get copied into the dockerfile take the permissions from the host machine's context. sure we could chmod it in the docker container but we feel that we shouldn't have to insecure our frontend pod for what essentially is an undocumented breaking change.

"Insecure your frontend pod" not sure what this means

ok but rysnc was used previously not mutagen

Best practice is to do like so

none of our files are executable

In this example, because I'm using our

kubernetes-exec

type of Test

then this literally is not the same

You should be using multi-stage Dockerfiles for the various types of dev/stage/prod containers you're working with, not use third-party tools to shim desired permissions in

i'm not trying to use it as a tool for deployment literally upgraded to 13.x and now my prod builds do not work. this isn't me using sync as a cludge, i just want the default behavior in 12.x to work in 13.x

I'm telling you that permissions should be set inside a multi-stage Dockerfile or independent Dockerfiles for each of your environments with permissions set inside

no

I'm not going to argue with your experience. But I can't continue to assist you using a tool in a way that doesn't conform with what we support. And unfortunately, in Bonsai, this is what we support

then you should document a breaking change.

@quaint-dress-831 Hi! I am having the same issue as described here after upgrading from

v0.12

to

v0.13

. I think some miscommunication/confusion happened in the above discussion, so I want to just clarify what is really happening in hope to resolve the issue. As I see when running container build action, garden copies project files to

.garden/build/

directory and from there to

util

container inside

garden-buildkit

pod (I am using this build mode). In

v0.12

files were copied as is, with the same unchanged permissions. But in

v0.13

on the stage of copying files from local machine to container in the remote cluster (

.garden/build/

->

util

container inside

garden-buildkit

pod) permissions were modified to

0600

for files and

0700

for directories. These are default permissions used by Mutagen when synchronizing files - https://mutagen.io/documentation/synchronization/permissions#permissions. This is not configurable from garden at all, so this is actually breaking change (or bug) when upgrading from

v0.12

to

v0.13

. Services built (through in-cluster building) and deployed by Garden

v0.12

were working and functioning properly, but not with

v0.13

anymore. I managed to solve issue with wrong permissions by using following Mutagen config at `~/.mutagen.yml`:

yaml
sync:
  defaults:
    permissions:
      defaultFileMode: 0644
      defaultDirectoryMode: 0755

However, I think, it can't be considered as a proper solution, because permissions are hardcoded for all docker images and are not defined within Garden project folder (not portable). Ideally, permissions must be copied over with files to

util

container in

garden-buildkit

pod as is. Alternatively, we can define permission for each docker image in corresponding

Dockerfile

, but in this case I would like you to confirm that current behavior won't change in future. Thank you in advance and please let me know if you need any additional info.

@silly-accountant-45583, I may have misunderstood then, apologies, and thank you for clarifying! I brought this to our core team minutes ago and we've identified this as fixed by https://github.com/garden-io/garden/pull/4824. Please keep your eyes on that PR; we're working on shipping this fix as a priority!

Hi @silly-accountant-45583 @wide-flag-41451 I'm really sorry that it took so long for us to indentify and fix this problem. I'm happy to tell you that we now released Garden 0.13.10 that solves this problem: https://github.com/garden-io/garden/releases/tag/0.13.10 The release notes also contain instructions how to solve the problem that images might be cached with the wrong permissions: > If you are using in-cluster build modes like kaniko and buildkit, images that have been built with Garden versions prior to this release might contain files with too restrictive permissions. To remediate this issue, you need to follow these steps: > > 1. Update all Garden clients to 0.13.10 or later (

garden self-update

) > 2. Rebuild all docker images (

garden build -f

) to make sure that permissions are okay in all cached images.

Thanks for fixing it. Sorry for the misunderstanding.

No need to be sorry, thank you so much for reporting this issue! I hope it helps 🥳