Pass file / artifact from one task module to another

Hey!

How can I move a file which is collected as an artifact to another module for use in a separate container?

E.g. create a file in a Julia container, export as artifact, run analysis on it in a Python container…

Hi @jeremiahpslewis!

How you do this depends a bit on your use case.

The simple approach is to run a task and have Garden copy the generated artifacts to your local machine and manually pass them on to your Python container.

A more complex but also less manual approach would be to use a shared volume between your modules.

Finally, as an honourable mention, our new hot reload functionality supports two-way syncs. So you can start your service in “dev mode”, shell into it (e.g with garden exec my-service /bin/sh), and what ever changes you make in the remote are synced right back to your local machine.

Would any of these approaches work for your use case?

Thanks for this! Glad I’m not missing anything from the docs; for data use cases, I think this might be a common pattern, e.g. download dataset / generate synthetic data / train model and then pass it on to a different container with specs for (EDA model training / model hosting). Is supplying artifacts as a module input something you might consider adding to the roadmap?

Hi Jeremiah

Apologies for the tardy reply. Artifacts as a module input does sound interesting but I don’t have a good feel for the scope. But definitely something to consider if you create a feature request on GitHub.

In the meantime, would share volumes not do the trick or is there some sequencing that needs to take place where you’d want to rely on the stack graph?