--log-dir has disappeared

[mreferre]

Hi there. I am experimenting with Spin in a (very) constrained environment where the only writable folder is /tmp. I used to be able to play with Spin easily by launching it with --log-dir /tmp and that worked just fine. The latest Spin release seems to have dropped that flag? I was able to workaround that problem by setting the $HOME variable to /tmp (because I noticed Spin trying to create folders/files in the $HOME folder. It works (but it's a bit extreme).

Is the --log-dir gone for good? It seemed to be useful.

Thanks!

[itowlson]

@mreferre Which build are you using? It's there for me with current GitHub HEAD, but listed under "Trigger Options" rather than "Options":

(I think the distinction between the two categories is an implementation detail... generally up options that take effect when the trigger runs rather than when the app is loading I believe.)

[mreferre]

Thanks @itowlson.

How do you get that list of flags? I am using Spin 0.5.0 and this is what I see when I explore it with the --help flag:

ubuntu@ip-172-31-43-115:~$ spin --version
spin 0.5.0 (b9fedcc 2022-09-02)
ubuntu@ip-172-31-43-115:~$ 

ubuntu@ip-172-31-43-115:~$ spin up --help
spin-up 
Start the Spin application

USAGE:
    spin up [OPTIONS]

OPTIONS:
        --allow-transient-write
            Set the static assets of the components in the temporary directory as writable

    -b, --bindle <BINDLE_ID>
            ID of application bindle

        --bindle-password <BINDLE_PASSWORD>
            Basic http auth password for the bindle server [env: BINDLE_PASSWORD=]

        --bindle-server <BINDLE_SERVER_URL>
            URL of bindle server [env: BINDLE_URL=]

        --bindle-username <BINDLE_USERNAME>
            Basic http auth username for the bindle server [env: BINDLE_USERNAME=]

    -f, --file <APP_CONFIG_FILE>
            Path to spin.toml

    -h, --help
            

    -k, --insecure
            Ignore server certificate errors from bindle server

        --temp <TMP>
            Temporary directory for the static assets of the components

ubuntu@ip-172-31-43-115:~$ 

I did not spend a considerable amount of time on it but I concluded what I concluded because the script I was using (which included the --log-dir flag didn't work anymore - as in Spin would run but would just ignore that flag) and when I explored the syntax (above) the flag was no longer there.

I am mildly confused but glad this could be a user error on my part with a path to happiness ;)

[itowlson]

Ah... do you have a spin.toml file in the current directory (or pointed to by -f)? The trigger options section appears only if there is a trigger 'available', because some options are trigger-specific. So even for --help Spin needs to be able to work out what trigger would run.

Using Spin 0.5.0, if I run spin up --help in a directory without a spin.toml file, I see the same shorter list as you. If I run it in a directory with a spin.toml file, I see the full list. I can also run spin up -f blah/spin.toml --help if the spin.toml is in another directory.

As your original issue was that --log-dir was being ignored for you (and not just the help UI), I checked with a HTTP application, and the --log-dir option is certainly working for me in Spin 0.5.0. Looking at the code, it may be the case that --log-dir doesn't take effect until Spin resolves the trigger (e.g. if something is logged during application load then that might not respect it), but I'm not sure. Could that be the situation for you, or was it not working even for request logging?

[mreferre]

Very interesting. Thanks for the engagement on this.

The environment has his quirks. In full disclosure, I am trying to run this stack in a Lambda function (hence the restrictions above - the only directory that is r/w is /tmp). I am baking in a container the spin binary and then I map an EFS file system (under /mnt/wasm) which contains the toml file and the wasm artifact. I am using the Lambda adapter for this, and here you can read more about the idea (this blog uses NGINX whereas here I am using Spin. Also, the blog usesreweb which is functionally equivalent to the Lambda adapter).

I am not suggesting this is a viable architecture in all cases. It's more of a science experiment.

What happened is that I tested this (Spin in Lambda) back in April and it worked just fine. When I tested it again a few days ago (with the only tangible difference was the spin version because in the Dockerfile I pull the LATEST version) it broke. There may be other subtle differences I would need to explore more but I am thinking the problem could be due to a change in the Spin behaviour.

Also, I read the description of the --log-dir flag as Log directory for the stdout and stderr of components (yes now that I am running in the same folder where I have the toml file I see all flags - good catch!).

The error I see in my Lambda is this:

Error: failed to create cache directory: /home/sbx_user1051/.cache/wasmtime
Caused by:
Read-only file system (os error 30)
Error: exit status: 1
EXTENSION	Name: lambda-adapter	State: Ready	Events: []

I have always been pointing to --log-dir as the culprit of the error ("it doesn't work / it's ignored") but right now I am leaning more to think that that flag indeed works for me too and the different behavior I am observing is not related to Log directory for the stdout and stderr of components.

IIRC back in April I had to add --log-dir at some point to make it work in Lambda so that may still be required but I am wondering if newer Spin versions do more "writes" that --log-dir alone cannot control and that Lambda cannot digest.

Mostly thinking out loudly here :)

Thanks!

[itowlson]

Yes, the cache was introduced in a recent version of Spin. That use of the cache path is controlled by --disable-cache; and I think you can change the path via a cache config file specified using --cache (config file syntax: https://docs.wasmtime.dev/cli-cache.html).

More generally, it sounds like we should provide documentation on which directories Spin writes to. That's not something I can put together right now, I'm afraid, but definitely worth doing. And I don't think we'd be able to guarantee it wouldn't change from version to version.

Looking at the usage of --log-dir, it does look like it's only used for the stdout and stderr of guest modules. Spin's own logging goes to the Rust trace destination, which by default is stderr. So you would need to use whatever mechanism Lambda provides for stderr to see that. Sorry for the confusion.

Hope this all helps! I'm intrigued to hear the results of your science experiment. And of course happy to keep digging if you still run into problems.

[mreferre]

Thanks for confirming my suspect. I played with it a bit more. If I add the --disable-cache flag to the spin up AND I remove the system variable setting HOME=/tmp I no longer get the Error: failed to create cache directory: /home/sbx_user1051/.cache/wasmtime error message BUT I get a long string of ERROR spin_http_engine: Error processing request: Read-only file system (os error 30).

My guess is that it's no longer trying to build the cache (there no explicit cache error message) BUT it's still trying to touch the file system somewhere in my home (and failing in doing so). If I add the variable setting HOME=/tmp all starts to work (with the --disable-cache even though with the variable we know that's not required).

I don't know what you want to get out of all this for users using Spin in a more "traditional" way. As I said mine is a bit of a science experiment and it would be a waste of time on your side to try to hammer spin to make it work. I am fine with the work-around I found with the HOME variable for now.

For context, what I am trying to do here is very much related to this other discussion: fermyon/installer#70. That is a bit more practical (IMO) and here I am just pushing the envelop even further by demonstrating being able to run the same WASM artifact (via Spin) on both EC2 (easy), Fargate (easy) and Lambda (a bit more bumpy as you can see). I think EC2/Fargate are legit, Lambda is the science-ish part.

Thanks!

[itowlson]

Thanks for the update. I'm wondering if the R/O error could have anything to do with file mounts (assets or transient writes from the guest code). But if you're okay with your workaround I'll defer any further investigation until things are quieter...!

[mreferre]

Good (excellent) question. I am traveling this week and I can't spend time on this but I have looked at the EFS configuration and, from what I am seeing (to be confirmed), the mount of the 6 months old setup that works is R/W while the mount I am using for the current setup is R/O. This has nuances because root can have R/W on the new setup (but anyone else doesn't). The fact that when I set the ENV variable $HOME to /tmp makes me think that having R/O access to the mount might not be the problem but at this point I am no longer sure. I will update this thread with my findings next week after playing with the mount and making it R/W for all users.

Thanks!

[mreferre]

I was eventually able to test this and I can confirm that with full R/W access to the EFS shared folder, the combination of --log-dir as well as --disable-cache did the trick and the Lambda function works without the need to set the HOME variable to /tmp.

I have implemented this exercise in this repository: https://github.com/mreferre/spin-wasm-multi-compute

The commit that I used to test the above was this: mreferre/spin-wasm-multi-compute@13eb048

Thanks a ton! I will close this issue now.

[itowlson]

Thanks for circling back - delighted to hear it's working and thanks for the link!