Logging verbosity in seastar

[travisdowns]

When Redpanda starts up, it outputs a large variety of log lines including things like configuration settings and other stuff, which is often very useful for diagnosing issues. Some of the stuff we would like to know is calculated/known by seastar, and we are very interested for it to be logged as well. One example was these log lines: #1428 which output the io schedule costs for various sized IOs. This was invaluable for diagnosing io scheduler issues.

In many cases it is not possible to log this information from the application itself because the log lines require access private or otherwise hidden implementation accessible only to seastar core.

If I'm reading between the lines correctly, the idea is that a "default" startup of seastar doesn't emit any log lines, other than warnings/unusual case (the line above was one of the only lines, perhaps the only one, that appeared in a typical startup). This is perfect for running tests etc, since there will be zero output unless something goes wrong. However, it means in Redpanda we get ~0 seastar logs also (except [1]).

We'd like to enable more seastar logging when we are running an already "log heavy" context like Redpanda. For example, we'd like to see the log line linked above. Of course, we don't want to undo the "zero logging" policy for tests or whatever.

Not sure the best way to accomplish that.

One suggestion would be that by default seastar-internal loggers are set to WARN log level, so tests etc only show WARN and above, and then in a context where we want to see more we make a call to adjust the level down to INFO during app initialization.

This is not just a bug report, I'm willing to implement the solution if we agree on one.

Other suggestions welcome.

---

[1] Except for Could not parse cgroups v1 file (memory/memory.limit_in_bytes).