feat: implement unified JSON/text logging format #4392

lngarrett · 2025-09-28T01:49:37Z

This provides a unified logging system where users can switch between human-readable text logs and structured JSON logs across all output destinations: console, files, and WebSocket/UI debug view.

Add logFormat setting to Gateway config (text/json)
Update UI logger to respect global logFormat setting
Add log format selector in Settings UI
Update Debug view to handle both text and JSON log formats
Implement driver logging configuration for JSON mode
Use DailyRotateFile for JSON file transport to maintain symlink functionality
Redirect driver internal file transport to /dev/null in JSON mode. See details below.
Add WebSocket streaming for both text and JSON formats
Add basic tests for logging format functionality

Why do I write a /dev/null file in JSON mode?

The zwave-js driver's internal logic creates a console transport when (isTTY && !logToFile) is true, but we needed logToFile: true to prevent console output in JSON mode. However, setting logToFile: true also creates an unwanted text file transport that writes to the same filename as our JSON logs, causing duplicate/conflicting file output.
Solution: I redirected the driver's internal file transport to /dev/null using filename: /dev/null in the driver's logConfig, which satisfies the driver's internal logic (preventing console transport creation) while discarding the unwanted text file output, allowing our custom JSON file transport to handle the actual file logging cleanly.

api/lib/ZwaveClient.ts

src/views/Debug.vue

api/lib/ZwaveClient.ts

lngarrett · 2025-09-29T14:52:35Z

Appreciate the feedback. I'll review these and make the requested changes.

lngarrett · 2025-09-30T01:03:31Z

I see a few multi-line logs from the driver still. I'd like to see if I can get the raw structured version out - working on this now.

lngarrett · 2025-09-30T01:24:24Z

I think I've structured all I can without driver changes. IIUC, The structured data gets permanently lost in the driver's logging pipeline when messageRecordToLines() converts the original MessageRecord objects (like {"source node id": 1, "callback id": 9}) into formatted string arrays (like ["source node id: 1", "callback id: 9"]) before the data ever reaches zwave-js-ui.

I could lightly parse this into a single line so that centralized logging systems are grok parse it more easily. But, I think it might be better to remain unopinionated on that and perhaps do additional structuring in the driver itself.

For example, here's a not-so-structured log from the driver:

{
    "context": {
        "direction": "outbound",
        "source": "driver"
    },
    "direction": "» ",
    "label": "DRIVER",
    "level": "verbose",
    "message": [
        "[Node 002] [REQ] [SendDataBridge]",
        "│ source node id:   1",
        "│ transmit options: 0x01",
        "│ callback id:      9",
        "└─[NoOperationCC]"
    ],
    "timestamp": "2025-09-29 21:17:21.878"
}

AlCalzone · 2025-09-30T06:59:13Z

I double-checked. Formatting the messages happens outside of the transport format, in https://github.com/zwave-js/zwave-js/blob/266f47f49f804609d633313fe128a81c987732ff/packages/zwave-js/src/lib/log/Driver.ts#L113

We'd need to add an option raw to the log config in the driver options and evaluate that to skip the more complex formatting and instead JSON.stringify the logEntry to get the message.

AlCalzone · 2025-09-30T07:53:46Z

I've added the driver option in zwave-js/zwave-js#8204

lngarrett · 2025-09-30T13:16:47Z

Great, thanks for adding that support. The log is now much more structured:

    {
        "context": {
            "direction": "outbound",
            "source": "driver"
        },
        "direction": "» ",
        "label": "DRIVER",
        "level": "verbose",
        "message": {
            "message": {
                "callback id": 9,
                "source node id": 1,
                "transmit options": "0x01"
            },
            "tags": [
                "Node 002",
                "REQ",
                "SendDataBridge"
            ]
        },
        "primaryTags": "[Node 002] [REQ] [SendDataBridge]",
        "timestamp": "2025-09-30 09:05:40.680"
    },

Edit: Noticed message.message - I've flattened that below.

robertsLando · 2025-10-01T07:04:34Z

api/lib/ZwaveClient.ts

+	 * Setup driver logging based on the configured format (text or JSON)
+	 * Uses the same logFormat setting as the gateway for consistency
+	 */
+	private setupDriverLogging(zwaveOptions: PartialZWaveOptions) {


I think the code in this function is overcomplicated and introduces lot of repeated code

I broke this up into more manageable functions and made it more DRY.

robertsLando · 2025-10-01T07:05:33Z

src/views/Debug.vue

 			// no need to make this reative
 			this.prevScrollTop = scrollTop
 		},
+		formatLog(data) {


as said I would like to avoid this kind of parsing if possible on frontend side

Right, I missed that sorry. I took Debug.vue back to original state and handled this via the zwave client. I left the original minimal formatting designed for text as it wasn't negatively impacting JSON.

robertsLando

@AlCalzone Do you think this could be simplified ?

lngarrett · 2025-10-06T21:12:24Z

@robertsLando @AlCalzone - Are there any other requested changes / feedback on this?

AlCalzone · 2025-10-09T08:40:54Z

api/lib/ZwaveClient.ts

+		const parseFormat = this.createParseDriverJsonFormat()
+		const jsonFormat = winston.format.combine(
+			parseFormat(),
+			winston.format.timestamp(),


Have you tested if this is needed?
The ZWaveLogInfo objects from zwave-js should already contain a timestamp.

I removed this, but as I was testing I noticed another issue. The UI logs are in UTC with timezone indication. However, the driver logs are in local time with no timezone indication. I could convert this, but I think this is probably a concern of the driver to support UTC? The challenge isn't so much the usage of localtime, but rather the lack of TZ info.

Do you agree that the driver should at least use a format with TZ, and possibly also be UTC?

api/lib/ZwaveClient.ts

AlCalzone · 2025-10-09T08:49:40Z

src/views/Settings.vue

+											:items="logFormats"
+											v-model="newGateway.logFormat"
+											label="Log Format"
+											hint="Choose between human-readable text or structured JSON logging"


I'm a bit afraid that making users choose hinders our troubleshooting capabilities. Everything from the automated bot comments when users post the wrong log, over the automatic log analyzer, to my feeble brain expects the text-based logging format.

Can we instead just add the JSON based logs as an additional transport that can be toggled?

I'm trying to think about how best to do this. My use case for JSON logs is to ship them to centralized logging tools - ideally this reads the STDOUT of the container/process. Alternatively, it would read known files, but that's less ideal as the files then need to be specified whereas STDOUT is assumed.

I see no problem in always making the web console text format. I actually almost did that, but decided consistency was better. But, I think it would make sense to do so.

Where I'm stuck is - what is your expectation of the format of STDOUT and file logs in the case where the additional JSON transport is toggled on?

I think configuring the STDOUT should be fine. Our most common troubleshooting use case is having users write debug logs to file and uploading those files.

It just occurred to me that this presents a conflict I believe:

Driver's raw option is global - it affects ALL transports (console, file, web UI)

When raw: true -> JSON for everything (loses human-readable formatting)

When raw: false -> Text for everything (can't get JSON console)

We could move human parsing logic to the UI, but that's duplicative to what the driver already does in text mode so it feels a bit hacky. Is there another way to solve this that I'm missing?

I guess I'll have to tie the message formatting into the transport format, so it can be toggled on a per-transport basis.
I've raised zwave-js/zwave-js#8375 for this. The solution probably involves getting rid of the raw option again though, so this PR should be considered blocked by that.

- Add support for JSON logging format alongside existing text format - JSON mode outputs structured JSON logs to console, files, and debug view - Text mode continues to output human-readable text with ANSI colors - Debug view displays raw format as received (JSON strings or formatted text) - Logging format configurable via gateway settings (logFormat: 'json' | 'text') - Both formats use consistent JSONTransport architecture - Maintains backward compatibility with existing text logging behavior This enables structured logging for better log parsing, monitoring, and integration with log aggregation systems while preserving the existing text format for human readability.

lngarrett · 2025-10-15T12:28:04Z

@AlCalzone I think I've addressed everything with the exception of one open question:

Remvoed additional timestamp. If you agree the missing TZ info should come from driver then this should be good to go.
Message flattening is removed
JSON/Text Toogle - I have one question above regarding what format to prefer in file, console, and UI formats when JSON logging is active.

lngarrett changed the title ~~Feature: Implement JSON structured logs~~ feat: implement unified JSON/text logging format Sep 28, 2025

robertsLando requested changes Sep 29, 2025

View reviewed changes

api/lib/ZwaveClient.ts Outdated Show resolved Hide resolved

src/views/Debug.vue Outdated Show resolved Hide resolved

src/views/Debug.vue Outdated Show resolved Hide resolved

robertsLando requested a review from AlCalzone September 29, 2025 13:15

AlCalzone requested changes Sep 29, 2025

View reviewed changes

api/lib/ZwaveClient.ts Outdated Show resolved Hide resolved

lngarrett force-pushed the master branch from c8af97e to 2267d93 Compare September 30, 2025 00:56

lngarrett requested review from AlCalzone and robertsLando September 30, 2025 00:57

AlCalzone mentioned this pull request Sep 30, 2025

feat: add driver option to skip log formatting of Z-Wave commands zwave-js/zwave-js#8204

Merged

lngarrett force-pushed the master branch from 2267d93 to 2373d7c Compare September 30, 2025 13:15

lngarrett force-pushed the master branch from 2373d7c to 3ea8119 Compare September 30, 2025 13:20

robertsLando requested changes Oct 1, 2025

View reviewed changes

lngarrett force-pushed the master branch 3 times, most recently from 63f355f to b0c263f Compare October 2, 2025 01:05

lngarrett requested a review from robertsLando October 2, 2025 01:05

robertsLando reviewed Oct 2, 2025

View reviewed changes

AlCalzone requested changes Oct 9, 2025

View reviewed changes

lngarrett force-pushed the master branch from b0c263f to 9adefe0 Compare October 9, 2025 13:26

AlCalzone mentioned this pull request Oct 21, 2025

Refactor serial message hierarchy formatting into a toggle-able transport formatter zwave-js/zwave-js#8375

Open

Copilot AI mentioned this pull request Oct 28, 2025

Refactor serial message hierarchy formatting into toggle-able transport formatter zwave-js/zwave-js#8387

Draft

Uh oh!

feat: implement unified JSON/text logging format #4392

Are you sure you want to change the base?

feat: implement unified JSON/text logging format #4392

Uh oh!

Conversation

lngarrett commented Sep 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lngarrett commented Sep 29, 2025

Uh oh!

lngarrett commented Sep 30, 2025

Uh oh!

lngarrett commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlCalzone commented Sep 30, 2025

Uh oh!

AlCalzone commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lngarrett commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lngarrett Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robertsLando left a comment

Choose a reason for hiding this comment

Uh oh!

lngarrett commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lngarrett Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlCalzone Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lngarrett commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lngarrett commented Sep 28, 2025 •

edited

Loading

lngarrett commented Sep 30, 2025 •

edited

Loading

AlCalzone commented Sep 30, 2025 •

edited

Loading

lngarrett commented Sep 30, 2025 •

edited

Loading

lngarrett Oct 2, 2025 •

edited

Loading

lngarrett commented Oct 6, 2025 •

edited

Loading

lngarrett Oct 9, 2025 •

edited

Loading

AlCalzone Oct 21, 2025 •

edited

Loading

lngarrett commented Oct 15, 2025 •

edited

Loading