Customizable Output Formatting and Custom Fields

[theShmoo]

Allow users to customize the output format and add custom fields for enhanced log readability.

Describe the problem this feature request solves

Currently, pamburus/hl provides a good way to read JSON log lines. However, it lacks the ability to customize the output format. This limits the user's ability to tailor the output to their specific needs and makes it difficult to add custom fields for more context or analysis.

Describe the solution you'd like

I propose adding the following features:

• Customizable Output Format: Allow users to define their own output format using a template or configuration file. This would enable users to control:
  • The order of fields
  • The display format of each field (e.g., date/time formatting, number formatting)
  • The inclusion/exclusion of specific fields
  • Adding separators or other visual elements
• Custom Fields: Enable users to add custom fields to the output. These fields could be derived from existing fields using expressions or functions. For example:
  • Calculate the duration between two timestamps
  • Extract parts of a string using regular expressions
  • Perform simple calculations on numeric fields

Describe alternatives you've considered

I have considered using other tools for reading JSON logs, but pamburus/hl is currently the best in terms of its core functionality. However, the lack of output customization hinders its usefulness for more advanced use cases.

Additional context

This feature would greatly enhance the flexibility and usability of pamburus/hl, making it a more powerful tool for analyzing JSON log data. It would allow users to create highly customized views of their logs, tailored to their specific needs.

Example use case:

Imagine a log line like this:

{"timestamp": "2024-10-27T10:00:00Z", "level": "INFO", "message": "Request received", "request_id": "12345", "duration_ms": 150}

With customizable output, I could define a format like:

[{{timestamp | date:"%Y-%m-%d %H:%M:%S"}}] [{{level}}] (Request ID: {{request_id}}) - {{message}} (Duration: {{duration_ms}}ms)

Resulting in an output like:

[2024-10-27 10:00:00] [INFO] (Request ID: 12345) - Request received (Duration: 150ms)

This provides a much clearer and more informative view of the log data.

[pamburus]

Thank you for your interest, detailed review and suggestions.

Yes, it is a good idea to make the output layout configuration more flexible.
I've been thinking about it already, maybe it's time to implement it.

[theShmoo]

Thank you for the fast answer!

I would be happy to help you by providing more use cases and priorities of the proposed features if you need them.

[pamburus]

You are always welcome to share your use cases or scenarios. It would be helpful.

But there are 2 major obstacles to overcome.

1. The internal data model is initially highly optimized for performance only for the currently supported cases and lacks flexibility. It does not contain hash tables for fields - all fields are processed sequentially. It also performs lazy parsing for fields only when necessary, i.e. the log record has not been filtered out, the field is not hidden, or its value is needed to apply filtering by field values. However, this lazy parsing is not properly cached, and multiple accesses to the same field multiple times may result in multiple parsings. This is not a big deal now, but to allow templated formatting, this should be improved.

2. There is a work in progress that aims to significantly improve log readability when there are a lot of fields or some fields have multi-line values. It is called expansion. It is intended to automatically pretty print long log records with one field per line and to use better multi-line formatting for multi-line fields. It will also be possible to manually control the thresholds for automatic expansion or force it on/off. It is almost done, but it has a lot of heuristics and dark magic in it, and I am not sure yet if I am ready to merge it into the master branch. The problem with it is that it contains significant refactorings that may interfere with the refactorings needed for this feature.

I'm experimenting with improvements to the internal data models, we'll see what can be done here.