Support for Multiple Regex Patterns in PatternLayout

[Neel1210]

Description:
Currently, users must develop and maintain custom Log4j2 plugins to mask sensitive data (e.g., emails, phone numbers, tokens). This creates significant overhead, especially in microservices architectures, where deploying and updating custom plugin binaries across dozens of services is difficult to manage.

I propose a native feature that allows users to define multiple regex patterns within a single configuration. This would eliminate the need for custom Java code and simplify the rollout of security compliance updates across distributed systems.

Configuration Example ( yml )

    PatternLayout:
      pattern: "%d{yyyy-MM-dd HH:mm:ss} [%t] %-5level %c{1} - %msg%n"
      replace:
        - regex: "([A-Za-z0-9._%+-])([A-Za-z0-9._%+-]*)(@[A-Za-z0-9.-]+\\.[A-Za-z]{2,})"
          replacement: "[EMAIL MASKED]"
        - regex: "([A-Za-z0-9-_=]+\\.[A-Za-z0-9-_=]+\\.)([A-Za-z0-9-_.+/=]+)"
          replacement: "[TOKEN MASKED]"

[vy]

@Neel1210, where exactly do you want this replacement to occur? That is, are you intending to replace the message, or the entire string produced by layout.encode(logEvent)? If the latter is okay with you, I'd suggest introducing a new layout that delegates to another layout (e.g., Pattern Layout) and performs the substitution on its output. This would be a better approach in terms of composability, since it works with any layout.

[Neel1210]

Currently, PatternLayout supports a single RegexReplacement element. My initial thought was to simply refactor this from a single reference to a collection (array/list) of references, allowing for multiple replacement policies within the existing framework.

Regarding your suggestion to introduce a delegating layout that masks the output of layout.encode(logEvent): I see the value in that approach, especially for composability across different layouts (JSON, XML, etc.). However, I need to do some internal research and testing to understand the performance and implementation details of a delegating layout.

In the meantime, I feel that expanding the existing RegexReplacement in PatternLayout is a viable short-term improvement or should we focus exclusively on the delegating layout approach?

[vy]

Regarding your suggestion to introduce a delegating layout that masks the output of layout.encode(logEvent): I see the value in that approach, especially for composability across different layouts (JSON, XML, etc.). However, I need to do some internal research and testing to understand the performance and implementation details of a delegating layout.

@Neel1210, I'd appreciate it if you can quickly check the feasibility of this at your side.

I don't think anything can be slower than what we already have in PatternLayout:

String str = buffer.toString();
str = replace.format(str);

By moving this logic to a dedicated generalized component, we can achieve

• Garbage-free replacements
• Layout-agnostic usability
• Home for further utilities (e.g., concatenating output from multiple layouts)

In the meantime, I feel that expanding the existing RegexReplacement in PatternLayout is a viable short-term improvement or should we focus exclusively on the delegating layout approach?

I am not really sure if we can overload replace to accept both RegexReplacement and RegexReplacement[]. I don't even know how to represent this in XSD. I'm also reluctant to add more knobs to PatternLayout. But that is me.

@ppkarwasz, WDYT?

[Neel1210]

@vy, I see your point regarding the performance limitations of the current PatternLayout implementation. Transitioning to a garbage-free, layout-agnostic component definitely makes more sense for the long-term health of the project.

I will shift my focus to investigating the 'Delegating Layout' approach. Since you mentioned concerns about the XSD and 'knobs,' do you have a specific existing class in the Log4j2 codebase I should look at as a reference for a delegating/wrapper layout?

[ppkarwasz]

I am not really sure if we can overload replace to accept both RegexReplacement and RegexReplacement[]. I don't even know how to represent this in XSD. I'm also reluctant to add more knobs to PatternLayout. But that is me.

This is actually not so hard: if we replace the RegexReplacement field with RegexReplacement[] it will still accept 0 or 1 elements of type RegexReplacement.

By moving this logic to a dedicated generalized component, we can achieve

• Garbage-free replacements
• Layout-agnostic usability
• Home for further utilities (e.g., concatenating output from multiple layouts)

Replacing an already stringified log event can be dangerous, since it can break structured layouts.

The problem of masking sensitive data in logs appears often in discussions (e.g. #1814 and #3152) and the usual advice is do not log sensitive data.

More precisely you can use some static analysis tool to make sure that sensitive information is not passed to loggers. One of the available tools is Palantir's safe-logging logging API and baseline-error-prone set of checks.

Another alternative is to wrap sensitive strings in objects that override toString() with a masked form.

[Neel1210]

The problem of masking sensitive data in logs appears often in discussions (e.g. #1814 and #3152) and the usual advice is do not log sensitive data.

@ppkarwasz, exactly. The discussions you linked highlight the need, but the reliance on custom plugins creates a significant maintenance burden for microservice architectures. In an environment with 20–30+ microservices, the operational overhead of managing and updating custom plugin binaries across every service becomes quite difficult.

This is actually not so hard: if we replace the RegexReplacement field with RegexReplacement[] it will still accept 0 or 1 elements of type RegexReplacement.

Regarding the implementation: I have tested the RegexReplacement[] approach in a fork, and it works as expected while maintaining backward compatibility for single-replacement configurations.

PoC Commit: [commit]
Demo Repo: [repo]

@vy, regarding the Universal Masking approach: Would the goal be to create something structurally similar to a RewriteAppender? A "MaskingAppender" that takes a nested Layout plus a collection of RegexReplacement elements would provide the composability you mentioned, solving the rollout issue for both Pattern and JSON layouts simultaneously.

[Neel1210]

@ppkarwasz @vy Attached is a reference implementation for a masking plugin designed to be loosely coupled with specific regex logic. By moving the pattern definitions into the Log4j2 configuration file, the plugin remains generic, highly reusable, and avoids the "maintenance tail" associated with hard-coded patterns.

You can find the full implementation and demo here: [Link to Repo]

The goal was to create a "plug-and-play" solution that we can suggest to other developers facing similar masking challenges until a native multi-regex feature is finalized. This approach allows users to define or update sensitive data patterns (emails, tokens, etc.) directly in the XML/YAML config without needing to recompile or redeploy custom plugin binaries for every new requirement.

Key Features:
Loosely Coupled: The masking engine is independent of the data patterns it processes.
Fully Configurable: All regex patterns are injected via the configuration file, allowing for real-time updates to security policies.

Implementation Details:

@Plugin(
        name = "LogWrapper",
        category = "Core",
        printObject = true
)
public class LogWrapper implements RewritePolicy {
    private final RegexReplacement[] replace;

    private LogWrapper(final RegexReplacement[] replace) {
        this.replace = replace;
    }

    @PluginFactory
    public static LogWrapper createPlugin(@PluginElement("Replace") final RegexReplacement[] replace) {
        return new LogWrapper(replace);
    }

    @Override
    public LogEvent rewrite(LogEvent event) {
        String message = event.getMessage().getFormattedMessage();
        for (RegexReplacement r : replace) {
            message = r.format(message);
        }
        return new Log4jLogEvent.Builder(event)
                .setMessage(new SimpleMessage(message))
                .build();
    }
}

Configuration File ( xml ) :

    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss} [%t] %-5level %c{1} - %msg%n"/>
        </Console>
        <Rewrite name="rewrite">
            <LogWrapper>
                <replace regex="([A-Za-z0-9._%+-])([A-Za-z0-9._%+-]*)(@[A-Za-z0-9.-]+\.[A-Za-z]{2,})"
                         replacement="[EMAIL]"/>
                <replace regex="\+?1?\s*\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}"
                         replacement="[PHONE]"/>
                <replace> <!-- Uses a CDATA Section -->
                    <regex><![CDATA[[A-Za-z0-9_%+-]{20,}\.[A-Za-z0-9_%+-]{20,}\.[A-Za-z0-9_%+-]+\b]]></regex>
                    <replacement>[TOKEN]</replacement>
                </replace>
            </LogWrapper>
            <Appender-Ref ref="Console"/>
        </Rewrite>
    </Appenders>

Configuration File ( yml ) :

  appenders:
    console:
      - name: Console
        target: SYSTEM_OUT
        PatternLayout:
          pattern: "%d{yyyy-MM-dd HH:mm:ss} [%t] %-5level %c{1} - %msg%n"
    rewrite:
      - name: rewrite
        LogWrapper:
          replace:
            - regex: "([A-Za-z0-9._%+-])([A-Za-z0-9._%+-]*)(@[A-Za-z0-9.-]+\\.[A-Za-z]{2,})"
              replacement: "[EMAIL MASKED]"
            - regex: "\\+?1?\\s*\\(?\\d{3}\\)?[\\s.-]?\\d{3}[\\s.-]?\\d{4}"
              replacement: "[PHONE]"
            - regex: "[A-Za-z0-9_%+-]{20,}\\.[A-Za-z0-9_%+-]{20,}\\.[A-Za-z0-9_%+-]+\\b"
              replacement: "[TOKEN]"
        AppenderRef:
          - ref: Console