[Security Solution] Reverting customization on prebuilt rule with missing base version after update does not reset is_customized status for fields covered by the scalar diff array

[pborgonovi]

Description:

When a prebuilt rule with a missing base version is customized and then updated, reverting the customization incorrectly leaves the rule marked as customized instead of resetting is_customized to false.
Same can be observed when order of steps is inverted: Customize -> Revert changes -> Update the rule

The modified field is one of the fields covered by the scalar diff array (e.g., tags, references, new_terms_fields, threat_index).

Once the rule is updated, it will have a base version again, as we always store the most recent version of a rule in the package. This means that after the update, the rule should behave like any other rule with a base version and should follow the standard logic where reverting the customization resets is_customized=false.

Evidence 1

Modifying References field:

Screen.Recording.2025-03-07.at.10.41.52.AM.mov

Evidence 2

Tags field:

Screen.Recording.2025-03-07.at.10.36.30.AM.mov

Inverted steps

Screen.Recording.2025-03-07.at.1.59.35.PM.mov

Kibana/Elasticsearch Stack version:

VERSION: 9.1.0
BUILD: 84372
COMMIT: 636c06b

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):

Prebuilt Rules Update

** Pre requisites:**

1. prebuiltRulesCustomizationEnabled flag is enabled
2. Prebuilt rules are available
3. Prebuilt rules with missing base version are available
4. At least one prebuilt rule with missing base version has update available

Steps to reproduce:

Scenario 1:

1. Customize a field covered by the scalar diff array (e.g., tags, references).
2. Upgrade the rule using the Rule Updates table or Upgrade Flyout.
3. Confirm that is_customized=true after the upgrade (expected behavior).
4. Revert the modifications via the rule customization options.

Scenario 2:

1. Customize a field covered by the scalar diff array (e.g., tags, references).
2. Revert the modifications via the rule customization options.
3. Confirm that is_customized=true after the changes are reverted (expected behavior).
4. Upgrade the rule using the Rule Updates table or Upgrade Flyout.

Current behavior:

After reverting changes and updating the rule (or updating the rule then reverting changes), the rule remains marked as customized (is_customized=true) even though no actual modifications exist and we've stored the most recent version.

Expected behavior:

If a rule customization is reverted, the rule should no longer be marked as customized, and is_customized should be set to false.
Once the rule is updated, it will have a base version again, as we always store the most recent version of a rule in the package. This means that after the update, the rule should behave like any other rule with a base version and should follow the standard logic where reverting the customization resets is_customized=false.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[pborgonovi]

It's working as expected as specified in 210358:

We should mark the rule as customized, only if the new rule settings are different from the current rule settings.
For example, adding a new tag should mark the rule as customized. Then, if the user removes this tag, the rule should remain to be marked as customized. This matches the current behavior.
However, if the user saves the rule without making any changes to it, it should keep its is_customized field as is. This is different from the current behavior.