[Security Solution] Prebuilt rule customization is lost on upgrade when Base version is missing

[banderror]

Epic: #174168
Should be fixed by: #187645
Related to: #200285

Summary

Note

The affected functionality is hidden behind the prebuiltRulesCustomizationEnabled feature flag.

Currently, when you customize a prebuilt rule which has an "update" (can be upgraded to a new version), and the customized field in the Target version is unchanged compared to the Base one, on upgrade the diff algorithm picks the Target version as the Final one, which effectively resets the user customization.

Steps to reproduce (example for the tags field):

• In the Base version of a prebuilt rule let's say you had some tags, e.g. ['foo', 'bar'].
• You customized this rule and removed all tags from it.
• In the Current version you now have an empty array of tags: [].
• In the Target version from Elastic these tags were not changed and stay the same: ['foo', 'bar'].

Expected result:

The diff algorithm should keep the user customization and pick the Current version as the Final one: [].

Actual result:

The diff algorithm picks the Target version as the Final one: ['foo', 'bar']. Which is incorrect by itself (the user loses their customization, although it's possible to restore it manually in the Upgrade flyout) and also generates a confusing diff that is shown by default between the Current and the Final versions.

Screenshots

Example video:

Screen.Recording.2024-11-14.at.3.57.45.PM.mov

Reason

Currently, this bug exists because the Base version doesn't exist in most cases. Without the Base version, the diff algorithm:

• identifies the upgrade case as -AB and SOLVABLE conflict
• cannot know for sure if it should keep the user customization (A) or the Target version (B) as the Final one
• and so it picks the Target version

The reason for the missing Base version is: our Fleet package with prebuilt rules currently doesn't ship all historical versions of prebuilt rules. We're working with the TRADE team on fixing this (#187645, elastic/detection-rules#4150 (comment)) and this is a release blocker.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)