[Contextual Security] update latest vulnerability transform to create an alias for latest index

[alexreal1314]

Summary

This PR should handle this issue and related to data-stream changes in this PR.

Migrates the vulnerabilities transform destination from a direct index pattern to a versioned index with an alias, enabling future zero-downtime migrations and preventing breaking changes.

• Added CDR_LATEST_NATIVE_VULNERABILITIES_INDEX_ALIAS: logs-cloud_security_posture.vulnerabilities_latest-default (backward-compatible alias)
• Added CDR_LATEST_NATIVE_VULNERABILITIES_INDEX_LATEST: security_solution-cloud_security_posture.vulnerability_latest-v1 (versioned physical index)
• Added CDR_LATEST_NATIVE_VULNERABILITIES_INDEX_PATTERN_V1: security_solution-cloud_security_posture.vulnerability_latest-* (template pattern)
• Updated CDR_VULNERABILITIES_INDEX_PATTERN to use the alias for queries

Transform Updates

• Bumped version from 8.15.0 to 8.16.0
  Updated destination index to versioned pattern: security_solution-cloud_security_posture.vulnerability_latest-v1
  Added previous version to DEPRECATED_VULN_TRANSFORM_VERSIONS
• Index & Alias Management (create_transforms.ts)
• Added createVulnerabilitiesIndexAlias() function that:
  • Creates the physical index if it doesn't exist
  • Creates alias pointing old index name → new versioned index
• Ensures backward compatibility with existing queries

Benefits
✅ Backward Compatibility: Existing queries using logs-cloud_security_posture.vulnerabilities_latest-default continue to work
✅ Future-Proof: Can create v2, v3, etc. and update alias without breaking queries
✅ Zero Downtime: Index version changes won't disrupt running systems
✅ Consistent Pattern: Aligns with how misconfigurations index is handled

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
• Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Before upgrading the integration
since this PR introduces a breaking change in the vulnerability.published_date mapping few actions need to be taken to ensure smooth upgrade.
if data preservation is important in latest index follow this procedure to preserve data and maintain mapping compatibility:

// 1. Create temporary index
PUT logs-cloud_security_posture.vulnerabilities_latest-temp
{
  "settings": {
    "index.codec": "best_compression"
  }
}

// 2. Reindex all documents from current latest index to temp
POST _reindex
{
  "source": {
    "index": "logs-cloud_security_posture.vulnerabilities_latest-default"
  },
  "dest": {
    "index": "logs-cloud_security_posture.vulnerabilities_latest-temp"
  }
}

// 3. Verify document count matches
GET logs-cloud_security_posture.vulnerabilities_latest-default/_count
GET logs-cloud_security_posture.vulnerabilities_latest-temp/_count

// 4. Delete the existing latest index (freeing up the name for alias)
DELETE logs-cloud_security_posture.vulnerabilities_latest-default

// 5. Upgrade the integration
// (This will create the new v1 index and alias)

// 6. Copy data from temp to new versioned index
POST _reindex
{
  "source": {
    "index": "logs-cloud_security_posture.vulnerabilities_latest-temp"
  },
  "dest": {
    "index": "security_solution-cloud_security_posture.vulnerability_latest-v1"
  }
}

// 7. Verify data in new index
GET security_solution-cloud_security_posture.vulnerability_latest-v1/_count
GET logs-cloud_security_posture.vulnerabilities_latest-default/_count  // via alias

// 8. Delete temporary index
DELETE logs-cloud_security_posture.vulnerabilities_latest-temp

Why This Approach?

• ✅ Preserves all existing vulnerability data
• ✅ Maintains mapping compatibility (reindex copies mappings)
• ✅ No downtime (temp index holds data during migration)
• ✅ Rollback possible (temp index can be kept as backup)

Alternative: Fresh Start (Data Loss)

If data preservation is not required:

// Simply delete the existing index
DELETE logs-cloud_security_posture.vulnerabilities_latest-default

new index will be recreated with new mappings.

Screen recording

Screen.Recording.2025-11-04.at.9.32.53.mov

[elasticmachine]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!
• Click to trigger kibana-deploy-project-from-pr for this PR!
• Click to trigger kibana-deploy-cloud-from-pr for this PR!

[kibanamachine]

Cloud deployments require a Github label, please add ci:cloud-deploy or ci:cloud-redeploy and trigger the job through the checkbox again.

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 75fb446
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-241700-75fb446aef4a

Failed CI Steps

• FTR Configs #18
• FTR Configs #18
• FTR Configs #87
• FTR Configs #87

Test Failures

• [job] [logs] FTR Configs #87 / Core Analysis - Entity Store @ess @skipInServerlessMKI Entity Store APIs get and list "before all" hook in "get and list"
• [job] [logs] FTR Configs #87 / Core Analysis - Entity Store @ess @skipInServerlessMKI Entity Store APIs get and list "before all" hook in "get and list"

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/cloud-security-posture-common	218	221	+3

Unknown metric groups

API count

id	before	after	diff
@kbn/cloud-security-posture-common	220	223	+3

History

• 💔 Build #357868 failed bf2d022