Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Meta][Amazon Security Lake] Supporting OCSF v1.1 #9607

Open
5 of 7 tasks
jamiehynds opened this issue Apr 16, 2024 · 4 comments
Open
5 of 7 tasks

[Meta][Amazon Security Lake] Supporting OCSF v1.1 #9607

jamiehynds opened this issue Apr 16, 2024 · 4 comments
Assignees
Labels
8.15 candidate enhancement New feature or request Epic Integration:amazon_security_lake Amazon Security Lake meta Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations]

Comments

@jamiehynds
Copy link

jamiehynds commented Apr 16, 2024

Our current Amazon Security Lake supports OCSF v1.0, which was the latest version of the schema when we initially shipped the integration. The OCSF schema has evolved since, and is now at v1.1.

Our Security Lake pipelines need to be adjusted to ensure we're inline with v1.1, including new event classes, objects and categories. Backward compatibility is not a significant concern in this case, as an example our security findings pipeline can be deprecated, as security findings were deprecated in OCSF v1.1. Related dashboards will also need to be removed, and new ones added to account for new classes introduced in v1.1

Once we have updated pipelines to support the latest OCSF version, we'll create an issue to build a generic OCSF to ECS package, for users who would like to ingest OCSF formatted data outside of Security Lake.

For a full list of v1.1 changes we need to adhere to, please see here: https://github.com/ocsf/ocsf-schema/blob/main/CHANGELOG.md#v110---january-25th-2024

Update: OCSF v1.2 is now available - https://github.com/ocsf/ocsf-schema/blob/main/CHANGELOG.md#v120---april-23rd-2024 - will let @ShourieG decide if we make the necessary changes for both 1.1 and 1.2 in this issue, or create a separate isseu for v1.2

Tasks

  1. Integration:amazon_security_lake Team:Security-Service Integrations enhancement
    ShourieG
  2. Integration:amazon_security_lake Team:Security-Service Integrations enhancement integration
    ShourieG
  3. Team:Fleet enhancement
  4. Team:Ecosystem enhancement
  5. Integration:amazon_security_lake Team:Security-Service Integrations enhancement integration
    ShourieG
@jamiehynds jamiehynds added the Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] label Apr 16, 2024
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@jamiehynds jamiehynds added Integration:amazon_security_lake Amazon Security Lake enhancement New feature or request 8.15 candidate labels Apr 16, 2024
@ShourieG ShourieG self-assigned this Apr 17, 2024
@jamiehynds jamiehynds added the Epic label May 2, 2024
@ShourieG
Copy link
Contributor

ShourieG commented May 6, 2024

@jamiehynds, as discussed we are initially starting the upgrade to OCSF v1.1 and targeting the release by 8.15. Post the upgrade to v1.1 we will start with the upgrades for supporting v1.2.

@ShourieG
Copy link
Contributor

ShourieG commented Jun 6, 2024

We are hitting the upper limit of 2048 fields per data stream while performing the ocsf v1.1 upgrade due to addition of new object types to base objects. I've discussed with the ecosystem team and an issue has been opened for addressing this limitation in future. Right now we might have to keep these new objects in a flattened state to bypass these limitations. cc: @jamiehynds @andrewkroh

@ShourieG
Copy link
Contributor

@narph @jamiehynds, PR is now merged. The issue could still be kept open since there are couple of related issues which are yet to be solved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
8.15 candidate enhancement New feature or request Epic Integration:amazon_security_lake Amazon Security Lake meta Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations]
Projects
None yet
Development

No branches or pull requests

4 participants