diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 999646eac74..ccbb59ed666 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -474,6 +474,7 @@ /packages/ti_domaintools @elastic/security-service-integrations /packages/ti_eclecticiq @elastic/security-service-integrations /packages/ti_eset @elastic/security-service-integrations +/packages/ti_flashpoint @elastic/security-service-integrations /packages/ti_google_threat_intelligence @elastic/security-service-integrations /packages/ti_greynoise @elastic/security-service-integrations /packages/ti_maltiverse @elastic/security-service-integrations diff --git a/packages/ti_flashpoint/_dev/build/build.yml b/packages/ti_flashpoint/_dev/build/build.yml new file mode 100644 index 00000000000..b2596b96490 --- /dev/null +++ b/packages/ti_flashpoint/_dev/build/build.yml @@ -0,0 +1,3 @@ +dependencies: + ecs: + reference: git@v9.2.0 diff --git a/packages/ti_flashpoint/_dev/build/docs/README.md b/packages/ti_flashpoint/_dev/build/docs/README.md new file mode 100644 index 00000000000..c574791f33f --- /dev/null +++ b/packages/ti_flashpoint/_dev/build/docs/README.md @@ -0,0 +1,109 @@ +# Flashpoint Integration for Elastic + +## Overview + +[Flashpoint](https://flashpoint.io/) is a comprehensive threat intelligence platform that delivers actionable insights from dark web, deep web, and technical sources. It combines human-curated intelligence with automated collection to help organizations identify emerging threats, monitor adversary activity, and assess cyber risk with enriched context. + +The Flashpoint integration for Elastic collects alerts from the **Flashpoint Ignite API** and visualizes them in Kibana. + +### Compatibility + +The Flashpoint integration is compatible with Ignite API version **1.2**. + +### How it works + +This integration periodically queries the Flashpoint Ignite API to retrieve logs. + +## What data does this integration collect? + +This integration collects log messages of the following type: + +- `Alert`: Collects `alert` logs from the Flashpoint Ignite API (endpoint: `/alert-management/v1/notifications`), + +### Supported use cases + +Integrating Flashpoint with Elastic SIEM provides centralized visibility into threat intelligence alerts. Kibana dashboards present key metrics such as `Total Alerts`, along with visualizations showing `Alerts by Data Type`, `Source`, and `Origin`. + +`Alert Trends over Time`, `Top Authors`, `MIME Types`, `Alert Sources`, and `Geographic Distribution` of related resources help analysts quickly monitor activity and investigate alerts. These insights support efficient threat monitoring and analysis workflows. + +## What do I need to use this integration? + +### From Flashpoint + +To collect data through the Flashpoint Ignite API, you need to provide an **API Token**. Authentication is handled using the **API Token**, which serves as the required credential. + +#### Retrieve an API Token: + +1. Log in to the **Flashpoint** Instance. +2. Click on your profile icon in the top-right corner and select **Manage API Tokens**. +3. Click **Generate Token**. +4. Enter a name for the API token and click **Generate Token**. +5. Copy and securely store the generated API token for use in the integration configuration. + +## How do I deploy this integration? + +This integration supports both Elastic Agentless-based and Agent-based installations. + +### Agentless-based installation + +Agentless integrations allow you to collect data without having to manage Elastic Agent in your cloud. They make manual agent deployment unnecessary, so you can focus on your data instead of the agent that collects it. For more information, refer to [Agentless integrations](https://www.elastic.co/guide/en/serverless/current/security-agentless-integrations.html) and the [Agentless integrations FAQ](https://www.elastic.co/guide/en/serverless/current/agentless-integration-troubleshooting.html). + +Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. This functionality is in beta and is subject to change. Beta features are not subject to the support SLA of official GA features. + +### Agent-based installation + +Elastic Agent must be installed. For more details, check the Elastic Agent [installation instructions](docs-content://reference/fleet/install-elastic-agents.md). You can install only one Elastic Agent per host. + + +### configure + +1. In the top search bar in Kibana, search for **Integrations**. +2. In the search bar, type **Flashpoint**. +3. Select the **Flashpoint** integration from the search results. +4. Select **Add Flashpoint** to add the integration. +5. Enable and configure only the collection methods which you will use. + + * To **Collect logs from Flashpoint API**, you'll need to: + + - Configure **API Token**. + - Adjust the integration configuration parameters if required, including the **Initial Interval**, **Interval**, **Page Size** etc. to enable data collection. + +6. Select **Save and continue** to save the integration. + +### Validation + +#### Dashboard populated + +1. In the top search bar in Kibana, search for **Dashboards**. +2. In the search bar, type **Flashpoint**, and verify the dashboard information is populated. + +## Performance and scaling + +For more information on architectures that can be used for scaling this integration, check the [Ingest Architectures](https://www.elastic.co/docs/manage-data/ingest/ingest-reference-architectures) documentation. + +## Reference + +### ECS field reference + +#### Alert + +{{fields "alert"}} + +### Example event + +#### Alert + +{{event "alert"}} + + +### Inputs used + +These input is used in the integration: + +- [cel](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-cel) + +### API usage + +This integration dataset uses the following API: + +* List Alerts (endpoint: `/alert-management/v1/notifications`)| diff --git a/packages/ti_flashpoint/_dev/deploy/docker/docker-compose.yml b/packages/ti_flashpoint/_dev/deploy/docker/docker-compose.yml new file mode 100644 index 00000000000..13f0d327c6a --- /dev/null +++ b/packages/ti_flashpoint/_dev/deploy/docker/docker-compose.yml @@ -0,0 +1,15 @@ +version: '3.8' +services: + ti_flashpoint: + image: docker.elastic.co/observability/stream:v0.20.0 + hostname: ti_flashpoint + ports: + - 8090 + volumes: + - ./files:/files:ro + environment: + PORT: '8090' + command: + - http-server + - --addr=:8090 + - --config=/files/config.yml diff --git a/packages/ti_flashpoint/_dev/deploy/docker/files/config.yml b/packages/ti_flashpoint/_dev/deploy/docker/files/config.yml new file mode 100644 index 00000000000..2ff1d6850de --- /dev/null +++ b/packages/ti_flashpoint/_dev/deploy/docker/files/config.yml @@ -0,0 +1,510 @@ +rules: + - path: /alert-management/v1/notifications + methods: ['GET'] + query_params: + size: "2" + created_after: "{created_after:.*}" + cursor: "1766048799.045972" + request_headers: + Authorization: "Bearer xxxx" + responses: + - status_code: 200 + headers: + Content-Type: + - 'application/json' + body: |- + { + "items": [ + { + "id": "392014d7-88ed-4f76-8410-305d38d3a1bc", + "resource": { + "id": "VDUKzAG2WOWkGefHoe4TFA", + "basetypes": [ + "article", + "news", + "newscatcher" + ], + "author": "up18news", + "authors": [ + "up18news" + ], + "country": "IN", + "description": "Up18 News Wittypen Launches Zerply.ai and Wins Silver at MarTechAI Awards Business News", + "created_at": { + "date-time": "2025-12-17T06:46:29+00:00", + "raw": "2025-12-17 06:46:29", + "timestamp": 1765953989 + }, + "link": "https://example.com", + "site": { + "title": "example.com" + }, + "sort_date": "2025-12-17T06:46:33Z", + "title": "Wittypen Launches Zerply.ai and Wins Silver at MarTechAI Awards" + }, + "reason": { + "id": "0f325ca9-a28f-421b-8d0c-db3329366e5f", + "name": "credit", + "text": "credit", + "origin": "searches", + "details": { + "sources": [ + "communities", + "media", + "marketplaces", + "news" + ], + "params": { + "exclude": {}, + "include": { + "date": { + "end": "now", + "label": "Last 30 Days", + "start": "now-30d" + } + } + } + }, + "entity": null + }, + "status": null, + "generated_at": "2025-12-17T06:54:21.863351Z", + "created_at": "2025-12-17T06:54:26.163659Z", + "tags": {}, + "highlights": { + "body.text/plain": [ + "No credit card" + ] + }, + "highlight_text": "No credit card is require", + "data_type": "news", + "parent_data_type": null, + "source": "news", + "is_read": false + }, + { + "id": "fab83d8c-3fc3-4267-ad71-1df7be82b7bf", + "resource": { + "id": "TIaevTSpXQGxKlMDuh9RHw", + "basetypes": [ + "chat", + "conversation", + "message", + "telegram" + ], + "container": { + "name": "My Tricky Deals | Official", + "native_id": "1117587920", + "title": "My Tricky Deals | Official" + }, + "created_at": { + "date-time": "2025-12-17T06:53:09+00:00", + "raw": "1765954389", + "timestamp": 1765954389 + }, + "site": { + "title": "Telegram" + }, + "site_actor": { + "names": { + "handle": "My Tricky Deals | Official" + }, + "native_id": "1117587920" + }, + "sort_date": "2025-12-17T06:53:09Z", + "title": "My Tricky Deals | Official" + }, + "reason": { + "id": "0f325ca9-a28f-421b-8d0c-db3329366e5f", + "name": "credit", + "text": "credit", + "origin": "searches", + "details": { + "sources": [ + "communities", + "media", + "marketplaces", + "news" + ], + "params": { + "exclude": {}, + "include": { + "date": { + "end": "now", + "label": "Last 30 Days", + "start": "now-30d" + } + } + } + }, + "entity": null + }, + "status": null, + "generated_at": "2025-12-17T06:54:14.756666Z", + "created_at": "2025-12-17T06:54:15.576748Z", + "tags": {}, + "highlights": { + "body.text/plain": [ + "Pureit Eco Water Save" + ] + }, + "highlight_text": "Pureit Eco Water Saver R", + "data_type": "chat", + "parent_data_type": null, + "source": "communities", + "is_read": false + } + ], + "pagination": { + "next": null, + "prev": null, + "total": null + } + } + - path: /alert-management/v1/notifications + methods: ['GET'] + query_params: + size: "2" + created_after: "{created_after:.*}" + cursor: "1766048799.045962" + request_headers: + Authorization: "Bearer xxxx" + responses: + - status_code: 200 + headers: + Content-Type: + - 'application/json' + body: |- + { + "items": [ + { + "id": "04d5f5ca-4c7f-40b2-b2f1-91a08acc2a0b", + "resource": { + "id": "BQoWAAO5X0uRdZgRAFUJMg", + "basetypes": [ + "article", + "comment", + "conversation", + "message", + "reddit", + "web" + ], + "container": { + "container": { + "name": "walmart_RX", + "native_id": "walmart_RX" + }, + "native_id": "1pklksl", + "title": "Why do patients get so upset when they have to go to the counseling window???" + }, + "created_at": { + "date-time": "2025-12-17T04:08:07+00:00", + "raw": "2025-12-17 04:08:07+00:00", + "timestamp": 1765944487 + }, + "site": { + "title": "Reddit" + }, + "site_actor": { + "names": { + "handle": "Jitae_Slay" + }, + "native_id": "Jitae_Slay" + }, + "section": "walmart_RX", + "sort_date": "2025-12-17T04:08:07Z", + "title": "Why do patients get so upset when they have to go to the counseling window???" + }, + "reason": { + "id": "0f325ca9-a28f-421b-8d0c-db3329366e5f", + "name": "credit", + "text": "credit", + "origin": "searches", + "details": { + "sources": [ + "communities", + "media", + "marketplaces", + "news" + ], + "params": { + "exclude": {}, + "include": { + "date": { + "end": "now", + "label": "Last 30 Days", + "start": "now-30d" + } + } + } + }, + "entity": null + }, + "status": null, + "generated_at": "2025-12-17T06:54:12.513737Z", + "created_at": "2025-12-17T06:54:13.809007Z", + "tags": {}, + "highlights": { + "body.text/plain": [ + "You developed by pharmacists." + ] + }, + "highlight_text": "You developed by pharmacists.", + "data_type": "reddit", + "parent_data_type": null, + "source": "communities", + "is_read": false + }, + { + "id": "d361ebdf-29e9-4063-8168-ede87124704b", + "resource": { + "id": "m-xxur1VXm6W7hN0IU0rHA", + "basetypes": [ + "chat", + "conversation", + "message", + "telegram" + ], + "container": { + "name": "BinX Chat", + "native_id": "2783168214", + "title": "BinX Chat" + }, + "created_at": { + "date-time": "2025-12-17T06:53:40+00:00", + "raw": "1765954420", + "timestamp": 1765954420 + }, + "site": { + "title": "Telegram" + }, + "site_actor": { + "names": { + "handle": "BinX Bot" + }, + "native_id": "7329244255" + }, + "sort_date": "2025-12-17T06:53:40Z", + "title": "BinX Chat" + }, + "reason": { + "id": "0f325ca9-a28f-421b-8d0c-db3329366e5f", + "name": "credit", + "text": "credit", + "origin": "searches", + "details": { + "sources": [ + "communities", + "media", + "marketplaces", + "news" + ], + "params": { + "exclude": {}, + "include": { + "date": { + "end": "now", + "label": "Last 30 Days", + "start": "now-30d" + } + } + } + }, + "entity": null + }, + "status": null, + "generated_at": "2025-12-17T06:54:05.410755Z", + "created_at": "2025-12-17T06:54:08.483601Z", + "tags": {}, + "highlights": { + "body.text/plain": [ + "BinX Bin Lookup " + ] + }, + "highlight_text": "BinX Bin Lookup", + "data_type": "chat", + "parent_data_type": null, + "source": "communities", + "is_read": false + } + ], + "pagination": { + "next": "http://{{ hostname }}:{{ env "PORT" }}/alert-management/v1/notifications?created_after=2025-12-02T00%3A00%3A00Z&cursor=1766048799.045972&size=2", + "prev": null, + "total": null + } + } + - path: /alert-management/v1/notifications + methods: ['GET'] + query_params: + size: "2" + created_after: "{created_after:.*}" + request_headers: + Authorization: "Bearer xxxx" + responses: + - status_code: 200 + headers: + Content-Type: + - 'application/json' + body: |- + { + "items": [ + { + "id": "e2af074a-970e-48a1-b34d-e63b573b8c2c", + "resource": { + "id": "zjtsfLGBWhGEWmNDICmOUQ", + "basetypes": [ + "chat", + "conversation", + "message", + "telegram" + ], + "container": { + "name": "Warnisx Scrapper", + "native_id": "3092489782", + "title": "Warnisx Scrapper" + }, + "created_at": { + "date-time": "2025-12-17T06:53:13+00:00", + "raw": "1765954393", + "timestamp": 1765954393 + }, + "site": { + "title": "Telegram" + }, + "site_actor": { + "names": { + "handle": "Warnisx Scrapper" + }, + "native_id": "3092489782" + }, + "sort_date": "2025-12-17T06:53:13Z", + "title": "Warnisx Scrapper" + }, + "reason": { + "id": "0f325ca9-a28f-421b-8d0c-db3329366e5f", + "name": "credit", + "text": "credit", + "origin": "searches", + "details": { + "sources": [ + "communities", + "media", + "marketplaces", + "news" + ], + "params": { + "exclude": {}, + "include": { + "date": { + "end": "now", + "label": "Last 30 Days", + "start": "now-30d" + } + } + } + }, + "entity": null + }, + "status": null, + "generated_at": "2025-12-17T06:54:02.025006Z", + "created_at": "2025-12-17T06:54:06.495138Z", + "tags": {}, + "highlights": { + "body.text/plain": [ + "temp" + ] + }, + "highlight_text": "temp", + "data_type": "chat", + "parent_data_type": null, + "source": "communities", + "is_read": false + }, + { + "id": "61e89ab1-bd39-459e-84b5-79023b721c7d", + "resource": { + "id": "SrcBxTtDV-O5U-ycJ_Cv_g", + "basetypes": [ + "chan", + "comment", + "conversation", + "message", + "web" + ], + "container": { + "container": { + "native_id": "a", + "title": "/a/ - Anime & Manga" + }, + "native_id": "284504102" + }, + "created_at": { + "date-time": "2025-12-15T11:43:32+00:00", + "raw": "2025-12-15 11:43:32+00:00", + "timestamp": 1765799012 + }, + "site": { + "title": "4chan" + }, + "site_actor": { + "names": { + "handle": "Anonymous (None)" + }, + "native_id": "Anonymous" + }, + "section": "/a/ - Anime & Manga", + "sort_date": "2025-12-15T11:43:32Z", + "title": "284504102" + }, + "reason": { + "id": "61e526db-8554-42bb-b590-e1f6a37cd4fb", + "name": "temp", + "text": "temp", + "origin": "searches", + "details": { + "sources": [ + "communities", + "media", + "marketplaces", + "news" + ], + "params": { + "exclude": {}, + "include": { + "date": { + "end": "now", + "label": "Last 30 Days", + "start": "now-30d" + } + } + } + }, + "entity": null + }, + "status": null, + "generated_at": "2025-12-17T06:53:57.570160Z", + "created_at": "2025-12-17T06:53:59.956272Z", + "tags": {}, + "highlights": { + "site_actor.names.aliases": [ + "Anonymous (None)", + "Anonymous (None)" + ], + "body.text/plain": [ + ">>28450" + ] + }, + "highlight_text": ">>28", + "data_type": "board", + "parent_data_type": null, + "source": "communities", + "is_read": false + } + ], + "pagination": { + "next": "http://{{ hostname }}:{{ env "PORT" }}/alert-management/v1/notifications?created_after=2025-12-02T00%3A00%3A00Z&cursor=1766048799.045962&size=2", + "prev": null, + "total": null + } + } diff --git a/packages/ti_flashpoint/changelog.yml b/packages/ti_flashpoint/changelog.yml new file mode 100644 index 00000000000..35fb1d2c290 --- /dev/null +++ b/packages/ti_flashpoint/changelog.yml @@ -0,0 +1,6 @@ +# newer versions go on top +- version: 0.1.0 + changes: + - description: Initial release. + type: enhancement + link: https://github.com/elastic/integrations/pull/1 diff --git a/packages/ti_flashpoint/data_stream/alert/_dev/test/pipeline/test-alert.log b/packages/ti_flashpoint/data_stream/alert/_dev/test/pipeline/test-alert.log new file mode 100644 index 00000000000..2083a6710a7 --- /dev/null +++ b/packages/ti_flashpoint/data_stream/alert/_dev/test/pipeline/test-alert.log @@ -0,0 +1,3 @@ +{"id":"30ac772b-0a8a-4f97-a146-4ce7284d1279","resource":{"id":"Qy-6bshmVX-pp4m4ZsMe5w","basetypes":["chat","conversation","message","telegram"],"container":{"name":"JOBS IN MALAYSIA","native_id":"1504998274","title":"JOBS IN MALAYSIA"},"created_at":{"date-time":"2025-12-12T10:36:53+00:00","raw":"1765535813","timestamp":1765535813},"media_v2":[{"media_type":"document","mime_type":"video/mp4"}],"site":{"title":"Telegram"},"site_actor":{"names":{"handle":"ameliazahra"},"native_id":"6011042960"},"sort_date":"2025-12-12T10:36:53Z","title":"JOBS IN MALAYSIA"},"reason":{"id":"0f325ca9-a28f-421b-8d0c-db3329366e5f","name":"credit","text":"credit","origin":"searches","details":{"sources":["communities","media","marketplaces","news"],"params":{"exclude":{},"include":{"date":{"end":"now","label":"Last 30 Days","start":"now-30d"}}}},"entity":null},"status":null,"generated_at":"2025-12-12T11:41:37.914873Z","created_at":"2025-12-12T11:41:41.035810Z","tags":{},"highlights":{"body.text/plain":["example text"]},"highlight_text":"example text","data_type":"chat","parent_data_type":null,"source":"communities","is_read":false} +{"id":"3057cbdf-1b7e-4267-9fed-a610300f702e","resource":{"id":"775CQYbcWw6JsQB13vhwKQ","basetypes":["article","news","newscatcher"],"author":"John Doe","authors":["John Doe"],"country":"IN","description":"Superstar Rajinikanth celebrated his 75th birthday today on the set of the movie Jailer 2. Photos from the celebration are now going viral on the internet.","created_at":{"date-time":"2025-12-12T08:13:15+00:00","raw":"2025-12-12 08:13:15","timestamp":1765527195},"link":"https://example.com","site":{"title":"example.com"},"sort_date":"2025-12-12T08:13:15Z","title":"Rajinikanth Turns 75! Birthday Celebration on Jailer 2 Set Trends Online"},"reason":{"id":"0f325ca9-a28f-421b-8d0c-db3329366e5f","name":"credit","text":"credit","origin":"searches","details":{"sources":["communities","media","marketplaces","news"],"params":{"exclude":{},"include":{"date":{"end":"now","label":"Last 30 Days","start":"now-30d"}}}},"entity":null},"status":null,"generated_at":"2025-12-12T11:41:34.900665Z","created_at":"2025-12-12T11:41:37.775308Z","tags":{},"highlights":{"body.text/plain":["Photos are viral."]},"highlight_text":"Photos are viral.","data_type":"news","parent_data_type":null,"source":"news","is_read":false} +{"id":"1fc0fc04-5a85-41ff-970d-acda1c0dd390","resource":{"id":"c4FF4mbNV7m2vBWPIaV0uw","basetypes":["article","news","newscatcher"],"author":"Alias Doe","authors":["Alias Doe"],"country":"IN","description":"Rajinikanth, who has ruled both South and Bollywood cinema, turns 75 today. Born on December 12, 1950, the superstar continues to deliver box-office blockbusters and is now gearing up for several exciting upcoming films.","created_at":{"date-time":"2025-12-12T09:06:06+00:00","raw":"2025-12-12 09:06:06","timestamp":1765530366},"link":"https://example.com","site":{"title":"example.com"},"sort_date":"2025-12-12T09:06:06Z","title":"Jailer 2 to Coolie 2: Rajinikanth at 75 Ready to Thrill Fans with Upcoming Hits"},"reason":{"id":"0f325ca9-a28f-421b-8d0c-db3329366e5f","name":"credit","text":"credit","origin":"searches","details":{"sources":["communities","media","marketplaces","news"],"params":{"exclude":{},"include":{"date":{"end":"now","label":"Last 30 Days","start":"now-30d"}}}},"entity":null},"status":null,"generated_at":"2025-12-12T11:41:35.896396Z","created_at":"2025-12-12T11:41:37.775301Z","tags":{},"highlights":{"body.text/plain":["16\nImage Credit."]},"highlight_text":"16\nImage Credit :","data_type":"news","parent_data_type":null,"source":"news","is_read":false} diff --git a/packages/ti_flashpoint/data_stream/alert/_dev/test/pipeline/test-alert.log-expected.json b/packages/ti_flashpoint/data_stream/alert/_dev/test/pipeline/test-alert.log-expected.json new file mode 100644 index 00000000000..2d19deff027 --- /dev/null +++ b/packages/ti_flashpoint/data_stream/alert/_dev/test/pipeline/test-alert.log-expected.json @@ -0,0 +1,270 @@ +{ + "expected": [ + { + "@timestamp": "2025-12-12T11:41:41.035Z", + "container": { + "id": "1504998274", + "name": "JOBS IN MALAYSIA" + }, + "ecs": { + "version": "9.2.0" + }, + "event": { + "created": "2025-12-12T11:41:41.035Z", + "id": "30ac772b-0a8a-4f97-a146-4ce7284d1279", + "kind": "alert", + "original": "{\"id\":\"30ac772b-0a8a-4f97-a146-4ce7284d1279\",\"resource\":{\"id\":\"Qy-6bshmVX-pp4m4ZsMe5w\",\"basetypes\":[\"chat\",\"conversation\",\"message\",\"telegram\"],\"container\":{\"name\":\"JOBS IN MALAYSIA\",\"native_id\":\"1504998274\",\"title\":\"JOBS IN MALAYSIA\"},\"created_at\":{\"date-time\":\"2025-12-12T10:36:53+00:00\",\"raw\":\"1765535813\",\"timestamp\":1765535813},\"media_v2\":[{\"media_type\":\"document\",\"mime_type\":\"video/mp4\"}],\"site\":{\"title\":\"Telegram\"},\"site_actor\":{\"names\":{\"handle\":\"ameliazahra\"},\"native_id\":\"6011042960\"},\"sort_date\":\"2025-12-12T10:36:53Z\",\"title\":\"JOBS IN MALAYSIA\"},\"reason\":{\"id\":\"0f325ca9-a28f-421b-8d0c-db3329366e5f\",\"name\":\"credit\",\"text\":\"credit\",\"origin\":\"searches\",\"details\":{\"sources\":[\"communities\",\"media\",\"marketplaces\",\"news\"],\"params\":{\"exclude\":{},\"include\":{\"date\":{\"end\":\"now\",\"label\":\"Last 30 Days\",\"start\":\"now-30d\"}}}},\"entity\":null},\"status\":null,\"generated_at\":\"2025-12-12T11:41:37.914873Z\",\"created_at\":\"2025-12-12T11:41:41.035810Z\",\"tags\":{},\"highlights\":{\"body.text/plain\":[\"example text\"]},\"highlight_text\":\"example text\",\"data_type\":\"chat\",\"parent_data_type\":null,\"source\":\"communities\",\"is_read\":false}", + "reason": "credit" + }, + "file": { + "mime_type": [ + "video/mp4" + ] + }, + "tags": [ + "preserve_duplicate_custom_fields" + ], + "ti_flashpoint": { + "alert": { + "created_at": "2025-12-12T11:41:41.035Z", + "data_type": "chat", + "generated_at": "2025-12-12T11:41:37.914Z", + "highlight_text": "example text", + "highlights": { + "body.text/plain": [ + "example text" + ] + }, + "id": "30ac772b-0a8a-4f97-a146-4ce7284d1279", + "is_read": false, + "reason": { + "details": { + "params": { + "include": { + "date": { + "end": "now", + "label": "Last 30 Days", + "start": "now-30d" + } + } + }, + "sources": [ + "communities", + "media", + "marketplaces", + "news" + ] + }, + "id": "0f325ca9-a28f-421b-8d0c-db3329366e5f", + "name": "credit", + "origin": "searches", + "text": "credit" + }, + "resource": { + "basetypes": [ + "chat", + "conversation", + "message", + "telegram" + ], + "container": { + "name": "JOBS IN MALAYSIA", + "native_id": "1504998274", + "title": "JOBS IN MALAYSIA" + }, + "created_at": "2025-12-12T10:36:53.000Z", + "id": "Qy-6bshmVX-pp4m4ZsMe5w", + "media_v2": [ + { + "media_type": "document", + "mime_type": "video/mp4" + } + ], + "site": { + "title": "Telegram" + }, + "site_actor": { + "names": { + "handle": "ameliazahra" + }, + "native_id": "6011042960" + }, + "sort_date": "2025-12-12T10:36:53.000Z", + "title": "JOBS IN MALAYSIA" + }, + "source": "communities" + } + } + }, + { + "@timestamp": "2025-12-12T11:41:37.775Z", + "ecs": { + "version": "9.2.0" + }, + "event": { + "created": "2025-12-12T11:41:37.775Z", + "id": "3057cbdf-1b7e-4267-9fed-a610300f702e", + "kind": "alert", + "original": "{\"id\":\"3057cbdf-1b7e-4267-9fed-a610300f702e\",\"resource\":{\"id\":\"775CQYbcWw6JsQB13vhwKQ\",\"basetypes\":[\"article\",\"news\",\"newscatcher\"],\"author\":\"John Doe\",\"authors\":[\"John Doe\"],\"country\":\"IN\",\"description\":\"Superstar Rajinikanth celebrated his 75th birthday today on the set of the movie Jailer 2. Photos from the celebration are now going viral on the internet.\",\"created_at\":{\"date-time\":\"2025-12-12T08:13:15+00:00\",\"raw\":\"2025-12-12 08:13:15\",\"timestamp\":1765527195},\"link\":\"https://example.com\",\"site\":{\"title\":\"example.com\"},\"sort_date\":\"2025-12-12T08:13:15Z\",\"title\":\"Rajinikanth Turns 75! Birthday Celebration on Jailer 2 Set Trends Online\"},\"reason\":{\"id\":\"0f325ca9-a28f-421b-8d0c-db3329366e5f\",\"name\":\"credit\",\"text\":\"credit\",\"origin\":\"searches\",\"details\":{\"sources\":[\"communities\",\"media\",\"marketplaces\",\"news\"],\"params\":{\"exclude\":{},\"include\":{\"date\":{\"end\":\"now\",\"label\":\"Last 30 Days\",\"start\":\"now-30d\"}}}},\"entity\":null},\"status\":null,\"generated_at\":\"2025-12-12T11:41:34.900665Z\",\"created_at\":\"2025-12-12T11:41:37.775308Z\",\"tags\":{},\"highlights\":{\"body.text/plain\":[\"Photos are viral.\"]},\"highlight_text\":\"Photos are viral.\",\"data_type\":\"news\",\"parent_data_type\":null,\"source\":\"news\",\"is_read\":false}", + "reason": "credit", + "url": "https://example.com" + }, + "related": { + "user": [ + "John Doe" + ] + }, + "tags": [ + "preserve_duplicate_custom_fields" + ], + "ti_flashpoint": { + "alert": { + "created_at": "2025-12-12T11:41:37.775Z", + "data_type": "news", + "generated_at": "2025-12-12T11:41:34.900Z", + "highlight_text": "Photos are viral.", + "highlights": { + "body.text/plain": [ + "Photos are viral." + ] + }, + "id": "3057cbdf-1b7e-4267-9fed-a610300f702e", + "is_read": false, + "reason": { + "details": { + "params": { + "include": { + "date": { + "end": "now", + "label": "Last 30 Days", + "start": "now-30d" + } + } + }, + "sources": [ + "communities", + "media", + "marketplaces", + "news" + ] + }, + "id": "0f325ca9-a28f-421b-8d0c-db3329366e5f", + "name": "credit", + "origin": "searches", + "text": "credit" + }, + "resource": { + "author": "John Doe", + "authors": [ + "John Doe" + ], + "basetypes": [ + "article", + "news", + "newscatcher" + ], + "country": "IN", + "created_at": "2025-12-12T08:13:15.000Z", + "description": "Superstar Rajinikanth celebrated his 75th birthday today on the set of the movie Jailer 2. Photos from the celebration are now going viral on the internet.", + "id": "775CQYbcWw6JsQB13vhwKQ", + "link": "https://example.com", + "site": { + "title": "example.com" + }, + "sort_date": "2025-12-12T08:13:15.000Z", + "title": "Rajinikanth Turns 75! Birthday Celebration on Jailer 2 Set Trends Online" + }, + "source": "news" + } + }, + "user": { + "name": "John Doe" + } + }, + { + "@timestamp": "2025-12-12T11:41:37.775Z", + "ecs": { + "version": "9.2.0" + }, + "event": { + "created": "2025-12-12T11:41:37.775Z", + "id": "1fc0fc04-5a85-41ff-970d-acda1c0dd390", + "kind": "alert", + "original": "{\"id\":\"1fc0fc04-5a85-41ff-970d-acda1c0dd390\",\"resource\":{\"id\":\"c4FF4mbNV7m2vBWPIaV0uw\",\"basetypes\":[\"article\",\"news\",\"newscatcher\"],\"author\":\"Alias Doe\",\"authors\":[\"Alias Doe\"],\"country\":\"IN\",\"description\":\"Rajinikanth, who has ruled both South and Bollywood cinema, turns 75 today. Born on December 12, 1950, the superstar continues to deliver box-office blockbusters and is now gearing up for several exciting upcoming films.\",\"created_at\":{\"date-time\":\"2025-12-12T09:06:06+00:00\",\"raw\":\"2025-12-12 09:06:06\",\"timestamp\":1765530366},\"link\":\"https://example.com\",\"site\":{\"title\":\"example.com\"},\"sort_date\":\"2025-12-12T09:06:06Z\",\"title\":\"Jailer 2 to Coolie 2: Rajinikanth at 75 Ready to Thrill Fans with Upcoming Hits\"},\"reason\":{\"id\":\"0f325ca9-a28f-421b-8d0c-db3329366e5f\",\"name\":\"credit\",\"text\":\"credit\",\"origin\":\"searches\",\"details\":{\"sources\":[\"communities\",\"media\",\"marketplaces\",\"news\"],\"params\":{\"exclude\":{},\"include\":{\"date\":{\"end\":\"now\",\"label\":\"Last 30 Days\",\"start\":\"now-30d\"}}}},\"entity\":null},\"status\":null,\"generated_at\":\"2025-12-12T11:41:35.896396Z\",\"created_at\":\"2025-12-12T11:41:37.775301Z\",\"tags\":{},\"highlights\":{\"body.text/plain\":[\"16\\nImage Credit.\"]},\"highlight_text\":\"16\\nImage Credit :\",\"data_type\":\"news\",\"parent_data_type\":null,\"source\":\"news\",\"is_read\":false}", + "reason": "credit", + "url": "https://example.com" + }, + "related": { + "user": [ + "Alias Doe" + ] + }, + "tags": [ + "preserve_duplicate_custom_fields" + ], + "ti_flashpoint": { + "alert": { + "created_at": "2025-12-12T11:41:37.775Z", + "data_type": "news", + "generated_at": "2025-12-12T11:41:35.896Z", + "highlight_text": "16\nImage Credit :", + "highlights": { + "body.text/plain": [ + "16\nImage Credit." + ] + }, + "id": "1fc0fc04-5a85-41ff-970d-acda1c0dd390", + "is_read": false, + "reason": { + "details": { + "params": { + "include": { + "date": { + "end": "now", + "label": "Last 30 Days", + "start": "now-30d" + } + } + }, + "sources": [ + "communities", + "media", + "marketplaces", + "news" + ] + }, + "id": "0f325ca9-a28f-421b-8d0c-db3329366e5f", + "name": "credit", + "origin": "searches", + "text": "credit" + }, + "resource": { + "author": "Alias Doe", + "authors": [ + "Alias Doe" + ], + "basetypes": [ + "article", + "news", + "newscatcher" + ], + "country": "IN", + "created_at": "2025-12-12T09:06:06.000Z", + "description": "Rajinikanth, who has ruled both South and Bollywood cinema, turns 75 today. Born on December 12, 1950, the superstar continues to deliver box-office blockbusters and is now gearing up for several exciting upcoming films.", + "id": "c4FF4mbNV7m2vBWPIaV0uw", + "link": "https://example.com", + "site": { + "title": "example.com" + }, + "sort_date": "2025-12-12T09:06:06.000Z", + "title": "Jailer 2 to Coolie 2: Rajinikanth at 75 Ready to Thrill Fans with Upcoming Hits" + }, + "source": "news" + } + }, + "user": { + "name": "Alias Doe" + } + } + ] +} diff --git a/packages/ti_flashpoint/data_stream/alert/_dev/test/pipeline/test-common-config.yml b/packages/ti_flashpoint/data_stream/alert/_dev/test/pipeline/test-common-config.yml new file mode 100644 index 00000000000..37e8fa225fd --- /dev/null +++ b/packages/ti_flashpoint/data_stream/alert/_dev/test/pipeline/test-common-config.yml @@ -0,0 +1,3 @@ +fields: + tags: + - preserve_duplicate_custom_fields diff --git a/packages/ti_flashpoint/data_stream/alert/_dev/test/system/test-default-config.yml b/packages/ti_flashpoint/data_stream/alert/_dev/test/system/test-default-config.yml new file mode 100644 index 00000000000..6b62313e822 --- /dev/null +++ b/packages/ti_flashpoint/data_stream/alert/_dev/test/system/test-default-config.yml @@ -0,0 +1,12 @@ +input: cel +service: ti_flashpoint +vars: + url: http://{{Hostname}}:{{Port}} + api_token: xxxx +data_stream: + vars: + preserve_original_event: true + preserve_duplicate_custom_fields: true + page_size: 2 +assert: + hit_count: 6 diff --git a/packages/ti_flashpoint/data_stream/alert/agent/stream/cel.yml.hbs b/packages/ti_flashpoint/data_stream/alert/agent/stream/cel.yml.hbs new file mode 100644 index 00000000000..b84a873cc3f --- /dev/null +++ b/packages/ti_flashpoint/data_stream/alert/agent/stream/cel.yml.hbs @@ -0,0 +1,133 @@ +config_version: 2 +interval: {{interval}} +resource.tracer: + enabled: {{enable_request_tracer}} + filename: "../../logs/cel/http-request-trace-*.ndjson" + maxbackups: 5 +{{#if proxy_url}} +resource.proxy_url: {{proxy_url}} +{{/if}} +{{#if ssl}} +resource.ssl: {{ssl}} +{{/if}} +{{#if http_client_timeout}} +resource.timeout: {{http_client_timeout}} +{{/if}} +{{#if max_executions}} +max_executions: {{max_executions}} +{{/if}} +resource.url: {{url}} + +state: + initial_interval: {{initial_interval}} + page_size: {{page_size}} + api_token: {{api_token}} +redact: + fields: + - api_token +program: | + state.with( + request( + "GET", + state.?next.url.orValue( + state.url.trim_right("/") + "/alert-management/v1/notifications?" + { + "size": [string(state.page_size)], + "created_after": [state.?cursor.last_timestamp.orValue((now - duration(state.initial_interval)).format("2006-01-02T15:04:05")) + "Z"] + }.format_query() + ) + ).with({ + "Header": { + "Accept": ["application/json"], + "Authorization": ["Bearer " + state.api_token], + } + }).do_request().as(resp, resp.StatusCode == 200 ? + resp.Body.decode_json().as(body, + { + // Publish events + "events": ((has(body.items) && size(body.items) > 0) ? + body.items.map(e, { + "message": e.encode_json() + }) + : + // Added placeholder to ensure cursor gets stored + // This will be dropped later in the pipeline + [{"message": "empty_events_placeholder"}] + ), + "cursor": { + // 'max_created_at' tracks the maximum 'created_at' timestamp + // encountered across all pages within the current polling cycle. + ?"max_created_at": (has(body.items) && size(body.items) > 0) ? + (has(state.?cursor.max_created_at) ? + optional.of( + max([ + state.cursor.max_created_at.parse_time("2006-01-02T15:04:05"), + body.items.map(e, timestamp(e.created_at)).max() + ]).format("2006-01-02T15:04:05") + ) + : + optional.of(body.items.map(e, timestamp(e.created_at)).max().format("2006-01-02T15:04:05")) + ) + : + state.?cursor.max_created_at, + + // 'last_timestamp' is finalized only on the last page of pagination + // and is used as the starting point for the next scheduled run. + ?"last_timestamp": (has(body.?pagination.next) && body.pagination.next == null) ? + ((has(body.items) && size(body.items) > 0) ? + (has(state.?cursor.max_created_at) ? + optional.of( + max([ + state.cursor.max_created_at.parse_time("2006-01-02T15:04:05"), + body.items.map(e, timestamp(e.created_at)).max(), + ]).format("2006-01-02T15:04:05") + ) + : + optional.of(body.items.map(e, timestamp(e.created_at)).max().format("2006-01-02T15:04:05")) + ) + : + state.?cursor.max_created_at) + : + state.?cursor.last_timestamp + }, + "next": { + ?"url": has(body.?pagination.next) && body.pagination.next != null ? body.?pagination.next : optional.none() + }, + "want_more": has(body.?pagination.next) && body.pagination.next != null + } + ) + : + { + "events": { + "error": { + "code": string(resp.StatusCode), + "id": string(resp.Status), + "message": "GET " + state.url.trim_right("/") + "/alert-management/v1/notifications: " + ( + size(resp.Body) != 0 ? + string(resp.Body) + : + string(resp.Status) + ' (' + string(resp.StatusCode) + ')' + ), + }, + }, + "next": {}, + "want_more": false + } + ) + ) +tags: +{{#if preserve_original_event}} + - preserve_original_event +{{/if}} +{{#if preserve_duplicate_custom_fields}} + - preserve_duplicate_custom_fields +{{/if}} +{{#each tags as |tag|}} + - {{tag}} +{{/each}} +{{#contains "forwarded" tags}} +publisher_pipeline.disable_host: true +{{/contains}} +{{#if processors}} +processors: +{{processors}} +{{/if}} diff --git a/packages/ti_flashpoint/data_stream/alert/elasticsearch/ingest_pipeline/default.yml b/packages/ti_flashpoint/data_stream/alert/elasticsearch/ingest_pipeline/default.yml new file mode 100644 index 00000000000..ddef3b484d9 --- /dev/null +++ b/packages/ti_flashpoint/data_stream/alert/elasticsearch/ingest_pipeline/default.yml @@ -0,0 +1,513 @@ +--- +description: Pipeline for processing alert logs. +processors: + - drop: + tag: drop_empty_events_placeholder_1f3e6c12 + if: ctx.message == 'empty_events_placeholder' + - set: + tag: set_ecs_version_to_9_2_0_3273339c + field: ecs.version + value: 9.2.0 + - terminate: + description: error message set and no data to process. + tag: terminate_data_collection_error_4c75f12b + if: ctx.error?.message != null && ctx.message == null && ctx.event?.original == null + + # remove agentless metadata + - remove: + description: Removes the fields added by Agentless as metadata, as they can collide with ECS fields. + tag: remove_agentless_tags_44eed408 + if: ctx.organization instanceof String && ctx.division instanceof String && ctx.team instanceof String + field: + - organization + - division + - team + ignore_missing: true + + # parse the event JSON + - rename: + description: Renames the original `message` field to `event.original` to store a copy of the original message. The `event.original` field is not touched if the document already has one; it may happen when Logstash sends the document. + tag: rename_message_to_event_original_c74b1d7e + if: ctx.event?.original == null + field: message + target_field: event.original + ignore_missing: true + - remove: + description: The `message` field is no longer required if the document has an `event.original` field. + tag: remove_message_84808ee4 + if: ctx.event?.original != null + field: message + ignore_missing: true + - json: + tag: json_event_original_into_json_5e54dc16 + field: event.original + target_field: json + + # Add fingerprint + - fingerprint: + tag: fingerprint_into__id_29432682 + fields: + - json.id + target_field: _id + ignore_missing: true + + # Set event.* fields + - set: + tag: set_event_kind_to_alert_39295792 + field: event.kind + value: alert + + # rename fields to snake_case (hyphen to underscore) + - script: + description: Convert field names from hyphen to underscore. + tag: script_normalize_field_names_9f4e694c + lang: painless + source: |- + // Replace '-' with '_' in field names + String normalize(String str) { + return str.replace('-', '_'); + } + + // Recursive function to process objects + def normalizeFields(def obj) { + if (obj instanceof Map) { + def newObj = new HashMap(); + for (entry in obj.entrySet()) { + String newKey = normalize(entry.getKey()); + newObj.put(newKey, normalizeFields(entry.getValue())); + } + return newObj; + } else if (obj instanceof List) { + def newList = new ArrayList(); + for (item in obj) { + newList.add(normalizeFields(item)); + } + return newList; + } + return obj; + } + + // Apply transformation + if (ctx.json != null) { + ctx.ti_flashpoint = ctx.ti_flashpoint ?: [:]; + ctx.ti_flashpoint.alert = normalizeFields(ctx.json); + ctx.remove('json'); + } + + # Date processors + - date: + tag: date_ti_flashpoint_alert_created_at_into_ti_flashpoint_alert_created_at_20fe7d61 + if: ctx.ti_flashpoint?.alert?.created_at != null && ctx.ti_flashpoint.alert.created_at != '' + field: ti_flashpoint.alert.created_at + target_field: ti_flashpoint.alert.created_at + formats: + - UNIX + - ISO8601 + - yyyy-MM-dd' 'HH:mm:ss + - yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ + - yyyy-MM-dd'T'HH:mm:ssXXXXX + - yyyy-MM-dd' 'HH:mm:ssXXXXX + on_failure: + - remove: + tag: remove_ti_flashpoint_alert_created_at_51709117 + field: + - ti_flashpoint.alert.created_at + - append: + tag: append_error_message_ca0f3ee7 + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - date: + tag: date_ti_flashpoint_alert_generated_at_into_ti_flashpoint_alert_generated_at_9448b857 + if: ctx.ti_flashpoint?.alert?.generated_at != null && ctx.ti_flashpoint.alert.generated_at != '' + field: ti_flashpoint.alert.generated_at + target_field: ti_flashpoint.alert.generated_at + formats: + - UNIX + - ISO8601 + - yyyy-MM-dd' 'HH:mm:ss + - yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ + - yyyy-MM-dd'T'HH:mm:ssXXXXX + - yyyy-MM-dd' 'HH:mm:ssXXXXX + on_failure: + - remove: + tag: remove_ti_flashpoint_alert_generated_at_b31b7376 + field: + - ti_flashpoint.alert.generated_at + - append: + tag: append_error_message_c1605d45 + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - date: + tag: date_ti_flashpoint_alert_resource_sort_date_into_ti_flashpoint_alert_resource_sort_date_c561a9d0 + if: ctx.ti_flashpoint?.alert?.resource?.sort_date != null && ctx.ti_flashpoint.alert.resource.sort_date != '' + field: ti_flashpoint.alert.resource.sort_date + target_field: ti_flashpoint.alert.resource.sort_date + formats: + - UNIX + - ISO8601 + - yyyy-MM-dd' 'HH:mm:ss + - yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ + - yyyy-MM-dd'T'HH:mm:ssXXXXX + - yyyy-MM-dd' 'HH:mm:ssXXXXX + on_failure: + - remove: + tag: remove_ti_flashpoint_alert_resource_sort_date_7047e135 + field: + - ti_flashpoint.alert.resource.sort_date + - append: + tag: append_error_message_d2ba01c0 + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - date: + tag: date_ti_flashpoint_alert_resource_created_at_date_time_into_ti_flashpoint_alert_resource_created_at_384a797f + if: ctx.ti_flashpoint?.alert?.resource?.created_at?.date_time != null && ctx.ti_flashpoint.alert.resource.created_at.date_time != '' + field: ti_flashpoint.alert.resource.created_at.date_time + target_field: ti_flashpoint.alert.resource.created_at + formats: + - UNIX + - ISO8601 + - yyyy-MM-dd' 'HH:mm:ss + - yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ + - yyyy-MM-dd'T'HH:mm:ssXXXXX + - yyyy-MM-dd' 'HH:mm:ssXXXXX + on_failure: + - remove: + tag: remove_ti_flashpoint_alert_resource_created_at_date_time_71d68711 + field: + - ti_flashpoint.alert.resource.created_at.date_time + - append: + tag: append_error_message_a649b53d + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + + # Convert to Long + - foreach: + tag: foreach_of_ti_flashpoint_alert_resource_media_v2_4f9463ee + if: ctx.ti_flashpoint?.alert?.resource?.media_v2 instanceof List + field: ti_flashpoint.alert.resource.media_v2 + processor: + convert: + tag: convert__ingest__value_image_enrichment_enrichments_v1_image_analysis_safe_search_adult_to_long_edf0408c + field: _ingest._value.image_enrichment.enrichments.v1.image_analysis.safe_search.adult + type: long + ignore_missing: true + on_failure: + - remove: + tag: remove__ingest__value_image_enrichment_enrichments_v1_image_analysis_safe_search_adult_3a2917f7 + field: + - _ingest._value.image_enrichment.enrichments.v1.image_analysis.safe_search.adult + - append: + tag: append_error_message_da8c3f2c + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + tag: foreach_of_ti_flashpoint_alert_resource_media_v2_4e94625b + if: ctx.ti_flashpoint?.alert?.resource?.media_v2 instanceof List + field: ti_flashpoint.alert.resource.media_v2 + processor: + convert: + tag: convert__ingest__value_image_enrichment_enrichments_v1_image_analysis_safe_search_medical_to_long_400543e1 + field: _ingest._value.image_enrichment.enrichments.v1.image_analysis.safe_search.medical + type: long + ignore_missing: true + on_failure: + - remove: + tag: remove__ingest__value_image_enrichment_enrichments_v1_image_analysis_safe_search_medical_23bec407 + field: + - _ingest._value.image_enrichment.enrichments.v1.image_analysis.safe_search.medical + - append: + tag: append_error_message_c6281267 + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + tag: foreach_of_ti_flashpoint_alert_resource_media_v2_4d9460c8 + if: ctx.ti_flashpoint?.alert?.resource?.media_v2 instanceof List + field: ti_flashpoint.alert.resource.media_v2 + processor: + convert: + tag: convert__ingest__value_image_enrichment_enrichments_v1_image_analysis_safe_search_racy_to_long_f34e64b1 + field: _ingest._value.image_enrichment.enrichments.v1.image_analysis.safe_search.racy + type: long + ignore_missing: true + on_failure: + - remove: + tag: remove__ingest__value_image_enrichment_enrichments_v1_image_analysis_safe_search_racy_110678e7 + field: + - _ingest._value.image_enrichment.enrichments.v1.image_analysis.safe_search.racy + - append: + tag: append_error_message_92aefeb7 + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + tag: foreach_of_ti_flashpoint_alert_resource_media_v2_54946bcd + if: ctx.ti_flashpoint?.alert?.resource?.media_v2 instanceof List + field: ti_flashpoint.alert.resource.media_v2 + processor: + convert: + tag: convert__ingest__value_image_enrichment_enrichments_v1_image_analysis_safe_search_spoof_to_long_a6af8703 + field: _ingest._value.image_enrichment.enrichments.v1.image_analysis.safe_search.spoof + type: long + ignore_missing: true + on_failure: + - remove: + tag: remove__ingest__value_image_enrichment_enrichments_v1_image_analysis_safe_search_spoof_643021eb + field: + - _ingest._value.image_enrichment.enrichments.v1.image_analysis.safe_search.spoof + - append: + tag: append_error_message_b1f82e31 + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + tag: foreach_of_ti_flashpoint_alert_resource_media_v2_53946a3a + if: ctx.ti_flashpoint?.alert?.resource?.media_v2 instanceof List + field: ti_flashpoint.alert.resource.media_v2 + processor: + convert: + tag: convert__ingest__value_image_enrichment_enrichments_v1_image_analysis_safe_search_violence_to_long_4517615d + field: _ingest._value.image_enrichment.enrichments.v1.image_analysis.safe_search.violence + type: long + ignore_missing: true + on_failure: + - remove: + tag: remove__ingest__value_image_enrichment_enrichments_v1_image_analysis_safe_search_violence_3c96564f + field: + - _ingest._value.image_enrichment.enrichments.v1.image_analysis.safe_search.violence + - append: + tag: append_error_message_e2baf973 + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + + # Convert to Boolean + - convert: + tag: convert_ti_flashpoint_alert_is_read_to_boolean_7b4e36be + field: ti_flashpoint.alert.is_read + type: boolean + ignore_missing: true + on_failure: + - remove: + tag: remove_ti_flashpoint_alert_is_read_4cbff047 + field: + - ti_flashpoint.alert.is_read + - append: + tag: append_error_message_6f9d072e + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + + # Convert to String + - convert: + tag: convert_ti_flashpoint_alert_reason_id_to_string_316a93cd + field: ti_flashpoint.alert.reason.id + type: string + ignore_missing: true + - convert: + tag: convert_ti_flashpoint_alert_resource_container_container_native_id_to_string_9656b5dd + field: ti_flashpoint.alert.resource.container.container.native_id + type: string + ignore_missing: true + - convert: + tag: convert_ti_flashpoint_alert_resource_container_native_id_to_string_354f4d16 + field: ti_flashpoint.alert.resource.container.native_id + type: string + ignore_missing: true + - convert: + tag: convert_ti_flashpoint_alert_resource_site_actor_native_id_to_string_c200f184 + field: ti_flashpoint.alert.resource.site_actor.native_id + type: string + ignore_missing: true + + # Map custom fields to corresponding ECS and related fields. + - set: + tag: set_event_created_from_ti_flashpoint_alert_created_at_13465d21 + field: event.created + copy_from: ti_flashpoint.alert.created_at + ignore_empty_value: true + - set: + tag: set_@timestamp_from_ti_flashpoint_alert_created_at_3ef47d75 + field: '@timestamp' + copy_from: ti_flashpoint.alert.created_at + ignore_empty_value: true + - set: + tag: set_event_id_from_ti_flashpoint_alert_id_1a35b175 + field: event.id + copy_from: ti_flashpoint.alert.id + ignore_empty_value: true + - set: + tag: set_event_reason_from_ti_flashpoint_alert_reason_name_a3c940ac + field: event.reason + copy_from: ti_flashpoint.alert.reason.name + ignore_empty_value: true + - set: + tag: set_user_name_from_ti_flashpoint_alert_resource_author_53cf9bf2 + field: user.name + copy_from: ti_flashpoint.alert.resource.author + ignore_empty_value: true + - set: + tag: set_container_name_from_ti_flashpoint_alert_resource_container_name_84f1e459 + field: container.name + copy_from: ti_flashpoint.alert.resource.container.name + ignore_empty_value: true + - set: + tag: set_container_id_from_ti_flashpoint_alert_resource_container_native_id_32f7eccb + field: container.id + copy_from: ti_flashpoint.alert.resource.container.native_id + ignore_empty_value: true + - set: + tag: set_event_url_from_ti_flashpoint_alert_resource_link_a6602e82 + field: event.url + copy_from: ti_flashpoint.alert.resource.link + ignore_empty_value: true + - foreach: + tag: foreach_of_ti_flashpoint_alert_resource_media_v2_12d375a3 + if: ctx.ti_flashpoint?.alert?.resource?.media_v2 instanceof List + field: ti_flashpoint.alert.resource.media_v2 + processor: + append: + tag: append_file_mime_type_fe55ae27 + field: file.mime_type + value: '{{{_ingest._value.mime_type}}}' + allow_duplicates: false + - foreach: + tag: foreach_of_ti_flashpoint_alert_resource_media_v2_13d37736 + if: ctx.ti_flashpoint?.alert?.resource?.media_v2 instanceof List + field: ti_flashpoint.alert.resource.media_v2 + processor: + append: + tag: append_file_hash_sha256_6ef886dd + field: file.hash.sha256 + value: '{{{_ingest._value.phash256}}}' + allow_duplicates: false + - foreach: + tag: foreach_of_ti_flashpoint_alert_resource_media_v2_14d378c9 + if: ctx.ti_flashpoint?.alert?.resource?.media_v2 instanceof List + field: ti_flashpoint.alert.resource.media_v2 + processor: + append: + tag: append_file_hash_sha1_c937ae03 + field: file.hash.sha1 + value: '{{{_ingest._value.sha1}}}' + allow_duplicates: false + - append: + tag: append_related_user_from_ti_flashpoint_alert_resource_author_c3f04661 + if: ctx.ti_flashpoint?.alert?.resource?.author != null + field: related.user + value: '{{{ti_flashpoint.alert.resource.author}}}' + allow_duplicates: false + - foreach: + tag: foreach_of_ti_flashpoint_alert_resource_authors_c4becc2a + if: ctx.ti_flashpoint?.alert?.resource?.authors instanceof List + field: ti_flashpoint.alert.resource.authors + processor: + append: + tag: append_related_user_38200aeb + field: related.user + value: '{{{_ingest._value}}}' + allow_duplicates: false + - foreach: + tag: foreach_of_ti_flashpoint_alert_resource_media_v2_15d37a5c + if: ctx.ti_flashpoint?.alert?.resource?.media_v2 instanceof List + field: ti_flashpoint.alert.resource.media_v2 + processor: + append: + tag: append_related_hash_0e227c8c + field: related.hash + value: '{{{_ingest._value.phash}}}' + allow_duplicates: false + - foreach: + tag: foreach_of_ti_flashpoint_alert_resource_media_v2_16d37bef + if: ctx.ti_flashpoint?.alert?.resource?.media_v2 instanceof List + field: ti_flashpoint.alert.resource.media_v2 + processor: + append: + tag: append_related_hash_a0128d9f + field: related.hash + value: '{{{_ingest._value.phash256}}}' + allow_duplicates: false + - foreach: + tag: foreach_of_ti_flashpoint_alert_resource_media_v2_17d37d82 + if: ctx.ti_flashpoint?.alert?.resource?.media_v2 instanceof List + field: ti_flashpoint.alert.resource.media_v2 + processor: + append: + tag: append_related_hash_224c210b + field: related.hash + value: '{{{_ingest._value.sha1}}}' + allow_duplicates: false + + # Remove duplicate custom fields if preserve_duplicate_custom_fields are not enabled + - foreach: + tag: foreach_of_ti_flashpoint_alert_resource_media_v2_ea27702d + if: ctx.ti_flashpoint?.alert?.resource?.media_v2 instanceof List && (ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields')) + field: ti_flashpoint.alert.resource.media_v2 + processor: + remove: + tag: remove_custom_duplicate_fields_for_media_v2_020834ca + field: + - _ingest._value.mime_type + - _ingest._value.phash256 + - _ingest._value.sha1 + ignore_missing: true + - remove: + tag: remove_custom_duplicate_fields_c637c818 + if: ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields') + field: + - ti_flashpoint.alert.created_at + - ti_flashpoint.alert.id + - ti_flashpoint.alert.reason.name + - ti_flashpoint.alert.resource.author + - ti_flashpoint.alert.resource.container.name + - ti_flashpoint.alert.resource.container.native_id + - ti_flashpoint.alert.resource.link + ignore_missing: true + + # Cleanup + - script: + description: This script processor iterates over the whole document to remove fields with null values. + tag: script_to_drop_null_values_8360f3de + lang: painless + source: |- + void handleMap(Map map) { + map.values().removeIf(v -> { + if (v instanceof Map) { + handleMap(v); + } else if (v instanceof List) { + handleList(v); + } + return v == null || v == '' || (v instanceof Map && v.size() == 0) || (v instanceof List && v.size() == 0) + }); + } + void handleList(List list) { + list.removeIf(v -> { + if (v instanceof Map) { + handleMap(v); + } else if (v instanceof List) { + handleList(v); + } + return v == null || v == '' || (v instanceof Map && v.size() == 0) || (v instanceof List && v.size() == 0) + }); + } + handleMap(ctx); + - set: + tag: set_event_kind_to_pipeline_error_92954dfa + if: ctx.error?.message != null + field: event.kind + value: pipeline_error + - append: + tag: append_tags_9fe66b2c + if: ctx.error?.message != null + field: tags + value: preserve_original_event + allow_duplicates: false +on_failure: + - append: + tag: append_error_message_e0c9bd63 + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + tag: set_event_kind_to_pipeline_error_f51b77ad + field: event.kind + value: pipeline_error + - append: + tag: append_tags_d762b9c5 + field: tags + value: preserve_original_event + allow_duplicates: false diff --git a/packages/ti_flashpoint/data_stream/alert/fields/base-fields.yml b/packages/ti_flashpoint/data_stream/alert/fields/base-fields.yml new file mode 100644 index 00000000000..19fd32acacd --- /dev/null +++ b/packages/ti_flashpoint/data_stream/alert/fields/base-fields.yml @@ -0,0 +1,16 @@ +- name: data_stream.dataset + external: ecs +- name: data_stream.namespace + external: ecs +- name: data_stream.type + external: ecs +- name: event.dataset + type: constant_keyword + external: ecs + value: ti_flashpoint.alert +- name: event.module + type: constant_keyword + external: ecs + value: ti_flashpoint +- name: '@timestamp' + external: ecs diff --git a/packages/ti_flashpoint/data_stream/alert/fields/beats.yml b/packages/ti_flashpoint/data_stream/alert/fields/beats.yml new file mode 100644 index 00000000000..4084f1dc7f5 --- /dev/null +++ b/packages/ti_flashpoint/data_stream/alert/fields/beats.yml @@ -0,0 +1,6 @@ +- name: input.type + type: keyword + description: Type of filebeat input. +- name: log.offset + type: long + description: Log offset. diff --git a/packages/ti_flashpoint/data_stream/alert/fields/ecs.yml b/packages/ti_flashpoint/data_stream/alert/fields/ecs.yml new file mode 100644 index 00000000000..605bae51ce8 --- /dev/null +++ b/packages/ti_flashpoint/data_stream/alert/fields/ecs.yml @@ -0,0 +1,9 @@ +# Define ECS constant fields as constant_keyword +- name: observer.product + external: ecs + type: constant_keyword + value: Ignite +- name: observer.vendor + external: ecs + type: constant_keyword + value: Flashpoint diff --git a/packages/ti_flashpoint/data_stream/alert/fields/fields.yml b/packages/ti_flashpoint/data_stream/alert/fields/fields.yml new file mode 100644 index 00000000000..cd086279f0c --- /dev/null +++ b/packages/ti_flashpoint/data_stream/alert/fields/fields.yml @@ -0,0 +1,202 @@ +- name: ti_flashpoint + type: group + fields: + - name: alert + type: group + fields: + - name: created_at + type: date + - name: data_type + type: keyword + - name: generated_at + type: date + - name: highlight_text + type: keyword + - name: highlights + type: group + fields: + - name: body + type: group + fields: + - name: text/plain + type: keyword + - name: container + type: group + fields: + - name: name + type: keyword + - name: description + type: keyword + - name: site + type: group + fields: + - name: title + type: keyword + - name: site_actor + type: group + fields: + - name: names + type: group + fields: + - name: aliases + type: keyword + - name: handle + type: keyword + - name: title + type: keyword + - name: id + type: keyword + - name: is_read + type: boolean + - name: parent_data_type + type: keyword + - name: reason + type: group + fields: + - name: details + type: group + fields: + - name: params + type: group + fields: + - name: exclude + type: flattened + - name: include + type: group + fields: + - name: date + type: group + fields: + - name: end + type: keyword + - name: label + type: keyword + - name: start + type: keyword + - name: sources + type: keyword + - name: entity + type: flattened + - name: id + type: keyword + - name: name + type: keyword + - name: origin + type: keyword + - name: text + type: keyword + - name: resource + type: group + fields: + - name: author + type: keyword + - name: authors + type: keyword + - name: basetypes + type: keyword + - name: container + type: group + fields: + - name: container + type: group + fields: + - name: name + type: keyword + - name: native_id + type: keyword + - name: title + type: keyword + - name: name + type: keyword + - name: native_id + type: keyword + - name: server + type: keyword + - name: title + type: keyword + - name: country + type: keyword + - name: created_at + type: date + - name: description + type: keyword + - name: id + type: keyword + - name: link + type: keyword + - name: media_v2 + type: group + fields: + - name: image_enrichment + type: group + fields: + - name: enrichments + type: group + fields: + - name: v1 + type: group + fields: + - name: image_analysis + type: group + fields: + - name: safe_search + type: group + fields: + - name: adult + type: long + - name: medical + type: long + - name: racy + type: long + - name: spoof + type: long + - name: violence + type: long + - name: media_type + type: keyword + - name: mime_type + type: keyword + - name: phash + type: keyword + - name: phash256 + type: keyword + - name: sha1 + type: keyword + - name: storage_uri + type: keyword + - name: parent_basetypes + type: keyword + - name: report + type: group + fields: + - name: summary + type: keyword + - name: title + type: keyword + - name: section + type: keyword + - name: site + type: group + fields: + - name: title + type: keyword + - name: site_actor + type: group + fields: + - name: names + type: group + fields: + - name: handle + type: keyword + - name: native_id + type: keyword + - name: sort_date + type: date + - name: title + type: keyword + - name: source + type: keyword + - name: status + type: keyword + - name: tags + type: flattened diff --git a/packages/ti_flashpoint/data_stream/alert/manifest.yml b/packages/ti_flashpoint/data_stream/alert/manifest.yml new file mode 100644 index 00000000000..0a0d89614dc --- /dev/null +++ b/packages/ti_flashpoint/data_stream/alert/manifest.yml @@ -0,0 +1,95 @@ +title: Alert +type: logs +streams: + - input: cel + title: Alert + description: Collect Alert logs from Flashpoint. + template_path: cel.yml.hbs + vars: + - name: initial_interval + type: text + title: Initial Interval + multi: false + required: true + show_user: true + default: 24h + description: How far back to pull the logs from Flashpoint API. Supported units for this parameter are h/m/s. + - name: interval + type: text + title: Interval + description: Duration between requests to the Flashpoint API. Supported units for this parameter are h/m/s. + multi: false + required: true + show_user: true + default: 5m + - name: page_size + type: integer + title: Page Size + description: Page size for the response of the Flashpoint API. + multi: false + required: true + show_user: false + default: 5000 + - name: max_executions + type: integer + title: Maximum Pages Per Interval + description: Maximum Pages Per Interval is the maximum number of pages that can be collected at each interval. + multi: false + required: false + show_user: false + default: 1000 + - name: enable_request_tracer + type: bool + title: Enable request tracing + multi: false + default: false + required: false + show_user: false + description: >- + The request tracer logs requests and responses to the agent's local file-system for debugging configurations. + Enabling this request tracing compromises security and should only be used for debugging. Disabling the request + tracer will delete any stored traces. + See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_enable) + for details. + - name: preserve_original_event + type: bool + title: Preserve original event + description: Preserves a raw copy of the original event, added to the field event.original. + multi: false + required: false + show_user: true + default: false + - name: tags + type: text + title: Tags + description: Tags for the data-stream. + multi: true + required: true + show_user: false + default: + - forwarded + - ti_flashpoint-alert + - name: http_client_timeout + type: text + title: HTTP Client Timeout + description: Duration before declaring that the HTTP client connection has timed out. Supported time units are ns, us, ms, s, m, h. + multi: false + required: true + show_user: false + default: 30s + - name: preserve_duplicate_custom_fields + required: false + title: Preserve duplicate custom fields + description: Preserve ti_flashpoint.alert fields that were copied to Elastic Common Schema (ECS) fields. + type: bool + multi: false + show_user: false + default: false + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: >- + Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. diff --git a/packages/ti_flashpoint/data_stream/alert/sample_event.json b/packages/ti_flashpoint/data_stream/alert/sample_event.json new file mode 100644 index 00000000000..da5fd464944 --- /dev/null +++ b/packages/ti_flashpoint/data_stream/alert/sample_event.json @@ -0,0 +1,118 @@ +{ + "@timestamp": "2025-12-17T06:53:59.956Z", + "agent": { + "ephemeral_id": "307ed7f9-a244-4a69-b58d-179cd44507de", + "id": "69029d8a-4a05-480c-ae8a-9477425fa1f4", + "name": "elastic-agent-47245", + "type": "filebeat", + "version": "8.18.0" + }, + "container": { + "id": "284504102" + }, + "data_stream": { + "dataset": "ti_flashpoint.alert", + "namespace": "32105", + "type": "logs" + }, + "ecs": { + "version": "9.2.0" + }, + "elastic_agent": { + "id": "69029d8a-4a05-480c-ae8a-9477425fa1f4", + "snapshot": false, + "version": "8.18.0" + }, + "event": { + "agent_id_status": "verified", + "created": "2025-12-17T06:53:59.956Z", + "dataset": "ti_flashpoint.alert", + "id": "61e89ab1-bd39-459e-84b5-79023b721c7d", + "ingested": "2025-12-18T12:49:50Z", + "kind": "alert", + "original": "{\"created_at\":\"2025-12-17T06:53:59.956272Z\",\"data_type\":\"board\",\"generated_at\":\"2025-12-17T06:53:57.570160Z\",\"highlight_text\":\"\\u0026gt;\\u0026gt;28\",\"highlights\":{\"body.text/plain\":[\"\\u0026gt;\\u0026gt;28450\"],\"site_actor.names.aliases\":[\"\\u003cmark\\u003eAnonymous\\u003c/mark\\u003e (None)\",\"\\u003cmark\\u003eAnonymous\\u003c/mark\\u003e (None)\"]},\"id\":\"61e89ab1-bd39-459e-84b5-79023b721c7d\",\"is_read\":false,\"parent_data_type\":null,\"reason\":{\"details\":{\"params\":{\"exclude\":{},\"include\":{\"date\":{\"end\":\"now\",\"label\":\"Last 30 Days\",\"start\":\"now-30d\"}}},\"sources\":[\"communities\",\"media\",\"marketplaces\",\"news\"]},\"entity\":null,\"id\":\"61e526db-8554-42bb-b590-e1f6a37cd4fb\",\"name\":\"temp\",\"origin\":\"searches\",\"text\":\"temp\"},\"resource\":{\"basetypes\":[\"chan\",\"comment\",\"conversation\",\"message\",\"web\"],\"container\":{\"container\":{\"native_id\":\"a\",\"title\":\"/a/ - Anime \\u0026 Manga\"},\"native_id\":\"284504102\"},\"created_at\":{\"date-time\":\"2025-12-15T11:43:32+00:00\",\"raw\":\"2025-12-15 11:43:32+00:00\",\"timestamp\":1765799012},\"id\":\"SrcBxTtDV-O5U-ycJ_Cv_g\",\"section\":\"/a/ - Anime \\u0026 Manga\",\"site\":{\"title\":\"4chan\"},\"site_actor\":{\"names\":{\"handle\":\"Anonymous (None)\"},\"native_id\":\"Anonymous\"},\"sort_date\":\"2025-12-15T11:43:32Z\",\"title\":\"284504102\"},\"source\":\"communities\",\"status\":null,\"tags\":{}}", + "reason": "temp" + }, + "input": { + "type": "cel" + }, + "tags": [ + "preserve_original_event", + "preserve_duplicate_custom_fields", + "forwarded", + "ti_flashpoint-alert" + ], + "ti_flashpoint": { + "alert": { + "created_at": "2025-12-17T06:53:59.956Z", + "data_type": "board", + "generated_at": "2025-12-17T06:53:57.570Z", + "highlight_text": ">>28", + "highlights": { + "body.text/plain": [ + ">>28450" + ], + "site_actor.names.aliases": [ + "Anonymous (None)", + "Anonymous (None)" + ] + }, + "id": "61e89ab1-bd39-459e-84b5-79023b721c7d", + "is_read": false, + "reason": { + "details": { + "params": { + "include": { + "date": { + "end": "now", + "label": "Last 30 Days", + "start": "now-30d" + } + } + }, + "sources": [ + "communities", + "media", + "marketplaces", + "news" + ] + }, + "id": "61e526db-8554-42bb-b590-e1f6a37cd4fb", + "name": "temp", + "origin": "searches", + "text": "temp" + }, + "resource": { + "basetypes": [ + "chan", + "comment", + "conversation", + "message", + "web" + ], + "container": { + "container": { + "native_id": "a", + "title": "/a/ - Anime & Manga" + }, + "native_id": "284504102" + }, + "created_at": "2025-12-15T11:43:32.000Z", + "id": "SrcBxTtDV-O5U-ycJ_Cv_g", + "section": "/a/ - Anime & Manga", + "site": { + "title": "4chan" + }, + "site_actor": { + "names": { + "handle": "Anonymous (None)" + }, + "native_id": "Anonymous" + }, + "sort_date": "2025-12-15T11:43:32.000Z", + "title": "284504102" + }, + "source": "communities" + } + } +} diff --git a/packages/ti_flashpoint/docs/README.md b/packages/ti_flashpoint/docs/README.md new file mode 100644 index 00000000000..aa2dfe17551 --- /dev/null +++ b/packages/ti_flashpoint/docs/README.md @@ -0,0 +1,306 @@ +# Flashpoint Integration for Elastic + +## Overview + +[Flashpoint](https://flashpoint.io/) is a comprehensive threat intelligence platform that delivers actionable insights from dark web, deep web, and technical sources. It combines human-curated intelligence with automated collection to help organizations identify emerging threats, monitor adversary activity, and assess cyber risk with enriched context. + +The Flashpoint integration for Elastic collects alerts from the **Flashpoint Ignite API** and visualizes them in Kibana. + +### Compatibility + +The Flashpoint integration is compatible with Ignite API version **1.2**. + +### How it works + +This integration periodically queries the Flashpoint Ignite API to retrieve logs. + +## What data does this integration collect? + +This integration collects log messages of the following type: + +- `Alert`: Collects `alert` logs from the Flashpoint Ignite API (endpoint: `/alert-management/v1/notifications`), + +### Supported use cases + +Integrating Flashpoint with Elastic SIEM provides centralized visibility into threat intelligence alerts. Kibana dashboards present key metrics such as `Total Alerts`, along with visualizations showing `Alerts by Data Type`, `Source`, and `Origin`. + +`Alert Trends over Time`, `Top Authors`, `MIME Types`, `Alert Sources`, and `Geographic Distribution` of related resources help analysts quickly monitor activity and investigate alerts. These insights support efficient threat monitoring and analysis workflows. + +## What do I need to use this integration? + +### From Flashpoint + +To collect data through the Flashpoint Ignite API, you need to provide an **API Token**. Authentication is handled using the **API Token**, which serves as the required credential. + +#### Retrieve an API Token: + +1. Log in to the **Flashpoint** Instance. +2. Click on your profile icon in the top-right corner and select **Manage API Tokens**. +3. Click **Generate Token**. +4. Enter a name for the API token and click **Generate Token**. +5. Copy and securely store the generated API token for use in the integration configuration. + +## How do I deploy this integration? + +This integration supports both Elastic Agentless-based and Agent-based installations. + +### Agentless-based installation + +Agentless integrations allow you to collect data without having to manage Elastic Agent in your cloud. They make manual agent deployment unnecessary, so you can focus on your data instead of the agent that collects it. For more information, refer to [Agentless integrations](https://www.elastic.co/guide/en/serverless/current/security-agentless-integrations.html) and the [Agentless integrations FAQ](https://www.elastic.co/guide/en/serverless/current/agentless-integration-troubleshooting.html). + +Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. This functionality is in beta and is subject to change. Beta features are not subject to the support SLA of official GA features. + +### Agent-based installation + +Elastic Agent must be installed. For more details, check the Elastic Agent [installation instructions](docs-content://reference/fleet/install-elastic-agents.md). You can install only one Elastic Agent per host. + + +### configure + +1. In the top search bar in Kibana, search for **Integrations**. +2. In the search bar, type **Flashpoint**. +3. Select the **Flashpoint** integration from the search results. +4. Select **Add Flashpoint** to add the integration. +5. Enable and configure only the collection methods which you will use. + + * To **Collect logs from Flashpoint API**, you'll need to: + + - Configure **API Token**. + - Adjust the integration configuration parameters if required, including the **Initial Interval**, **Interval**, **Page Size** etc. to enable data collection. + +6. Select **Save and continue** to save the integration. + +### Validation + +#### Dashboard populated + +1. In the top search bar in Kibana, search for **Dashboards**. +2. In the search bar, type **Flashpoint**, and verify the dashboard information is populated. + +## Performance and scaling + +For more information on architectures that can be used for scaling this integration, check the [Ingest Architectures](https://www.elastic.co/docs/manage-data/ingest/ingest-reference-architectures) documentation. + +## Reference + +### ECS field reference + +#### Alert + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date | +| data_stream.dataset | The field can contain anything that makes sense to signify the source of the data. Examples include `nginx.access`, `prometheus`, `endpoint` etc. For data streams that otherwise fit, but that do not have dataset set we use the value "generic" for the dataset value. `event.dataset` should have the same value as `data_stream.dataset`. Beyond the Elasticsearch data stream naming criteria noted above, the `dataset` value has additional restrictions: \* Must not contain `-` \* No longer than 100 characters | constant_keyword | +| data_stream.namespace | A user defined namespace. Namespaces are useful to allow grouping of data. Many users already organize their indices this way, and the data stream naming scheme now provides this best practice as a default. Many users will populate this field with `default`. If no value is used, it falls back to `default`. Beyond the Elasticsearch index naming criteria noted above, `namespace` value has the additional restrictions: \* Must not contain `-` \* No longer than 100 characters | constant_keyword | +| data_stream.type | An overarching type for the data stream. Currently allowed values are "logs" and "metrics". We expect to also add "traces" and "synthetics" in the near future. | constant_keyword | +| event.dataset | Name of the dataset. If an event source publishes more than one type of log or events (e.g. access log, error log), the dataset is used to specify which one the event comes from. It's recommended but not required to start the dataset name with the module name, followed by a dot, then the dataset name. | constant_keyword | +| event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | constant_keyword | +| input.type | Type of filebeat input. | keyword | +| log.offset | Log offset. | long | +| observer.product | The product name of the observer. | constant_keyword | +| observer.vendor | Vendor name of the observer. | constant_keyword | +| ti_flashpoint.alert.created_at | | date | +| ti_flashpoint.alert.data_type | | keyword | +| ti_flashpoint.alert.generated_at | | date | +| ti_flashpoint.alert.highlight_text | | keyword | +| ti_flashpoint.alert.highlights.body.text/plain | | keyword | +| ti_flashpoint.alert.highlights.container.name | | keyword | +| ti_flashpoint.alert.highlights.description | | keyword | +| ti_flashpoint.alert.highlights.site.title | | keyword | +| ti_flashpoint.alert.highlights.site_actor.names.aliases | | keyword | +| ti_flashpoint.alert.highlights.site_actor.names.handle | | keyword | +| ti_flashpoint.alert.highlights.title | | keyword | +| ti_flashpoint.alert.id | | keyword | +| ti_flashpoint.alert.is_read | | boolean | +| ti_flashpoint.alert.parent_data_type | | keyword | +| ti_flashpoint.alert.reason.details.params.exclude | | flattened | +| ti_flashpoint.alert.reason.details.params.include.date.end | | keyword | +| ti_flashpoint.alert.reason.details.params.include.date.label | | keyword | +| ti_flashpoint.alert.reason.details.params.include.date.start | | keyword | +| ti_flashpoint.alert.reason.details.sources | | keyword | +| ti_flashpoint.alert.reason.entity | | flattened | +| ti_flashpoint.alert.reason.id | | keyword | +| ti_flashpoint.alert.reason.name | | keyword | +| ti_flashpoint.alert.reason.origin | | keyword | +| ti_flashpoint.alert.reason.text | | keyword | +| ti_flashpoint.alert.resource.author | | keyword | +| ti_flashpoint.alert.resource.authors | | keyword | +| ti_flashpoint.alert.resource.basetypes | | keyword | +| ti_flashpoint.alert.resource.container.container.name | | keyword | +| ti_flashpoint.alert.resource.container.container.native_id | | keyword | +| ti_flashpoint.alert.resource.container.container.title | | keyword | +| ti_flashpoint.alert.resource.container.name | | keyword | +| ti_flashpoint.alert.resource.container.native_id | | keyword | +| ti_flashpoint.alert.resource.container.server | | keyword | +| ti_flashpoint.alert.resource.container.title | | keyword | +| ti_flashpoint.alert.resource.country | | keyword | +| ti_flashpoint.alert.resource.created_at | | date | +| ti_flashpoint.alert.resource.description | | keyword | +| ti_flashpoint.alert.resource.id | | keyword | +| ti_flashpoint.alert.resource.link | | keyword | +| ti_flashpoint.alert.resource.media_v2.image_enrichment.enrichments.v1.image_analysis.safe_search.adult | | long | +| ti_flashpoint.alert.resource.media_v2.image_enrichment.enrichments.v1.image_analysis.safe_search.medical | | long | +| ti_flashpoint.alert.resource.media_v2.image_enrichment.enrichments.v1.image_analysis.safe_search.racy | | long | +| ti_flashpoint.alert.resource.media_v2.image_enrichment.enrichments.v1.image_analysis.safe_search.spoof | | long | +| ti_flashpoint.alert.resource.media_v2.image_enrichment.enrichments.v1.image_analysis.safe_search.violence | | long | +| ti_flashpoint.alert.resource.media_v2.media_type | | keyword | +| ti_flashpoint.alert.resource.media_v2.mime_type | | keyword | +| ti_flashpoint.alert.resource.media_v2.phash | | keyword | +| ti_flashpoint.alert.resource.media_v2.phash256 | | keyword | +| ti_flashpoint.alert.resource.media_v2.sha1 | | keyword | +| ti_flashpoint.alert.resource.media_v2.storage_uri | | keyword | +| ti_flashpoint.alert.resource.parent_basetypes | | keyword | +| ti_flashpoint.alert.resource.report.summary | | keyword | +| ti_flashpoint.alert.resource.report.title | | keyword | +| ti_flashpoint.alert.resource.section | | keyword | +| ti_flashpoint.alert.resource.site.title | | keyword | +| ti_flashpoint.alert.resource.site_actor.names.handle | | keyword | +| ti_flashpoint.alert.resource.site_actor.native_id | | keyword | +| ti_flashpoint.alert.resource.sort_date | | date | +| ti_flashpoint.alert.resource.title | | keyword | +| ti_flashpoint.alert.source | | keyword | +| ti_flashpoint.alert.status | | keyword | +| ti_flashpoint.alert.tags | | flattened | + + +### Example event + +#### Alert + +An example event for `alert` looks as following: + +```json +{ + "@timestamp": "2025-12-17T06:53:59.956Z", + "agent": { + "ephemeral_id": "307ed7f9-a244-4a69-b58d-179cd44507de", + "id": "69029d8a-4a05-480c-ae8a-9477425fa1f4", + "name": "elastic-agent-47245", + "type": "filebeat", + "version": "8.18.0" + }, + "container": { + "id": "284504102" + }, + "data_stream": { + "dataset": "ti_flashpoint.alert", + "namespace": "32105", + "type": "logs" + }, + "ecs": { + "version": "9.2.0" + }, + "elastic_agent": { + "id": "69029d8a-4a05-480c-ae8a-9477425fa1f4", + "snapshot": false, + "version": "8.18.0" + }, + "event": { + "agent_id_status": "verified", + "created": "2025-12-17T06:53:59.956Z", + "dataset": "ti_flashpoint.alert", + "id": "61e89ab1-bd39-459e-84b5-79023b721c7d", + "ingested": "2025-12-18T12:49:50Z", + "kind": "alert", + "original": "{\"created_at\":\"2025-12-17T06:53:59.956272Z\",\"data_type\":\"board\",\"generated_at\":\"2025-12-17T06:53:57.570160Z\",\"highlight_text\":\"\\u0026gt;\\u0026gt;28\",\"highlights\":{\"body.text/plain\":[\"\\u0026gt;\\u0026gt;28450\"],\"site_actor.names.aliases\":[\"\\u003cmark\\u003eAnonymous\\u003c/mark\\u003e (None)\",\"\\u003cmark\\u003eAnonymous\\u003c/mark\\u003e (None)\"]},\"id\":\"61e89ab1-bd39-459e-84b5-79023b721c7d\",\"is_read\":false,\"parent_data_type\":null,\"reason\":{\"details\":{\"params\":{\"exclude\":{},\"include\":{\"date\":{\"end\":\"now\",\"label\":\"Last 30 Days\",\"start\":\"now-30d\"}}},\"sources\":[\"communities\",\"media\",\"marketplaces\",\"news\"]},\"entity\":null,\"id\":\"61e526db-8554-42bb-b590-e1f6a37cd4fb\",\"name\":\"temp\",\"origin\":\"searches\",\"text\":\"temp\"},\"resource\":{\"basetypes\":[\"chan\",\"comment\",\"conversation\",\"message\",\"web\"],\"container\":{\"container\":{\"native_id\":\"a\",\"title\":\"/a/ - Anime \\u0026 Manga\"},\"native_id\":\"284504102\"},\"created_at\":{\"date-time\":\"2025-12-15T11:43:32+00:00\",\"raw\":\"2025-12-15 11:43:32+00:00\",\"timestamp\":1765799012},\"id\":\"SrcBxTtDV-O5U-ycJ_Cv_g\",\"section\":\"/a/ - Anime \\u0026 Manga\",\"site\":{\"title\":\"4chan\"},\"site_actor\":{\"names\":{\"handle\":\"Anonymous (None)\"},\"native_id\":\"Anonymous\"},\"sort_date\":\"2025-12-15T11:43:32Z\",\"title\":\"284504102\"},\"source\":\"communities\",\"status\":null,\"tags\":{}}", + "reason": "temp" + }, + "input": { + "type": "cel" + }, + "tags": [ + "preserve_original_event", + "preserve_duplicate_custom_fields", + "forwarded", + "ti_flashpoint-alert" + ], + "ti_flashpoint": { + "alert": { + "created_at": "2025-12-17T06:53:59.956Z", + "data_type": "board", + "generated_at": "2025-12-17T06:53:57.570Z", + "highlight_text": ">>28", + "highlights": { + "body.text/plain": [ + ">>28450" + ], + "site_actor.names.aliases": [ + "Anonymous (None)", + "Anonymous (None)" + ] + }, + "id": "61e89ab1-bd39-459e-84b5-79023b721c7d", + "is_read": false, + "reason": { + "details": { + "params": { + "include": { + "date": { + "end": "now", + "label": "Last 30 Days", + "start": "now-30d" + } + } + }, + "sources": [ + "communities", + "media", + "marketplaces", + "news" + ] + }, + "id": "61e526db-8554-42bb-b590-e1f6a37cd4fb", + "name": "temp", + "origin": "searches", + "text": "temp" + }, + "resource": { + "basetypes": [ + "chan", + "comment", + "conversation", + "message", + "web" + ], + "container": { + "container": { + "native_id": "a", + "title": "/a/ - Anime & Manga" + }, + "native_id": "284504102" + }, + "created_at": "2025-12-15T11:43:32.000Z", + "id": "SrcBxTtDV-O5U-ycJ_Cv_g", + "section": "/a/ - Anime & Manga", + "site": { + "title": "4chan" + }, + "site_actor": { + "names": { + "handle": "Anonymous (None)" + }, + "native_id": "Anonymous" + }, + "sort_date": "2025-12-15T11:43:32.000Z", + "title": "284504102" + }, + "source": "communities" + } + } +} +``` + + +### Inputs used + +These input is used in the integration: + +- [cel](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-cel) + +### API usage + +This integration dataset uses the following API: + +* List Alerts (endpoint: `/alert-management/v1/notifications`)| diff --git a/packages/ti_flashpoint/img/ti_flashpoint-alerts_dashboard.png b/packages/ti_flashpoint/img/ti_flashpoint-alerts_dashboard.png new file mode 100644 index 00000000000..6c9e49e2ba1 Binary files /dev/null and b/packages/ti_flashpoint/img/ti_flashpoint-alerts_dashboard.png differ diff --git a/packages/ti_flashpoint/img/ti_flashpoint-logo.svg b/packages/ti_flashpoint/img/ti_flashpoint-logo.svg new file mode 100644 index 00000000000..982608a02f6 --- /dev/null +++ b/packages/ti_flashpoint/img/ti_flashpoint-logo.svg @@ -0,0 +1,5 @@ + + + + + diff --git a/packages/ti_flashpoint/kibana/dashboard/ti_flashpoint-467e6747-8c82-4bd6-8ba5-2ec3d0e3b826.json b/packages/ti_flashpoint/kibana/dashboard/ti_flashpoint-467e6747-8c82-4bd6-8ba5-2ec3d0e3b826.json new file mode 100644 index 00000000000..b30344c303e --- /dev/null +++ b/packages/ti_flashpoint/kibana/dashboard/ti_flashpoint-467e6747-8c82-4bd6-8ba5-2ec3d0e3b826.json @@ -0,0 +1,1157 @@ +{ + "attributes": { + "controlGroupInput": { + "chainingSystem": "HIERARCHICAL", + "controlStyle": "oneLine", + "ignoreParentSettingsJSON": { + "ignoreFilters": false, + "ignoreQuery": false, + "ignoreTimerange": false, + "ignoreValidations": false + }, + "panelsJSON": { + "11023e32-443a-491a-b014-5a3b491df250": { + "explicitInput": { + "dataViewId": "logs-*", + "fieldName": "ti_flashpoint.alert.source", + "searchTechnique": "prefix", + "selectedOptions": [], + "sort": { + "by": "_count", + "direction": "desc" + }, + "title": "Source" + }, + "grow": true, + "order": 0, + "type": "optionsListControl", + "width": "medium" + }, + "af3850da-13a7-4b27-8ea4-be9dd8a5599c": { + "explicitInput": { + "dataViewId": "logs-*", + "fieldName": "ti_flashpoint.alert.reason.origin", + "searchTechnique": "prefix", + "selectedOptions": [], + "sort": { + "by": "_count", + "direction": "desc" + }, + "title": "Origin" + }, + "grow": true, + "order": 1, + "type": "optionsListControl", + "width": "medium" + } + }, + "showApplySelections": false + }, + "description": "This dashboard shows Alert collected by the Flashpoint Integration.", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ti_flashpoint.alert" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ti_flashpoint.alert" + } + } + } + ], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "optionsJSON": { + "hidePanelTitles": false, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "useMargins": true + }, + "panelsJSON": [ + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "id": "", + "params": { + "fontSize": 12, + "markdown": "This dashboard provides centralized visibility into alerts generated by Flashpoint.\n\nIt includes a pie chart showing Alerts by Data Types and a control panel highlighting Source and Origin. Alert activity over time is visualized through a line chart, while tables present Top Authors and Top MIME Types. A bar chart displays Alerts by Sources, and a key metric highlights the Total Alerts. Geographic distribution of alert-related resources is shown using a map of Top Resource Countries.\n\nTogether, these visualizations help teams quickly monitor alert trends and investigate alert sources and content.\n\n**[Integration Page](/app/integrations/detail/ti_flashpoint)**", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + } + }, + "gridData": { + "h": 18, + "i": "72d519d5-48f5-49cf-9a6f-370b8312c7d0", + "w": 12, + "x": 0, + "y": 0 + }, + "panelIndex": "72d519d5-48f5-49cf-9a6f-370b8312c7d0", + "title": "Overview", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-6f00c009-41f1-4d06-8d36-3cc606912204", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "6f00c009-41f1-4d06-8d36-3cc606912204": { + "columnOrder": [ + "904b4032-327e-4cc3-8fd8-1ad630c1393a" + ], + "columns": { + "904b4032-327e-4cc3-8fd8-1ad630c1393a": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Total Alerts", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "event.id" + } + }, + "incompleteColumns": {} + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "color": "#CC5642", + "layerId": "6f00c009-41f1-4d06-8d36-3cc606912204", + "layerType": "data", + "metricAccessor": "904b4032-327e-4cc3-8fd8-1ad630c1393a" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsMetric" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "language": "kuery", + "query": "" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false + }, + "gridData": { + "h": 18, + "i": "25ef2514-078f-4210-9bd2-6d1bc5e6d114", + "w": 9, + "x": 12, + "y": 0 + }, + "panelIndex": "25ef2514-078f-4210-9bd2-6d1bc5e6d114", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-f845e60c-45fb-44c5-a80d-d16335bc82df", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "f845e60c-45fb-44c5-a80d-d16335bc82df": { + "columnOrder": [ + "afb63d73-ea40-4c4e-85cf-44e11c94d3f2", + "946becc6-1f4f-42b6-a0c9-c814e65447cd" + ], + "columns": { + "946becc6-1f4f-42b6-a0c9-c814e65447cd": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "afb63d73-ea40-4c4e-85cf-44e11c94d3f2": { + "customLabel": true, + "dataType": "date", + "isBucketed": true, + "label": "Timestamp", + "operationType": "date_histogram", + "params": { + "dropPartials": false, + "includeEmptyRows": true, + "interval": "auto" + }, + "scale": "interval", + "sourceField": "@timestamp" + } + }, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "946becc6-1f4f-42b6-a0c9-c814e65447cd" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "f845e60c-45fb-44c5-a80d-d16335bc82df", + "layerType": "data", + "position": "top", + "seriesType": "line", + "showGridlines": false, + "xAccessor": "afb63d73-ea40-4c4e-85cf-44e11c94d3f2" + } + ], + "legend": { + "isVisible": true, + "position": "right", + "shouldTruncate": false, + "showSingleSeries": false + }, + "preferredSeriesType": "line", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "language": "kuery", + "query": "" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false + }, + "gridData": { + "h": 18, + "i": "2eb2574a-0c7b-4a64-aba1-83e0fd407f6a", + "w": 27, + "x": 21, + "y": 0 + }, + "panelIndex": "2eb2574a-0c7b-4a64-aba1-83e0fd407f6a", + "title": "Alerts over Time", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-f78bc1be-327e-43c6-b7f9-a35a37f35313", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "f78bc1be-327e-43c6-b7f9-a35a37f35313": { + "columnOrder": [ + "4ed429e2-597d-4b99-baff-1f9fe8c59d6c", + "4116d40d-b913-49c3-b265-a2cab2b7b335" + ], + "columns": { + "4116d40d-b913-49c3-b265-a2cab2b7b335": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "4ed429e2-597d-4b99-baff-1f9fe8c59d6c": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Data Type", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "4116d40d-b913-49c3-b265-a2cab2b7b335", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": true, + "parentFormat": { + "id": "terms" + }, + "size": 5 + }, + "scale": "ordinal", + "sourceField": "ti_flashpoint.alert.data_type" + } + }, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "f78bc1be-327e-43c6-b7f9-a35a37f35313", + "layerType": "data", + "legendDisplay": "show", + "metrics": [ + "4116d40d-b913-49c3-b265-a2cab2b7b335" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "4ed429e2-597d-4b99-baff-1f9fe8c59d6c" + ], + "truncateLegend": false + } + ], + "shape": "pie" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsPie" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "language": "kuery", + "query": "" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false + }, + "gridData": { + "h": 19, + "i": "1974815d-9fa9-4189-93ec-ca60f0b44a26", + "w": 24, + "x": 0, + "y": 18 + }, + "panelIndex": "1974815d-9fa9-4189-93ec-ca60f0b44a26", + "title": "Alerts by Data Types", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-18831400-03d9-4b80-a77e-184b46f64567", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "18831400-03d9-4b80-a77e-184b46f64567": { + "columnOrder": [ + "d2defe3a-ebc2-4554-b7b0-8bcbf171e7e2", + "8f4d760b-b0b5-449c-a1e7-da7e0c7ed4e3" + ], + "columns": { + "8f4d760b-b0b5-449c-a1e7-da7e0c7ed4e3": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "d2defe3a-ebc2-4554-b7b0-8bcbf171e7e2": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Source", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "8f4d760b-b0b5-449c-a1e7-da7e0c7ed4e3", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": true, + "parentFormat": { + "id": "terms" + }, + "size": 10 + }, + "scale": "ordinal", + "sourceField": "ti_flashpoint.alert.source" + } + }, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "8f4d760b-b0b5-449c-a1e7-da7e0c7ed4e3" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "18831400-03d9-4b80-a77e-184b46f64567", + "layerType": "data", + "position": "top", + "seriesType": "bar_horizontal_stacked", + "showGridlines": false, + "xAccessor": "d2defe3a-ebc2-4554-b7b0-8bcbf171e7e2" + } + ], + "legend": { + "isVisible": true, + "position": "right", + "shouldTruncate": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "language": "kuery", + "query": "" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false + }, + "gridData": { + "h": 19, + "i": "6a30acd7-711b-482d-8283-76b6be2127cf", + "w": 24, + "x": 24, + "y": 18 + }, + "panelIndex": "6a30acd7-711b-482d-8283-76b6be2127cf", + "title": "Alerts by Sources", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-dd697971-f8aa-44aa-b590-d034ea804fde", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "dd697971-f8aa-44aa-b590-d034ea804fde": { + "columnOrder": [ + "45323419-fd30-441f-a6b3-b6d123b8f5a5", + "8d2ceb02-3bcc-430a-a437-7d813e7c1841" + ], + "columns": { + "45323419-fd30-441f-a6b3-b6d123b8f5a5": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "MIME Type", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "8d2ceb02-3bcc-430a-a437-7d813e7c1841", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "size": 10 + }, + "scale": "ordinal", + "sourceField": "file.mime_type" + }, + "8d2ceb02-3bcc-430a-a437-7d813e7c1841": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "columns": [ + { + "columnId": "45323419-fd30-441f-a6b3-b6d123b8f5a5" + }, + { + "columnId": "8d2ceb02-3bcc-430a-a437-7d813e7c1841" + } + ], + "layerId": "dd697971-f8aa-44aa-b590-d034ea804fde", + "layerType": "data" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsDatatable" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "language": "kuery", + "query": "" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false + }, + "gridData": { + "h": 18, + "i": "3cac33c9-2ecb-4a73-96eb-871e96ffda0a", + "w": 24, + "x": 24, + "y": 37 + }, + "panelIndex": "3cac33c9-2ecb-4a73-96eb-871e96ffda0a", + "title": "Top MIME Types", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-bc945500-30d4-4e6f-8563-37bfdc7754c2", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "bc945500-30d4-4e6f-8563-37bfdc7754c2": { + "columnOrder": [ + "77817bed-4039-4275-9e5e-b90b0ac865b1", + "0ed90960-d3c0-4949-b075-fbc56df68616" + ], + "columns": { + "0ed90960-d3c0-4949-b075-fbc56df68616": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "77817bed-4039-4275-9e5e-b90b0ac865b1": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Author", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "0ed90960-d3c0-4949-b075-fbc56df68616", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "size": 10 + }, + "scale": "ordinal", + "sourceField": "ti_flashpoint.alert.resource.authors" + } + }, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "columns": [ + { + "columnId": "77817bed-4039-4275-9e5e-b90b0ac865b1" + }, + { + "columnId": "0ed90960-d3c0-4949-b075-fbc56df68616" + } + ], + "layerId": "bc945500-30d4-4e6f-8563-37bfdc7754c2", + "layerType": "data" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsDatatable" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "language": "kuery", + "query": "" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false + }, + "gridData": { + "h": 18, + "i": "092c00d7-b687-4efc-ae6d-09e44293a878", + "w": 24, + "x": 0, + "y": 37 + }, + "panelIndex": "092c00d7-b687-4efc-ae6d-09e44293a878", + "title": "Top Authors", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-2f1d5bd6-512d-458c-91d1-93c0bec36a07", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "2f1d5bd6-512d-458c-91d1-93c0bec36a07": { + "columnOrder": [ + "c5f92ae4-11ad-4788-876c-d3379629d99c", + "41ca4103-abea-4c18-932a-02e64244e38f" + ], + "columns": { + "41ca4103-abea-4c18-932a-02e64244e38f": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "c5f92ae4-11ad-4788-876c-d3379629d99c": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Resource Country", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "41ca4103-abea-4c18-932a-02e64244e38f", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": true, + "parentFormat": { + "id": "terms" + }, + "secondaryFields": [], + "size": 10 + }, + "scale": "ordinal", + "sourceField": "ti_flashpoint.alert.resource.country" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layerId": "2f1d5bd6-512d-458c-91d1-93c0bec36a07", + "layerType": "data", + "regionAccessor": "c5f92ae4-11ad-4788-876c-d3379629d99c", + "valueAccessor": "41ca4103-abea-4c18-932a-02e64244e38f" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsChoropleth" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "language": "kuery", + "query": "" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false + }, + "gridData": { + "h": 22, + "i": "b331d751-e8a6-4dd2-ad80-38f5bdaed8c7", + "w": 48, + "x": 0, + "y": 55 + }, + "panelIndex": "b331d751-e8a6-4dd2-ad80-38f5bdaed8c7", + "title": "Top Resource Countries", + "type": "lens" + } + ], + "timeRestore": false, + "title": "[Logs Flashpoint] Alerts", + "version": 3 + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2025-12-16T13:49:16.812Z", + "created_by": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0", + "id": "ti_flashpoint-467e6747-8c82-4bd6-8ba5-2ec3d0e3b826", + "references": [ + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "25ef2514-078f-4210-9bd2-6d1bc5e6d114:indexpattern-datasource-layer-6f00c009-41f1-4d06-8d36-3cc606912204", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "2eb2574a-0c7b-4a64-aba1-83e0fd407f6a:indexpattern-datasource-layer-f845e60c-45fb-44c5-a80d-d16335bc82df", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "1974815d-9fa9-4189-93ec-ca60f0b44a26:indexpattern-datasource-layer-f78bc1be-327e-43c6-b7f9-a35a37f35313", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "6a30acd7-711b-482d-8283-76b6be2127cf:indexpattern-datasource-layer-18831400-03d9-4b80-a77e-184b46f64567", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "3cac33c9-2ecb-4a73-96eb-871e96ffda0a:indexpattern-datasource-layer-dd697971-f8aa-44aa-b590-d034ea804fde", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "092c00d7-b687-4efc-ae6d-09e44293a878:indexpattern-datasource-layer-bc945500-30d4-4e6f-8563-37bfdc7754c2", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "b331d751-e8a6-4dd2-ad80-38f5bdaed8c7:indexpattern-datasource-layer-2f1d5bd6-512d-458c-91d1-93c0bec36a07", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "controlGroup_11023e32-443a-491a-b014-5a3b491df250:optionsListDataView", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "controlGroup_af3850da-13a7-4b27-8ea4-be9dd8a5599c:optionsListDataView", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + } + ], + "type": "dashboard", + "typeMigrationVersion": "10.2.0", + "updated_by": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0" +} \ No newline at end of file diff --git a/packages/ti_flashpoint/manifest.yml b/packages/ti_flashpoint/manifest.yml new file mode 100644 index 00000000000..ad850a2313e --- /dev/null +++ b/packages/ti_flashpoint/manifest.yml @@ -0,0 +1,95 @@ +format_version: 3.3.2 +name: ti_flashpoint +title: Flashpoint +version: 0.1.0 +description: Collect logs from Flashpoint with Elastic Agent. +type: integration +categories: + - security +conditions: + kibana: + version: ^8.18.0 || ^9.0.0 + elastic: + subscription: basic +screenshots: + - src: /img/ti_flashpoint-alerts_dashboard.png + title: Alerts Dashboard + size: 600x600 + type: image/png +icons: + - src: /img/ti_flashpoint-logo.svg + title: Flashpoint logo + size: 32x32 + type: image/svg+xml +policy_templates: + - name: ti_flashpoint + title: Flashpoint + description: Collect logs from Flashpoint. + deployment_modes: + default: + enabled: true + agentless: + enabled: true + organization: security + division: engineering + team: security-service-integrations + inputs: + - type: cel + title: Collect logs from Flashpoint API + description: Collecting logs via Flashpoint API. + vars: + - name: url + type: url + title: URL + description: Base URL of the Flashpoint Instance. + multi: false + required: true + show_user: false + default: https://api.flashpoint.io + - name: api_token + type: password + title: API Token + secret: true + description: API Token to authenticate with Flashpoint API. + multi: false + required: true + show_user: true + - name: proxy_url + type: text + title: Proxy URL + description: URL to proxy connections in the form of http[s]://:@:. Please ensure your username and password are in URL encoded format. + multi: false + required: false + show_user: false + - name: ssl + type: yaml + title: SSL Configuration + description: SSL configuration options. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-ssl.html#ssl-common-config) for details. + multi: false + required: false + show_user: false + default: | + #certificate_authorities: + # - | + # -----BEGIN CERTIFICATE----- + # MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF + # ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2 + # MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB + # BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n + # fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl + # 94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t + # /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP + # PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41 + # CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O + # BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux + # 8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D + # 874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw + # 3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA + # H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu + # 8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0 + # yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk + # sxSmbIUfc2SGJGCJD4I= + # -----END CERTIFICATE----- +owner: + github: elastic/security-service-integrations + type: elastic