Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Validation of the YAML structure of query files #660

Merged
merged 36 commits into from
Sep 26, 2023
Merged

Validation of the YAML structure of query files #660

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
37808d6
Update first version of query schema
Furoxu Apr 29, 2023
0b7c7dd
Adding JSON Schema library to dependencies
Furoxu Apr 29, 2023
6f9f917
Add tests to validate the correctness of existing queries
Furoxu Apr 29, 2023
5f07ad6
Fix typos & missing description in queries
Furoxu Apr 29, 2023
5e29316
Convert path to Path type
FlorianBracq Apr 29, 2023
c3c6c85
If path is not a dir, use relative path from config file
FlorianBracq Apr 29, 2023
599b9b9
Update validation method to handle expected fails
FlorianBracq Apr 30, 2023
e4e7da7
Fix path resolution for custom queries
FlorianBracq Apr 30, 2023
e30ccd8
Add missing keys
FlorianBracq Apr 30, 2023
9c9c1c2
Fix incorrect type names
FlorianBracq Apr 30, 2023
5d3808d
Remove empty parameters from query files
FlorianBracq Apr 30, 2023
8f106c4
Merge branch 'main' into yaml-validation
ianhelle Jul 6, 2023
87a0682
Merge branch 'main' of https://github.com/microsoft/msticpy into yaml…
FlorianBracq Jul 20, 2023
a04df4d
Fixing typo
FlorianBracq Jul 20, 2023
9e9c553
Adding new metadata fields
FlorianBracq Jul 20, 2023
09fc21c
Remove empty parameters
FlorianBracq Jul 20, 2023
900416f
Add SQL data environment
FlorianBracq Jul 20, 2023
ac60367
Ignore folder containing Sentinel queries
FlorianBracq Jul 20, 2023
051ca61
Fix call to absolute
FlorianBracq Aug 1, 2023
bf87c1f
Update JSON validation schema to have an open list of providers
FlorianBracq Aug 1, 2023
77acc04
Replacing tabs with spaces
FlorianBracq Aug 1, 2023
6732d7b
s/oneOf/anyOf/
FlorianBracq Aug 1, 2023
d0d8e8d
Rework condition to ignore sentinel_query_import_data
FlorianBracq Aug 1, 2023
e32f31f
Simplify comparison for python 3.8
FlorianBracq Aug 1, 2023
b0d2696
Increasing the size of description to 1024 characters
FlorianBracq Aug 3, 2023
debd5f7
Merge branch 'main' into yaml-validation
ianhelle Aug 7, 2023
2b08b37
Merge branch 'main' of https://github.com/Microsoft/msticpy into yaml…
FlorianBracq Aug 18, 2023
f75587b
Adding metadata, sources and defaults as mandatory
FlorianBracq Aug 18, 2023
8903a70
Ignore msticpyconfig files
FlorianBracq Aug 18, 2023
fd7dd45
Remove defaults as a required key
FlorianBracq Sep 5, 2023
0c67cc7
Created dedicated tests for query validation
FlorianBracq Sep 5, 2023
2830580
Add multiple sample of valid and invalid queries
FlorianBracq Sep 5, 2023
1c6bc58
Update test_pkg_config based on commit's comments
FlorianBracq Sep 5, 2023
efeae91
Merge branch 'main' into yaml-validation
FlorianBracq Sep 5, 2023
2957607
Merge branch 'main' into yaml-validation
ianhelle Sep 5, 2023
0c9ff2d
Merge branch 'main' into yaml-validation
ianhelle Sep 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
270 changes: 270 additions & 0 deletions .schemas/queries.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,270 @@
{
"$schema": "http://json-schema.org/draft-07/schema",
"type": "object",
"properties": {
"metadata": {
"$ref": "#/$defs/metadata"
},
"defaults": {
"type": "object",
"properties": {
"metadata": {
"type": "object",
"properties": {
"data_source": {
"type": "string"
},
"data_families": {
"type": "array",
"items": {
"type": "string"
}
},
"pivot": {
"type": "object",
"patternProperties": {
".*": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
"top": {
"type": "integer"
}
}
},
"parameters": {
"$ref": "#/$defs/parameter"
}
}
},
"sources": {
"type": "object",
"patternProperties": {
".*": {
"$ref": "#/$defs/query"
}
}
}
},
"required": [
"metadata",
"sources"
],
"$defs": {
"description": {
"type": "string",
"minLength": 5,
"maxLength": 1024
},
"metadata": {
"type": "object",
"properties": {
"version": {
"type": "integer"
},
"description": {
"$ref": "#/$defs/description"
},
"data_environments": {
"type": "array",
"items": {
"anyOf": [
{
"enum": [
"AzureSecurityCenter",
"AzureSentinel",
"Cybereason",
"Elastic",
"Kusto",
"LocalData",
"LogAnalytics",
"M365D",
"MDATP",
"MDE",
"Mordor",
"MSGraph",
"MSSentinel",
"OSQueryLogs",
"OTRF",
"ResourceGraph",
"SecurityGraph",
"Splunk",
"Sumologic"
]
},
{
"type": "string"
}
]
}
},
"data_families": {
"type": "array",
"items": {
"type": "string"
}
},
"tags": {
"type": "array",
"items": {
"type": "string"
}
},
"aliases": {
"oneOf": [
{
"type": "array",
"items": {
"type": "string"
}
},
{
"type": "string"
}
]
},
"cluster": {
"type": "string"
},
"clusters": {
"type": "array",
"items": {
"type": "string"
}
},
"cluster_groups": {
"type": "array",
"items": {
"type": "string"
}
},
"database": {
"type": "string"
}
},
"additionalProperties": false,
"required": [
"version",
"description",
"data_environments",
"data_families"
]
},
"parameter": {
"type": "object",
"patternProperties": {
".*": {
"type": "object",
"properties": {
"description": {
"$ref": "#/$defs/description"
},
"type": {
"type": "string",
"enum": [
"str",
"datetime",
"int",
"float",
"list"
]
},
"default": {
"oneOf": [
{
"type": "string"
},
{
"type": "array"
},
{
"type": "integer"
}
]
},
"aliases": {
"oneOf": [
{
"type": "array",
"items": {
"type": "string"
}
},
{
"type": "string"
}
]
}
},
"additionalProperties": false,
"required": [
"description",
"type"
]
}
}
},
"query": {
"type": "object",
"properties": {
"description": {
"$ref": "#/$defs/description"
},
"metadata": {
"anyOf": [
{
"$ref": "#/$defs/metadata"
},
true
]
},
"parameters": {
"$ref": "#/$defs/parameter"
},
"args": {
"type": "object",
"properties": {
"query": {
"type": "string"
},
"uri": {
"type": "string"
}
},
"required": [
"query"
]
},
"query_macros": {
"patternProperties": {
".*": {
"type": "object",
"properties": {
"description": {
"$ref": "#/$defs/description"
},
"value": {
"type": "string"
}
},
"additionalProperties": false,
"required": [
"description",
"value"
]
}
}
}
},
"additionalProperties": false,
"required": [
"description",
"args"
]
}
}
}
4 changes: 2 additions & 2 deletions msticpy/data/queries/cybereason/cybereason_processes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ sources:
parameters:
command:
description: Command to search for
type: string
type: str
find_process_by_suspicions:
description: Search for process with a specific suspicion
metadata:
Expand Down Expand Up @@ -108,7 +108,7 @@ sources:
parameters:
suspicion:
description: Suspicion that the process should have
type: string
type: str
find_process_by_pid:
description: Search for a process by pid and hostname
metadata:
Expand Down
10 changes: 0 additions & 10 deletions msticpy/data/queries/localdata/local_data.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,68 +7,58 @@ metadata:
defaults:
metadata:
data_source: 'security_alert'
parameters:
ianhelle marked this conversation as resolved.
Show resolved Hide resolved
sources:
list_alerts:
description: Retrieves list of alerts
metadata:
data_families: [SecurityAlert]
args:
query: alerts_list.pkl
parameters:
list_host_processes:
description: List processes on host
metadata:
data_families: [WindowsSecurity]
args:
query: processes_on_host.pkl
parameters:
list_host_logons:
description: List logons on host
metadata:
data_families: [WindowsSecurity]
args:
query: host_logons.pkl
parameters:
list_host_logon_failures:
description: List logon failures on host
metadata:
data_families: [WindowsSecurity]
args:
query: failed_logons.pkl
parameters:
list_host_events:
description: List events failures on host
metadata:
data_families: [WindowsSecurity]
args:
query: all_events_df.pkl
parameters:
get_process_tree:
description: Get process tree for a process
metadata:
data_families: [WindowsSecurity]
args:
query: process_tree.pkl
parameters:
list_azure_network_flows_by_ip:
description: List Azure Network flows by IP address
metadata:
data_families: [Network]
args:
query: az_net_comms_df.pkl
parameters:
list_azure_network_flows_by_host:
description: List Azure Network flows by host name
metadata:
data_families: [Network]
args:
query: az_net_comms_df.pkl
parameters:
list_all_signins_geo:
description: List all Azure AD logon events
metadata:
data_families: [Azure]
args:
query: aad_logons.pkl
parameters:
4 changes: 1 addition & 3 deletions msticpy/data/queries/m365d/kql_m365_alerts.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ sources:
| where Timestamp <= datetime({end})
{add_query_items}"
uri: None
parameters:
list_alerts_with_evidence:
description: Retrieves list of alerts with their evidence
metadata:
Expand All @@ -50,7 +49,6 @@ sources:
SourceSystem1, Type1
{add_query_items}"
uri: None
parameters:
host_alerts:
description: Lists alerts associated with host/device name
metadata:
Expand Down Expand Up @@ -284,4 +282,4 @@ sources:
parameters:
app_name:
description: The name of the application
type: str
type: str
Loading