Merge pull request SigmaHQ#990 from neu5ron/es_backend

ES and Readme from SOC Prime

tools/README.md

@@ -291,17 +291,28 @@ You can add the following depending on additional information from your answers/
     * If you want to prevent case sensitive bypasses you can add the following to your command: `--backend-option case_insensitive_whitelist="*"`
     * If you want to prevent case sensitive bypasses but only for certain fields, you can use an option like this: `-backend-option keyword_field="" --backend-option case_insensitive_whitelist="*CommandLine*, *ProcessName*, *Image*, process.*, *FileName*, *Path*, *ServiceName*, *ShareName*, file.*, *Directory*, *directory*, *hash*, *Hash*, *Object*, ComputerName, *Subject*, *Target*, *Service*"`
 
-2. If you are using analyzed (text) fields or your index template portion of `strings_as_keyword` contains `text` then you can add the following:
+1. If you are using analyzed (text) fields or your index template portion of `strings_as_keyword` contains `text` then you can add the following:
 
-```bash
---backend-option keyword_base_fields="*" --backend-option analyzed_sub_field_name=".text"
-```
+    ```bash
+    --backend-option keyword_base_fields="*" --backend-option analyzed_sub_field_name=".text"
+    ```
 
-3. If you only have some analyzed fields then you would use an example like this:
+1. If you only have some analyzed fields then you would use an example like this:
 
-```bash
---backend-option keyword_base_fields="*" --backend-option analyzed_sub_field_name=".text" --backend-option analyzed_sub_fields="TargetUserName, SourceUserName, TargetHostName, CommandLine, ProcessName, ParentProcessName, ParentImage, Image"
-```
+    ```bash
+    --backend-option keyword_base_fields="*" --backend-option analyzed_sub_field_name=".text" --backend-option analyzed_sub_fields="TargetUserName, SourceUserName, TargetHostName, CommandLine, ProcessName, ParentProcessName, ParentImage, Image"
+    ```
+
+1. If you only have some analyzed fields then you would use an example like this:
+
+    ```bash
+    --backend-option keyword_base_fields="*" --backend-option analyzed_sub_field_name=".text" --backend-option analyzed_sub_fields="TargetUserName, SourceUserName, TargetHostName, CommandLine, ProcessName, ParentProcessName, ParentImage, Image"
+    ```
+1. Use an analyzed field or different field for queries that contain wildcard(s)
+
+    ```bash
+    --backend-option wildcard_use_keyword="false"
+    ```
 
 ### Elastic - Some Final Examples
 
@@ -310,23 +321,29 @@ So putting it all together to help show everything from above, here are some "fu
 * base field keyword & no analyzed field w/ case insensitivity (covers elastic 7 with beats/ecs (default)mappings) and using winlogbeat with modules enabled. Also, keeps `winlog.channel` from making case insensitive as is not necessary (ie: the `keyword_whitelist` option)
 
 ```bash
-sigma -t es-qs -c tools/config/winlogbeat-modules-enabled.yml --backend-option keyword_field="" --backend-option case_insensitive_whitelist="*" --backend-option keyword_whitelist="winlog.channel" rules/windows/process_creation/win_office_shell.yml
+tools/sigmac -t es-qs -c tools/config/winlogbeat-modules-enabled.yml --backend-option keyword_field="" --backend-option case_insensitive_whitelist="*" --backend-option keyword_whitelist="winlog.channel" rules/windows/process_creation/win_office_shell.yml
 ```
 
 * base field keyword & subfield is analyzed(.text) and winlogbeat with modules enabled
 
 ```bash
-sigma -t es-qs -c tools/config/winlogbeat-modules-enabled.yml --backend-option keyword_base_fields="*" --backend-option analyzed_sub_field_name=".text" rules/windows/process_creation/win_office_shell.yml
+tools/sigmac -t es-qs -c tools/config/winlogbeat-modules-enabled.yml --backend-option keyword_base_fields="*" --backend-option analyzed_sub_field_name=".text" rules/windows/process_creation/win_office_shell.yml
 ```
 
 * base field keyword & only some analyzed fields and winlogbeat without modules enabled
 
 ```bash
-tools/sigmac -t es-dsl -c tools/config/winlogbeat.yml  --backend-option keyword_base_fields="*" --backend-option analyzed_sub_field_name=".text" --backend-option analyzed_sub_fields="TargetUserName, SourceUserName, TargetHostName, CommandLine, ProcessName, ParentProcessName, ParentImage, Image" rules/windows/process_creation/win_office_shell.yml
+tools/sigmac -t es-qs -c tools/config/winlogbeat.yml  --backend-option keyword_base_fields="*" --backend-option analyzed_sub_field_name=".text" --backend-option analyzed_sub_fields="TargetUserName, SourceUserName, TargetHostName, CommandLine, ProcessName, ParentProcessName, ParentImage, Image" rules/windows/process_creation/win_office_shell.yml
 ```
 
 * using beats/ecs Elastic 7 with case insensitive and some .text fields and winlogbeat without modules enabled
 
 ```bash
-tools/sigmac -t es-dsl -c tools/config/winlogbeat.yml --backend-option keyword_base_fields="*" --backend-option analyzed_sub_field_name=".text" --backend-option keyword_whitelist="winlog.channel,winlog.event_id" --backend-option case_insensitive_whitelist="*" --backend-option analyzed_sub_fields="TargetUserName, SourceUserName, TargetHostName, CommandLine, ProcessName, ParentProcessName, ParentImage, Image" rules/windows/process_creation/win_office_shell.yml
+tools/sigmac -t es-qs -c tools/config/winlogbeat.yml --backend-option keyword_base_fields="*" --backend-option analyzed_sub_field_name=".text" --backend-option keyword_whitelist="winlog.channel,winlog.event_id" --backend-option case_insensitive_whitelist="*" --backend-option analyzed_sub_fields="TargetUserName, SourceUserName, TargetHostName, CommandLine, ProcessName, ParentProcessName, ParentImage, Image" rules/windows/process_creation/win_office_shell.yml
+```
+
+* using keyword as a subfield and custom analyzed field as a subfield with winlogbeat mappings
+
+```bash
+tools/sigmac -t es-qs -c tools/config/winlogbeat.yml --backend-option keyword_field=".keyword" --backend-option analyzed_sub_field_name=".security" rules/windows/sysmon/sysmon_wmi_susp_scripting.yml
 ```

tools/sigma/backends/elasticsearch.py

@@ -20,6 +20,7 @@
 import sys
 import os
 from random import randrange
+from distutils.util import strtobool 
 
 import sigma
 import yaml
@@ -64,7 +65,8 @@ class ElasticsearchWildcardHandlingMixin(object):
             ("keyword_whitelist", None, "Fields to always set as keyword. Bypasses case insensitive options. Valid options are: list of fields, single field. Also, wildcards * and ? allowed.", None),
             ("keyword_blacklist", None, "Fields to never set as keyword (ie: always set as analyzed field). Bypasses case insensitive options. Valid options are: list of fields, single field. Also, wildcards * and ? allowed.", None),
             ("case_insensitive_whitelist", None, "Fields to make the values case insensitive regex. Automatically sets the field as a keyword. Valid options are: list of fields, single field. Also, wildcards * and ? allowed.", None),
-            ("case_insensitive_blacklist", None, "Fields to exclude from being made into case insensitive regex. Valid options are: list of fields, single field. Also, wildcards * and ? allowed.", None)
+            ("case_insensitive_blacklist", None, "Fields to exclude from being made into case insensitive regex. Valid options are: list of fields, single field. Also, wildcards * and ? allowed.", None),
+            ("wildcard_use_keyword", "true", "Use analyzed field or wildcard field if the query uses a wildcard value (ie: '*mall_wear.exe'). Set this to 'False' to use analyzed field or wildcard field. Valid options are: true/false", None)
             )
     reContainsWildcard = re.compile("(?:(?<!\\\\)|\\\\\\\\)[*?]").search
     uuid_regex = re.compile( "[0-9a-fA-F]{8}(\\\)?-[0-9a-fA-F]{4}(\\\)?-[0-9a-fA-F]{4}(\\\)?-[0-9a-fA-F]{4}(\\\)?-[0-9a-fA-F]{12}", re.IGNORECASE )
@@ -99,6 +101,10 @@ def __init__(self, *args, **kwargs):
             self.case_insensitive_blacklist = self.case_insensitive_blacklist.replace(' ','').split(',')
         except AttributeError:
             self.case_insensitive_blacklist = list()
+        try:
+            self.wildcard_use_keyword = strtobool(self.wildcard_use_keyword.lower().strip())
+        except AttributeError:
+            self.wildcard_use_keyword = False
 
     def containsWildcard(self, value):
         """Determine if value contains wildcard."""
@@ -154,10 +160,8 @@ def fieldNameMapping(self, fieldname, value, *agg_option):
             keyword_subfield_name = '.%s'%keyword_subfield_name
 
         # Set naming for analyzed fields
-        if analyzed_subfield_name != '' and not keyword_subfield_name.startswith('.'):
+        if analyzed_subfield_name != '':
             analyzed_subfield_name = '.%s'%analyzed_subfield_name
-        else:
-            analyzed_subfield_name = ''
 
         # force keyword on agg_option used in Elasticsearch DSL query key
         if agg_option:
@@ -191,7 +195,7 @@ def fieldNameMapping(self, fieldname, value, *agg_option):
             self.matchKeyword = True
         elif self.CaseInSensitiveField:
             self.matchKeyword = True
-        elif (type(value) == list and any(map(self.containsWildcard, value))) or self.containsWildcard(value):
+        elif self.wildcard_use_keyword and ( (type(value) == list and any(map(self.containsWildcard, value))) or self.containsWildcard(value) ):
             self.matchKeyword = True
         else:
             self.matchKeyword = False
@@ -213,6 +217,8 @@ def makeCaseInSensitiveValue(self, value):
             #value = re.sub( r"((?<!\\)(\\))\*$", "\g<1>\\*", value )
             # Make upper/lower
             value = re.sub( r"[A-Za-z]", lambda x: "[" + x.group( 0 ).upper() + x.group( 0 ).lower() + "]", value )
+            # Turn `.` into wildcard, only if odd number of '\'(because this would mean already escaped)
+            value = re.sub( r"(((?<!\\)(\\\\)+)|(?<!\\))\.", "\g<1>\.", value )
             # Turn `*` into wildcard, only if odd number of '\'(because this would mean already escaped)
             value = re.sub( r"(((?<!\\)(\\\\)+)|(?<!\\))\*", "\g<1>.*", value )
             # Escape additional values that are treated as specific "operators" within Elastic. (ie: @, ?, &, <, >, and ~)