Merge branch 'master' into devel

CHANGELOG.md

@@ -0,0 +1,90 @@
+# Release Notes
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
+from version 0.14.0.
+
+## Unreleased
+
+Changes from this section will be contained in the next release.
+
+### Added
+
+* sigma-similarity tool
+* LimaCharlie backend
+* Default configurations for some backends that are used if no configuration is passed.
+* Regular expression support for es-dsl backend (propagates to backends derived from this like elastalert-dsl)
+* Value modifiers:
+    * startswith
+    * endswith
+
+### Changed
+
+* Removal of line breaks in elastalert output
+* Searches not bound to fields are restricted to keyword fields in es-qs backend
+* Graylog backend now based on es-qs backend
+
+## 0.13
+
+### Added
+
+* Index mappings for Sumologic
+* Malicious cmdlets in wdatp
+* QRadar support for keyword searches
+* QRadar mapping improvements
+* QRadar field selection
+* QRadar type regex modifier support
+* Elasticsearch keyword field blacklisting with wildcards
+* Added dateField configuration parameter in xpack-watcher backend
+* Field mappings in configurations
+* Field name mapping for conditional fields
+* Value modifiers:
+    * utf16
+    * utf16le
+    * wide
+    * utf16be
+
+### Changed
+
+* Improved --backend-config help text
+
+### Fixed
+
+* Backend errors in ala
+* Slash escaping within es-dsl wildcard queries
+* QRadar backend config
+* QRadar field name and value escaping and handling
+* Elasticsearch wildcard detection pattern
+* Aggregation on keyword field in es-dsl backend
+
+## 0.12.1
+
+### Fixed
+
+* Missing build dependency
+
+## 0.12
+
+### Added
+
+* Usage of "Channel" field in ELK Windows configuration
+* Fields to mappings
+* xpack-watcher actions index and webhook
+* Config for Winlogbeat 7.x
+* Value modifiers
+* Regular expression support
+
+### Changed
+
+* Warning/error messages
+* Sumologic value cleaning
+* Explicit OR for Elasticsearch query strings
+* Listing of available configurations on missing configuration error
+
+### Fixed
+
+* Conditions in es-dsl backend
+* Sumologic handling of null values
+* Ignore timeframe detection keyword in all/any of conditions

CHANGELOG.md.j2

@@ -0,0 +1,38 @@
+## {{ version.minor }}.{{ version.major }}.{{ version.patch }} ({{ date }})
+
+### Added
+
+{% for item in added %}
+* {{ item | indent(2) }}
+{% endfor %}
+
+### Changed
+
+{% for item in changed %}
+* {{ item | indent(2) }}
+{% endfor %}
+
+### Deprecated
+
+{% for item in deprecated %}
+* {{ item | indent(2) }}
+{% endfor %}
+
+### Removed
+
+{% for item in removed %}
+* {{ item | indent(2) }}
+{% endfor %}
+
+### Fixed
+
+{% for item in fixed %}
+* {{ item | indent(2) }}
+{% endfor %}
+
+### Security
+
+{% for item in security %}
+* {{ item | indent(2) }}
+{% endfor %}
+

tests/test-modifiers.yml

@@ -13,4 +13,6 @@ detection:
             - foo
             - bar
             - bla
+        end|endswith: test
+        start|startswith: test
     condition: selection

tools/sigma/backends/elasticsearch.py

@@ -212,8 +212,6 @@ def escapeSlashes(self, value):
 
     def generateMapItemNode(self, node):
         key, value = node
-        if type(value) not in (str, int, list, type(None)):
-            raise TypeError("Map values must be strings, numbers, lists or null, not " + str(type(value)))
         if type(value) is list:
             res = {'bool': {'should': []}}
             for v in value:
@@ -230,7 +228,7 @@ def generateMapItemNode(self, node):
         elif value is None:
             key_mapped = self.fieldNameMapping(key, value)
             return { "bool": { "must_not": { "exists": { "field": key_mapped } } } }
-        else:
+        elif type(value) in (str, int):
             key_mapped = self.fieldNameMapping(key, value)
             if self.matchKeyword:   # searches against keyowrd fields are wildcard searches, phrases otherwise
                 queryType = 'wildcard'
@@ -239,6 +237,11 @@ def generateMapItemNode(self, node):
                 queryType = 'match_phrase'
                 value_cleaned = self.cleanValue(str(value))
             return {queryType: {key_mapped: value_cleaned}}
+        elif isinstance(value, SigmaRegularExpressionModifier):
+            key_mapped = self.fieldNameMapping(key, value)
+            return { 'regexp': { key_mapped: str(value) } }
+        else:
+            raise TypeError("Map values must be strings, numbers, lists, null or regular expression, not " + str(type(value)))
 
     def generateValueNode(self, node):
         return {'multi_match': {'query': node, 'fields': [], 'type': 'phrase'}}