release-1038 #40
release-1038 #40
Conversation
|
If you'd like someone to review syntax, logic. Someone like @ebonura-fastly will be good for that. However all we're really doing (outside the auto generations) is added a few functions, and making sure the float values that configure the timeouts make it to the http client and some validations which I've tested. I've unit tested it, again for a few various use-cases, and with around 20k requests worth of data. These match up with the requests manually. |
| request_limit = The amount of request objects returned in the array. Default: 100. Max:1000 | ||
| disable_catchup = Disables catch-up behavior. Request feed will always be ingested from now and the delta (and offset). We recommend keeping this as checked for request feeds with large amounts of requests. | ||
| twenty_hour_catchup = In the event the last time stored is >24hours the TA will try can try and catch-up from exactly 24 hours ago, otherwise resets to now - delta. Disable catchup must be false in order to work. | ||
| attack_and_anomaly_signals_only = Only retrieves requests that contain attack or anomaly signals. Please evaluate your signal configuration if there are overly inclusive signals creating excessive requests. | ||
| request_timeout = Configures Request Timeout for HTTP operations. Consider increasing if on a slow connection or pagination batches are large. | ||
| read_timeout = Configures Read Timeout for HTTP operations. Consider increasing if on a slow connection or pagination batches are large. | ||
|
|
||
| [SigsciEvent://<name>] | ||
| site_api_name = This is the Site API Name. It should not be a URL. | ||
| disable_catchup = Time is always set based from now - delta (Interval). Recommended to be True. Default: True. | ||
| twenty_hour_catchup = If last stored timestamp was over 24 hours, resets to exactly 24 hours ago instead to meet API limitations. | ||
| request_timeout = Configures Request Timeout for HTTP operations. Consider increasing if on a slow connection or pagination batches are large. | ||
| read_timeout = Configured Read Timeout for HTTP operations. Consider increasing if on a slow connection or pagination batches are large. | ||
|
|
||
| [SigsciActivity://<name>] | ||
| place_holder = It was required to have one option even if it isn't needed. You can skip this one. No newline at end of file | ||
| disable_catchup = | ||
| twenty_hour_catchup = | ||
| request_timeout = Configures Request Timeout for HTTP operations. Consider increasing if on a slow connection or pagination batches are large. | ||
| read_timeout = Configures Read Timeout for HTTP operations. Consider increasing if on a slow connection or pagination batches are large. |
There was a problem hiding this comment.
I recommend normalizing all the docs here so they have the same text for the same items. Also include the default and recommended values for each (right now some have them and some not).
There was a problem hiding this comment.
twenty_hour_catchup is an odd name for 24-hour catchup. Consider calling it something like max_interval_catchup or full_day_catchup? Not critical, though.
sigsci_TA_for_splunk/appserver/static/js/build/globalConfig.json
Outdated
Show resolved
Hide resolved
| if attack_and_anomaly_signals_only: | ||
| attack_signals = [ | ||
| "USERAGENT", | ||
| "AWS-SSRF", | ||
| "BACKDOOR", | ||
| "CMDEXE", | ||
| "SQLI", | ||
| "TRAVERSAL", | ||
| "XSS", | ||
| "XXE" | ||
| ] | ||
| anomaly_signals = [ | ||
| "2FA-DISABLED", "2FA-CHANGED", "ABNORMALPATH", "ADDRESS-CHANGED", "ALLOWED", | ||
| "BHH", "BLOCKED", "BODY-PARSER-EVASION", "CODEINJECTION", "COMPRESSED", | ||
| "CC-VAL-ATTEMPT", "CC-VAL-FAILURE", "CC-VAL-SUCCESS", "CVE-2017-5638", | ||
| "CVE-2017-7269", "CVE-2017-9805", "CVE-2018-11776", "CVE-2018-15961", | ||
| "CVE-2018-9206", "CVE-2019-0192", "CVE-2019-0193", "CVE-2019-0232", | ||
| "CVE-2019-11580", "CVE-2019-14234", "CVE-2019-16759", "CVE-2019-2725", | ||
| "CVE-2019-3396", "CVE-2019-3398", "CVE-2019-5418", "CVE-2019-6340", | ||
| "CVE-2019-8394", "CVE-2019-8451", "CVE-2021-26084", "CVE-2021-26855", | ||
| "CVE-2021-40438", "CVE-2021-44228", "CVE-2021-44228-STRICT", | ||
| "CVE-2022-22963", "CVE-2022-22965", "CVE-2022-26134", "CVE-2022-42889", | ||
| "CVE-2023-34362", "CVE-2023-38218", "DATACENTER", "DOUBLEENCODING", | ||
| "EMAIL-CHANGED", "EMAIL-VALIDATION", "FORCEFULBROWSING", "GC-VAL-ATTEMPT", | ||
| "GC-VAL-FAILURE", "GC-VAL-SUCCESS", "GRAPHQL-API", "GRAPHQL-DUPLICATE-VARIABLES", | ||
| "GRAPHQL-IDE", "GRAPHQL-INTROSPECTION", "GRAPHQL-DEPTH", | ||
| "GRAPHQL-MISSING-REQUIRED-OPERATION-NAME", | ||
| "GRAPHQL-UNDEFINED-VARIABLES", "HTTP403", "HTTP404", "HTTP429", | ||
| "HTTP4XX", "HTTP500", "HTTP503", "HTTP5XX", "IMPOSTOR", "INFO-VIEWED", | ||
| "INSECURE-AUTH", "NOTUTF8", "INVITE-FAILURE", "INVITE-ATTEMPT", | ||
| "INVITE-SUCCESS", "JSON-ERROR", "KBA-CHANGED", "LOGINATTEMPT", | ||
| "LOGINDISCOVERY", "LOGINFAILURE", "LOGINSUCCESS", "MALFORMED-DATA", | ||
| "SANS", "MESSAGE-SENT", "NO-CONTENT-TYPE", "NOUA", "NULLBYTE", | ||
| "OOB-DOMAIN", "PW-CHANGED", "PW-RESET-ATTEMPT", "PW-RESET-FAILURE", | ||
| "PW-RESET-SUCCESS", "PRIVATEFILE", "rate-limit", "REGATTEMPT", "REGFAILURE", | ||
| "REGSUCCESS", "RSRC-ID-ENUM-ATTEMPT", "RSRC-ID-ENUM-FAILURE", | ||
| "RSRC-ID-ENUM-SUCCESS", "RESPONSESPLIT", "SCANNER", "SIGSCI-IP", | ||
| "TORNODE", "WRONG-API-CLIENT", "USER-ID-ENUM-ATTEMPT", | ||
| "USER-ID-ENUM-FAILURE", "USER-ID-ENUM-SUCCESS", "WEAKTLS", "XML-ERROR" | ||
| ] | ||
| attack_tags = ",".join(attack_signals) | ||
| anomaly_tags = ",".join(anomaly_signals) | ||
| url = f"{url}&tags={attack_tags},{anomaly_tags}" |
There was a problem hiding this comment.
This may be a bit messy to keep in sync if these are all hard-coded.
There was a problem hiding this comment.
Yep. I wanted to make an API call to populate that array. I'm not too sure what to do about this feature, I honestly think its an anti pattern and don't want it in here - it was just a request that was made for a prospect, unless I write them a side build that contains it. That may actually be better than introducing more problems. They are not on Splunk cloud, so i can just tarball it for them.
Adds the following features:
Adds toggle to disabling catching up, ideally should have done this since the beginning to avoid stale states. When this is turned on, from/until times are always calculated from now - delta. Can also be used to save anything stuck without having to dive into the kv store.
Adds configurable connect/read timeouts to each input for the HTTP client. I've opted to add this per input due to the fact global configuration parameters cannot be validated, for splunk cloud users, that's problematic given the lack of logs. These have limitations of 300 seconds, but should unblock a few customers who were hitting limits sometimes.
Adds some extra configuration around catchup if it is enabled, either to reset to now - delta (default) if the timestamp is older than 24 hours, or exactly 24 hours ago.
Adds the ability to query only attack / anomaly signals for people who catch all request feeds, this was added as a request for a separate build that was provided to a customer, so I've left it in for when they upgrade. It is not recommended to be used, but it can reduce traffic. This is problematic if any new signals are added.
Handles the POST parameter changes to the Feed Endpoint. Pagination is now done via POST parameters rather than query params for the request feed endpoint.
I would prefer to gut any ability to catchup all together, but atleast its an option and it's set as the default behavior now.