diff --git a/.gitignore b/.gitignore index fab190b3b3..6c6b8a83bb 100644 --- a/.gitignore +++ b/.gitignore @@ -18,6 +18,10 @@ slack_bot_token_secret # Ignore daemon output files daemon/ +# Ignore private AGENTS.md, may contain private paths +private/AGENTS.md + + # Ignore the results folders 2019- 2020- diff --git a/.secrets.baseline b/.secrets.baseline index 86c4471ba3..6efec3d377 100644 --- a/.secrets.baseline +++ b/.secrets.baseline @@ -149,7 +149,7 @@ "filename": "config/slips.yaml", "hashed_secret": "4cac50cee3ad8e462728e711eac3e670753d5016", "is_verified": false, - "line_number": 278 + "line_number": 295 } ], "dataset/test14-malicious-zeek-dir/http.log": [ @@ -7185,5 +7185,5 @@ } ] }, - "generated_at": "2026-03-27T14:25:16Z" + "generated_at": "2026-04-08T14:13:03Z" } diff --git a/AGENTS.md b/AGENTS.md index 48cb9d39e9..db3770c38b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,42 +1,136 @@ # AGENTS.md -## Project overview -- Entry point: `slips.py` (starts the main process, spawns modules, runs in interactive/daemon modes). -- Core framework code lives in `slips/`, `slips_files/`, and `managers/`. -- Detection/analysis modules are in `modules/` (implement the `IModule` interface). -- Configuration is in `config/` (main config: `config/slips.yaml`). -- Tests live under `tests/` (unit + integration suites). -- Documentation is in `docs/` (see `docs/contributing.md` for contribution workflow, branching, and PR expectations). -- UIs/tools: `SlipsWeb/`, `webinterface/`, `webinterface.sh`, and `kalipso.sh`. - -## Build and test commands -- Run locally (no build step): - - `./slips.py -e 1 -f dataset/test7-malicious.pcap -o output_dir` -- Build the Docker image (from `docs/installation.md`): - - `docker build --no-cache -t slips -f docker/Dockerfile .` - - If build networking fails: `docker build --network=host --no-cache -t slips -f docker/Dockerfile .` -- Run the Docker image: - - `docker run -it --rm --net=host slips` - -## Code style guidelines -- Python formatting is enforced via pre-commit: - - Black with `--line-length 79` (see `.pre-commit-config.yaml`). - - Ruff is used for linting and autofixes. -- Keep docstrings at the top of files where present (pre-commit `check-docstring-first`). -- Maintain clean whitespace (no trailing whitespace, final newline). -- Follow existing module patterns (`IModule` in `slips_files/common/abstracts/module.py`). - -## Testing instructions -- The canonical test runner is `tests/run_all_tests.sh` (runs unit tests then integration tests). -- Equivalent manual sequence (from `tests/run_all_tests.sh`): - - `./slips.py -cc` - - `printf "0" | ./slips.py -k` - - `python3 -m pytest tests/ --ignore="tests/integration_tests" -n 7 -p no:warnings -vvvv -s` - - `python3 tests/destrctor.py` - - `./slips.py -cc` - - `printf "0" | ./slips.py -k` - - `python3 -m pytest -s tests/integration_tests/test_portscans.py -p no:warnings -vv` - - `python3 -m pytest -s tests/integration_tests/test_dataset.py -p no:warnings -vv` - - `python3 -m pytest -s tests/integration_tests/test_config_files.py -p no:warnings -vv` - - `printf "0" | ./slips.py -k` - - `./slips.py -cc` +## 1. Project Overview + +- Entry point: `slips.py` + - Starts the main process + - Spawns modules + - Supports interactive and daemon modes + +- Core code directories: + - `slips/` + - `slips_files/` + - `managers/` + +- Detection modules: + - Located in `modules/` + - Must implement `IModule` from: + `slips_files/common/abstracts/module.py` + +- Configuration: + - Main file: `config/slips.yaml` + +- Tests: + - Located in `tests/` + - Includes unit and integration tests + +- Documentation: + - Located in `docs/` + - Contribution guide: `docs/contributing.md` + +- UI / tools: + - `SlipsWeb/` + - `webinterface/` + - `webinterface.sh` + - `kalipso.sh` + +- Repository root: + - All commands MUST be executed from `StratosphereLinuxIPS/` + +--- + +## 2. Build and Run + +### to run slips locally +./slips.py -e 1 -f dataset/test7-malicious.pcap -o output_dir + +### Build Docker image +docker build --no-cache -t slips -f docker/Dockerfile . + +- If networking fails: + +docker build --network=host --no-cache -t slips -f docker/Dockerfile . + +### Run Docker container +docker run -it --rm --net=host slips + +## 3. Code Style Rules + +These rules MUST be followed: + +- No trailing whitespace +- File must end with a newline +- Docstring must be the first statement in a file (if present) +- Avoid using environment variables, use variables from slips/config.yaml instead. + +### Paths: +- NEVER use absolute paths +- ALWAYS use relative paths +### Files: +- If a non-debug file is created → MUST be added with git add +### Documentation: +If a feature is added → MUST update relevant docs in docs/ +### Functions: +- Every new function MUST include a docstring +Docstrings MUST include: +- Short description +- Parameters (if applicable) +- Return value (if applicable) + +## 4. Testing +- Canonical test runner +tests/run_all_tests.sh +## 5. Unit Test Update Workflow + +When instructed to "update unit tests", follow EXACTLY: + +Step 1 — Run tests +python3 -m pytest tests/unit/ \ + --ignore="tests/integration_tests" \ + -n 7 -p no:warnings -vvvv -s + +Step 2 — Identify failures +Collect ALL failing tests + +Step 3 — Fix tests +Update failing tests ONE BY ONE +Do NOT batch fixes + +Step 4 — Add missing tests for new files +For every new source file in the branch: + +- Mirror its directory under tests/unit/ + +- C/reate file: +test_.py +- Add unit tests for that file + +Step 5 — Add tests for new functions +- Identify functions added in this branch (not in origin/develop) +- Add unit tests for each new function + +Step 6 — Test structure rules +- MUST use @pytest.mark.parametrize when applicable + +EACH test MUST: +Start with object creation using module_factory + +Step 7 — Re-run tests +Run the same pytest command again +Ensure ALL tests pass + +Step 8 — Git tracking +If new test files were created → run: +git add + +Step 9 — Failure fallback +If tests are still failing and cannot be fixed: +STOP +Report the issue + +## 6. Custom Instructions +ALSO apply rules from: +private/AGENTS.md + +If conflicts occur: +Prefer private/AGENTS.md diff --git a/config/slips.yaml b/config/slips.yaml index d5a2e4f87a..43408606db 100644 --- a/config/slips.yaml +++ b/config/slips.yaml @@ -1,14 +1,13 @@ # This configuration file controls several aspects of the working of Slips. --- -modes: +output: # Define the file names for the default output. stdout: slips.log stderr: errors.log - logsfile: slips.log + logs: slips.log ############################# -# Parameters that can be also specified with modifiers in the command line parameters: # The verbosity is related to how much data you want to see about the @@ -77,6 +76,11 @@ parameters: # whitelist file, current commit and date metadata_dir: true + # This directory stores databases and runtime-generated files that must + # persist across different slips runs and should not be overwritten. + # this dir is inside slips root dir. + permanent_dir: permanent + # Default pcap packet filter. Used with zeek pcapfilter : 'ip or not ip' # If you want more important traffic and forget the multicast and broadcast # traffic, you can use @@ -112,7 +116,7 @@ parameters: # label: malicious # label: unknown label: normal - # If Zeek files are rotated or not to avoid running out of disk. + # determines if Zeek files are rotated or not to avoid running out of disk. # Zeek rotation is enabled by default when using an interface, # which means Slips will delete all Zeek log files after 1 day # of running @@ -126,17 +130,16 @@ parameters: # Whitespace between the numeric constant and time unit is optional. # Appending the letter s to the time unit in order to # pluralize it is also optional - # rotation_period = 30min - # rotation_period = 2hr - # rotation_period = 30sec - rotation_period: 1day + # default_rotation_interval: 30min + # default_rotation_interval: 2hr + default_rotation_interval: 30sec + #default_rotation_interval: 1day # How many days Slips keeps the rotated Zeek files before deleting them. # Value should be in days # set it to 0 day if you want to delete them immediately - # keep_rotated_files_for : 1 day + # keep_rotated_files_for : 0 day # keep_rotated_files_for : 2 day - # keep_rotated_files_for : 3 day keep_rotated_files_for: 1 day # How many minutes to wait for all modules to finish before killing them @@ -201,9 +204,9 @@ modules: # List of modules to ignore. By default we always ignore the template, # do not remove it from the list # Add the names of other modules that you want to disable - # (they all should be lowercase with no special characters). Example, - # threatintelligence, blocking, networkdiscovery, timeline, virustotal, - # rnnccdetection, flowmldetection, updatemanager + # (use module snake_case names). Example, + # threat_intelligence, blocking, network_discovery, timeline, virustotal, + # rnn_cc_detection, flow_ml_detection, update_manager disable: [template] # For each line in timeline file there is a timestamp. @@ -212,10 +215,10 @@ modules: timeline_human_timestamp: true ############################# -flowmldetection: +flow_ml_detection: # This is a module that uses machine learning for detection. # It can be used in train mode or test mode. - # The mode 'train' should be used to tell the flowmldetection module + # The mode 'train' should be used to tell the flow_ml_detection module # that the flows received are all for training. # A label should be provided in the [Parameters] section # mode : train @@ -226,7 +229,7 @@ flowmldetection: mode: test ############################# -bruteforcing: +brute_force_detector: # Minimum number of SSH attempts from one source to one destination # before Slips considers it brute forcing. ssh_attempt_threshold: 9 @@ -299,7 +302,7 @@ virustotal: ############################# threatintelligence: - # By default, slips starts without the TI files, and runs the Update Manager + # By default, slips starts without the TI files, and runs the update_manager # in the background. If this option is set to true, slips will not start # analyzing the flows until the update manager finished and all TI files are # loaded successfully. @@ -325,7 +328,7 @@ threatintelligence: # The remote TI files will be temporaly stored in this directory download_path_for_remote_threat_intelligence: modules/threat_intelligence/remote_data_files/ - # Update period of Threat Intelligence files. How often should Slips update + # Update period of threat_intelligence files. How often should Slips update # the IoCs. # The expected value is in seconds. # 1 day = 86400 seconds @@ -389,8 +392,8 @@ whitelists: local_whitelist_path: config/whitelist.conf ############################# -flowalerts: - # For the flowalerts module +flow_alerts: + # For the flow_alerts module # We need a thrshold to determine a long connection in seconds. # In Slips by default is 25 minutes long_connection_threshold: 1500 @@ -487,7 +490,7 @@ exporting_alerts: taxii_timeout: 10 ############################# -CESNET: +cesnet: # Slips also supports exporting and importing evidence in the IDEA format to/from # warden servers of CESNET organization in Czech Republic. send_alerts: false @@ -597,7 +600,7 @@ global_p2p: use_global_p2p: False iris_conf: config/iris_config.yaml bootstrapping_node: False - bootstrapping_modules: ["fidesModule", "irisModule"] + bootstrapping_modules: ["fides", "iris"] ############################# local_p2p: diff --git a/conftest.py b/conftest.py index d4aeb1c03c..889a70b337 100644 --- a/conftest.py +++ b/conftest.py @@ -23,7 +23,6 @@ parent_dir = os.path.dirname(current_dir) sys.path.insert(0, parent_dir) - # Suppress TensorFlow logs from C++ backend os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3" # 3 = ERROR # TensorFlow logs oneDNN messages even with TF_CPP_MIN_LOG_LEVEL=3. diff --git a/docs/FAQ.md b/docs/FAQ.md index 3619ac1b64..78468c8e17 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -12,7 +12,7 @@ If the tensorflow version you're using isn't compatible with your architecture, you will get the "Illegal instruction" error and slips will terminate. To fix this you can disable the modules that use tensorflow by adding -```rnn-cc-detection, flowmldetection``` to the ```disable``` key in ```config/slips.yaml``` +```rnn-cc-detection, flow_ml_detection``` to the ```disable``` key in ```config/slips.yaml``` ## Docker time is not in sync with that of the host diff --git a/docs/P2P.md b/docs/P2P.md index 699ecdb6a1..e22daf9fbe 100644 --- a/docs/P2P.md +++ b/docs/P2P.md @@ -112,7 +112,7 @@ development of new trust models and modelling behavior of the P2P network. To use the experiments, clone the https://github.com/stratosphereips/p2p4slips-experiments repository into -`modules/p2ptrust/testing/experiments`. +`modules/p2p_trust/testing/experiments`. The experiments run independently (outside of Slips) and start all processes that are needed, including relevant parts of Slips. @@ -148,6 +148,8 @@ The network then replies with a score and confidence for the IP. The higher the Once we get the score of the IP, we store it in the database, and we alert if the score of this IP is more than 0 (threat level=info). +The persistent local P2P runtime directory is stored under the directory configured by ```parameters.permanent_dir``` in ```config/slips.yaml```. By default, this is ```permanent/p2p_trust_runtime/```. + ### Answering the network's request about an IP diff --git a/docs/bruteforcing.md b/docs/brute_force_detector.md similarity index 77% rename from docs/bruteforcing.md rename to docs/brute_force_detector.md index 5bc23a0d9f..1ebd0356dd 100644 --- a/docs/bruteforcing.md +++ b/docs/brute_force_detector.md @@ -1,6 +1,6 @@ -# Bruteforcing Module +# Brute force detector Module -The `Bruteforcing` module detects SSH bruteforcing by combining repeated SSH sessions, Zeek SSH metadata, client software banners, and Zeek notice confirmations. +The `brute_force_detector` module detects SSH brute forcing by combining repeated SSH sessions, Zeek SSH metadata, client software banners, and Zeek notice confirmations. This module is loaded automatically by Slips like the rest of the modules in `modules/`, unless it is explicitly disabled in `config/slips.yaml`. @@ -35,7 +35,7 @@ It uses the following inputs: For each SSH flow, the module first checks the Zeek SSH authentication outcome: -- If `auth_success` is `true` or `T`, the flow is ignored for bruteforcing. +- If `auth_success` is `true` or `T`, the flow is ignored for `brute_force_detector`. - If `auth_attempts` is greater than `0`, that value is added to the bruteforce campaign counter. - If `auth_attempts` is `0` or missing, but the SSH session is not marked successful, the module counts the session as one suspected password attempt. @@ -43,7 +43,7 @@ The last rule is important for datasets where Zeek records repeated SSH handshak ### Threshold and Reporting -The default SSH bruteforcing threshold is `9` attempts. +The default SSH brute force detector threshold is `9` attempts. After the threshold is reached, the module does not alert on every new attempt. Instead, it uses sparse bucketed reporting so alerts become less frequent over time but never completely stop. With the default threshold, the alert points are: @@ -61,7 +61,7 @@ The evidence threat level is `medium`. Confidence grows with the number of attempted passwords: -- first bruteforcing evidence starts at the configured threshold +- first brute force detector evidence starts at the configured threshold - full confidence is reached at `30` attempts - suspicious SSH client banners add a small confidence bonus - a Zeek `SSH::Password_Guessing` notice acts as confirmation and promotes confidence using Zeek's confirmed connection count @@ -81,7 +81,7 @@ The module emits `PASSWORD_GUESSING` evidence with: Example description: ```text -SSH bruteforcing from 147.32.80.40 to 147.32.80.37 on SSH 902/tcp. Attempts observed: 24. Client banner: libssh libssh2_1.11.0 from software.log. Confidence: 0.89. by Slips +SSH brute force detector from 147.32.80.40 to 147.32.80.37 on SSH 902/tcp. Attempts observed: 24. Client banner: libssh libssh2_1.11.0 from software.log. Confidence: 0.89. by Slips ``` ## Zeek Confirmation @@ -89,17 +89,17 @@ SSH bruteforcing from 147.32.80.40 to 147.32.80.37 on SSH 902/tcp. Attempts obse If Zeek raises `SSH::Password_Guessing` in `notice.log`, the module: - emits an evidence immediately based on the notice -- stores the notice as confirmation for later bruteforcing evidence +- stores the notice as confirmation for later `brute_force_detector` evidence - uses the confirmed connection count from the Zeek notice to increase confidence -If Zeek does not generate `notice.log` for SSH password guessing, the module still detects bruteforcing from `ssh.log` and `software.log`. +If Zeek does not generate `notice.log` for SSH password guessing, the module still detects `brute_force_detector` events from `ssh.log` and `software.log`. ## Configuration The module currently exposes: ```yaml -bruteforcing: +brute_force_detector: ssh_attempt_threshold: 9 ``` @@ -107,9 +107,9 @@ This value is read from `config/slips.yaml`. ## Relationship With Flow Alerts -SSH bruteforcing is now handled by the `Bruteforcing` module. +SSH brute force detector is now handled by the `brute_force_detector` module. -The `Flow Alerts` module still handles: +The `flow_alerts` module still handles: - successful SSH detections - Zeek port-scan notices diff --git a/docs/contributing.md b/docs/contributing.md index 320e775ca1..67fdf1426a 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -141,6 +141,9 @@ The goal of suppressing errors by default is the most errors should be handled b ### How are the modules loaded? - All modules in the modules/ directory that implement the IModule interface are automatically imported by slips, for more technical details check the load_modules() function in managers/process_manager.py +- Module names must use snake_case and follow the `x_y_doer` style used by names such as `http_analyzer`. +- Set the module class `name` to the same snake_case identifier as the module directory and main file. +- Keep the module directory name, the main module file, the config section name, and any module references in docs aligned with that same snake_case name. ### There's some missing code in all modules, what's happening? @@ -187,8 +190,8 @@ Using one of these 3 ways Variables used in the trust evaluation and its accompanied processes, such as database-backup in persistent SQLite storage and memory persistent Redis database of Slips, are strings, integers and floats grouped into custom dataclasses. Aforementioned data classes can -be found in modules/fidesModule/model. The reader may find that all of the floating variables are in the interval <-1; 1> -and some of them are between <0; 1>, please refer to the modules/fidesModule/model directory. +be found in modules/fides/model. The reader may find that all of the floating variables are in the interval <-1; 1> +and some of them are between <0; 1>, please refer to the modules/fides/model directory. The Fides Module is designed to cooperate with a global-peer-to-peer module. The communication is done using Slips' Redis channel, for more information please refer to communication and messages sections above. @@ -248,8 +251,8 @@ redis_client.publish(channel, message) print(f"Message published to channel '{channel}'.") ``` -For more information about message handling, please also refer to modules/fidesModule/messaging/message_handler.py -and to modules/fidesModule/messaging/dacite/core.py for message parsing. +For more information about message handling, please also refer to modules/fides/messaging/message_handler.py +and to modules/fides/messaging/dacite/core.py for message parsing. ### **Communication** @@ -257,7 +260,7 @@ The module uses Slips' Redis to receive and send messages related to trust intel evaluation of trust in peers and alert message dispatch. **Used Channels** -modules/fidesModule/messaging/message_handler.py +modules/fides/messaging/message_handler.py | **Slips Channel Name** | **Purpose** | |-----------------|-------------------------------------------------------------------------| | `slips2fides` | Provides communication channel from Slips to Fides | @@ -282,4 +285,4 @@ For more details, the code [here](https://github.com/stratosphereips/fides/tree/ | `tl2nl_peers_reliability` | `fides2network` | NetworkBridge.send_peers_reliability(...) | Sends peer reliability, this message is only for network layer and is not dispatched to the network. | -Implementations of Fides_Module-network-communication can be found in ```modules/fidesModule/messaging/network_bridge.py```. +Implementations of Fides_Module-network-communication can be found in ```modules/fides/messaging/network_bridge.py```. diff --git a/docs/create_new_module.md b/docs/create_new_module.md index 62b1fc2820..b8bbda9e4e 100644 --- a/docs/create_new_module.md +++ b/docs/create_new_module.md @@ -61,6 +61,7 @@ cp -a modules/template modules/local_connection_detector ### Changing the Name of the Module Each module in Slips should have a name, author and description. +Use a snake_case module package and main file name in the `x_y_doer` style already used in the repository, for example `http_analyzer` or `local_connection_detector`. We should change the name inside the python file by finding the lines with the name and description in the class 'Module' and changing them: diff --git a/docs/exporting.md b/docs/exporting.md index 323721895c..a0b3ac1ea8 100644 --- a/docs/exporting.md +++ b/docs/exporting.md @@ -1,6 +1,6 @@ # Exporting Slips Alerts -Slips supports exporting alerts to other systems using different modules (ExportingAlerts, CESNET sharing etc.) +Slips supports exporting alerts to other systems using different modules (`exporting_alerts`, `cesnet`, etc.) For now the supported systems are: diff --git a/docs/fides_module.md b/docs/fides.md similarity index 92% rename from docs/fides_module.md rename to docs/fides.md index a8c92b4ed9..282972eb94 100644 --- a/docs/fides_module.md +++ b/docs/fides.md @@ -43,6 +43,8 @@ To enable it, change ```use_fides=False``` to ```use_fides=True``` in ```config/ And start Slips on your interface. +The Fides shared SQLite cache is stored under the directory configured by ```parameters.permanent_dir```. By default, it is created in ```permanent/databases/```, so it persists across different Slips runs. + ## How it works: Slips interacts with other slips peers for the following purposes: @@ -62,7 +64,7 @@ If a peer generates an alert based on evidence of an attack, it can alert other ## Logs Slips contains a minimal log file for reports received by other peers and peer updates in the ```output``` directory if not manually specified using the appropriate slips parameter upon start. -The custom logger ```modules/fidesModule/utils/logger.py``` code is used by the Fides Module for internal logging. +The custom logger ```modules/fides/utils/logger.py``` code is used by the Fides Module for internal logging. ## Implementation notes and credit The mathematical models for the trust evaluation were written by Lukáš Forst as part of his [Master Thesis](https://dspace.cvut.cz/handle/10467/101312). diff --git a/docs/flowalerts.md b/docs/flow_alerts.md similarity index 98% rename from docs/flowalerts.md rename to docs/flow_alerts.md index bf6ea58069..0b56591bd4 100644 --- a/docs/flowalerts.md +++ b/docs/flow_alerts.md @@ -1,4 +1,4 @@ -# Flow Alerts Module +# `flow_alerts` Module The module of flow alerts has several behavioral techniques to detect attacks by analyzing the content of each flow alone. @@ -195,13 +195,13 @@ section for more info Slips alerts when 3+ invalid SMTP login attempts occurs within 10s -## SSH Bruteforcing +## SSH brute_force_detector -SSH bruteforcing is documented in the dedicated `Bruteforcing` module page: +SSH brute force detector is documented in the dedicated `brute_force_detector` module page: -- [Bruteforcing Module](bruteforcing.md) +- [brute_force_detector Module](brute_force_detector.md) -The `Flow Alerts` module still detects successful SSH sessions, but SSH password guessing is no longer owned by `Flow Alerts`. +The `flow_alerts` module still detects successful SSH sessions, but SSH password guessing is no longer owned by `flow_alerts`. ## DGA diff --git a/docs/immune/immune_architecture.md b/docs/immune/immune_architecture.md index 8b7851dd2e..99d8eb607c 100644 --- a/docs/immune/immune_architecture.md +++ b/docs/immune/immune_architecture.md @@ -1,4 +1,4 @@ -# Architecture Design of Slips Immune +# Architecture Design of Slips Immune ## Introduction @@ -49,7 +49,7 @@ The human innate system is composed of a large group of cells and activities, bu The non-self detection theory of the human immune system says that there is no such thing as detecting the concept of self in cells. The only concept of self is trained during weak matching in the training of new T-cells and B-cells, to ensure they only receive pathogen parts from MHC Class II molecules. Apart from this, the human immune system does not have a concept of self. What it has is a concept of non-self, achieved by explicitly detecting the PAMPs of known pathogens. ### Detection of Non-self with PAMPs in Slips -Most of the current modules in Slips version <= 1.1.12 play the role of the innate system. These include: Input, Output, mlflowdetection, arp, cesnet, cyst, ensembling, exporting_alerts, http_analyzer, ip_info, irisModule, kalipso, leak_detector, network_discovery, p2ptrust, riskiq, rnn_cc_detection, threat_intelligence, virustotal, and update_manager. Some core parts of Slips are also part of the immune system, such as output. +Most of the current modules in Slips version <= 1.1.12 play the role of the innate system. These include: Input, Output, mlflowdetection, arp, cesnet, cyst, ensembling, exporting_alerts, http_analyzer, ip_info, iris_module, kalipso, leak_detector, network_discovery, p2p_trust, risk_iq, rnn_cc_detection, threat_intelligence, virustotal, and update_manager. Some core parts of Slips are also part of the immune system, such as output. These modules are part of the innate system because they are very generic (e.g., port scan detector, unknown HTTP User-Agent, etc.), which makes them easy to create and train but also prone to false positives. Therefore, detections like unknown HTTP User-Agent are considered PAMPs by Slips since they are attack-associated patterns. @@ -116,17 +116,17 @@ The human adaptive immune system has many characteristics described, but for the ### Guided-random new detectors For the adaptive immune system, Slips implements the creation of many Zeek detection scripts that are guided-randomly adapted (not evolved and not a genetic algorithm) to get very precise fast rule-based detectors with context information. -The negative selection will be done using a local offline database of benign traffic (described later). +The negative selection will be done using a local offline database of benign traffic (described later). The idea of the guided-random creation is to: 1. Create many guided-random detection scripts for Zeek using the new local LLM module. It is random only for certain parts of the Zeek script. - 1. A basic template of the detector is given + 1. A basic template of the detector is given 2. An LLM choses each part of the template according to what it is expected. If it is a URL then a URL, if it is a domain, then a domain. 2. __Positive Selection__. Each script should: 1. Recognize some traffic 2. Be syntactically correct 3. Compile and load - 4. Compile + 4. Compile 3. __Negative Selection__. Each script should: 1. Be tested against a large DB of benign traffic. 2. If any match happens the script is discarded. @@ -162,7 +162,7 @@ The two pathway activation works like this 5. The second activation comes from the DAMPs. DAMPs can be generated by the local Slips or received from the P2P network. 6. Only when both signals are received then the stronger defense is used and Slips isolates or blocks the attacker using ARP cache poisoning techniques. - + ### Evolution of Pattern Matching Detectors Upon Search When the adaptive system receives a PAMP detection together with some context flows, it will try to determine whether it has stored Zeek detector scripts that can match the context. @@ -218,78 +218,77 @@ In a sense, Slips already performs a form of Anergy when its designers detect a ## Immunoregulation -Immunoregulation refers to all the actions taken by the immune system to: +Immunoregulation refers to all the actions taken by the immune system to: -1. **Amplification** — Ensure the entire system is aware of the threat so it is not overlooked. -2. **Control the Power of the Response** — Avoid overreacting to a threat. -3. **Ensure the Response Is Timely** — Act quickly enough to counter the threat. -4. **Slow Down After the Threat Is Gone** — Stop actions once the threat has been removed. +1. **Amplification** — Ensure the entire system is aware of the threat so it is not overlooked. +2. **Control the Power of the Response** — Avoid overreacting to a threat. +3. **Ensure the Response Is Timely** — Act quickly enough to counter the threat. +4. **Slow Down After the Threat Is Gone** — Stop actions once the threat has been removed. -In Slips, immunoregulation will be implemented in two ways: +In Slips, immunoregulation will be implemented in two ways: -1. Inside the local Slips host. -2. Through communication with other peers in the P2P network. +1. Inside the local Slips host. +2. Through communication with other peers in the P2P network. ### Slips Host #### Amplification -There is no need for amplification inside a single Slips host, since the whole system already has the information. +There is no need for amplification inside a single Slips host, since the whole system already has the information. #### Control Power of Answer -- The innate system blocks via the firewall. -- The adaptive system blocks via an ARP poisoning attack. +- The innate system blocks via the firewall. +- The adaptive system blocks via an ARP poisoning attack. #### Be Sure the Answer Is on Time -- Firewall rules are added to the local firewall for fast response. -- Zeek scripts are injected into the Zeek process. -- The ARP poisoning attack is executed as soon as it is approved by the adaptive system. +- Firewall rules are added to the local firewall for fast response. +- Zeek scripts are injected into the Zeek process. +- The ARP poisoning attack is executed as soon as it is approved by the adaptive system. #### Slowdown After the Threat Is Gone -The slowdown depends on the number of evidences and alerts generated in the current time window. +The slowdown depends on the number of evidences and alerts generated in the current time window. -Currently, Slips implements the following behaviour: -- After the attacker is blocked, all new alerts from that attacker are still stored. -- When the attacker stops attacking, it enters a probation period of one time window. During this period, the attacker host is expected not to generate any alerts. -- If no alerts are generated in that probation period, then in the second time window after the last block, the attacker is unblocked. +Currently, Slips implements the following behaviour: +- After the attacker is blocked, all new alerts from that attacker are still stored. +- When the attacker stops attacking, it enters a probation period of one time window. During this period, the attacker host is expected not to generate any alerts. +- If no alerts are generated in that probation period, then in the second time window after the last block, the attacker is unblocked. - If the attacker continues to attack and generate evidences and alerts, it remains blocked. ### P2P Network #### Amplification -For alerts generated from PAMPs and profile violations (DAMPs): -- Every time the local Slips generates an alert, send it into the P2P network. -- Every time the local Slips receives an alert in the P2P network from another peer, and that alert originated in that peer, resend it to the P2P network. - - (This ensures alerts generated by peer A and sent by peer A are amplified, but ignores alerts that peer A forwards on behalf of peer B.) +For alerts generated from PAMPs and profile violations (DAMPs): +- Every time the local Slips generates an alert, send it into the P2P network. +- Every time the local Slips receives an alert in the P2P network from another peer, and that alert originated in that peer, resend it to the P2P network. + - (This ensures alerts generated by peer A and sent by peer A are amplified, but ignores alerts that peer A forwards on behalf of peer B.) #### Control Power of Answer -- The innate system blocks via the firewall. -- The adaptive system blocks via an ARP poisoning attack. +- The innate system blocks via the firewall. +- The adaptive system blocks via an ARP poisoning attack. #### Be Sure the Answer Is on Time -- Firewall rules are added to the local firewall for fast response. -- Zeek scripts are injected into the Zeek process. -- The ARP poisoning attack is executed as soon as it is approved by the adaptive system. +- Firewall rules are added to the local firewall for fast response. +- Zeek scripts are injected into the Zeek process. +- The ARP poisoning attack is executed as soon as it is approved by the adaptive system. #### Slowdown After the Threat Is Gone -The slowdown of the response depends on the number of PAMPs and DAMPs still received by peers in the network. +The slowdown of the response depends on the number of PAMPs and DAMPs still received by peers in the network. -- When the attack stops, no more evidences are generated by peers and no further alerts should be sent or amplified. -- When PAMPs stop, the firewall block and ARP poisoning should also stop. (The firewall only requires PAMPs to be activated, but the ARP poisoning attack requires both PAMPs and DAMPs.) -- When the attack stops, the host’s behaviour should also return to normal, which will eventually stop DAMPs. This typically takes more time. +- When the attack stops, no more evidences are generated by peers and no further alerts should be sent or amplified. +- When PAMPs stop, the firewall block and ARP poisoning should also stop. (The firewall only requires PAMPs to be activated, but the ARP poisoning attack requires both PAMPs and DAMPs.) +- When the attack stops, the host’s behaviour should also return to normal, which will eventually stop DAMPs. This typically takes more time. -A waiting function, based on the time window, will be applied so these changes do not happen immediately after flows stop being received. +A waiting function, based on the time window, will be applied so these changes do not happen immediately after flows stop being received. # Stopping the Threats -The main goal of the human immune system is to protect us by killing pathogens, neutralising them, or expelling them. Slips needs to do the same in order to be effective. - -To stop threats, Slips implements two actions: +The main goal of the human immune system is to protect us by killing pathogens, neutralising them, or expelling them. Slips needs to do the same in order to be effective. -1. **Block in the local firewall** - - When an alert is generated for a host (PAMPs are detected by the innate system only), the host is blocked in the firewall of the local machine. +To stop threats, Slips implements two actions: -2. **ARP Cache Poison Attack** - - When both PAMPs and DAMPs are detected, the attacker host is isolated through an ARP cache poisoning attack. +1. **Block in the local firewall** + - When an alert is generated for a host (PAMPs are detected by the innate system only), the host is blocked in the firewall of the local machine. +2. **ARP Cache Poison Attack** + - When both PAMPs and DAMPs are detected, the attacker host is isolated through an ARP cache poisoning attack. diff --git a/docs/index.rst b/docs/index.rst index 51eb2d4625..8687e3f8b1 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -17,7 +17,7 @@ This documentation gives an overview how Slips works, how to use it and how to h - **Detection modules**. Explanation of detection modules in Slips, types of input and output. See :doc:`Detection modules `. -- **Bruteforcing**. Dedicated documentation for the SSH bruteforcing detection module. See :doc:`Bruteforcing `. +- **brute_force_detector**. Dedicated documentation for the SSH brute force detector module. See :doc:`brute_force_detector `. - **HTTPS anomaly detection**. Detailed design and behavior of the HTTPS anomaly detector. See :doc:`HTTPS anomaly detection `. @@ -25,7 +25,7 @@ This documentation gives an overview how Slips works, how to use it and how to h - **Training with your own data**. Explanation on how to re-train the machine learning system of Slips with your own traffic (normal or malicious).See :doc:`Training `. -- **Detections per Flow**. Explanation on how Slips works to make detections on each flow with different techniques. See :doc:`Flow Alerts `. +- **Detections per Flow**. Explanation on how Slips works to make detections on each flow with different techniques. See :doc:`flow_alerts `. - **Exporting**. The exporting module allows Slips to export to Slack and STIX servers. See :doc:`Exporting `. @@ -53,14 +53,14 @@ This documentation gives an overview how Slips works, how to use it and how to h usage architecture detection_modules - bruteforcing + brute_force_detector https_anomaly_detection - flowalerts + flow_alerts features training exporting P2P - fides_module + fides create_new_module datasets immune/Immune diff --git a/docs/installation.md b/docs/installation.md index 55bdc1150f..e39549d7d6 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -81,12 +81,12 @@ In addition to the full stratosphereips/slips:latest image, there is now a minim * rnn_cc_detection/ * timeline/ * kalipso/ -* p2ptrust/ -* flowmldetection/ +* p2p_trust/ +* flow_ml_detection/ * cyst/ * cesnet/ * exporting_alerts/ -* riskiq/ +* risk_iq/ * template/ * blocking/ * virustotal/ diff --git a/docs/slips_in_action.md b/docs/slips_in_action.md index 2df49498fa..34047ab854 100644 --- a/docs/slips_in_action.md +++ b/docs/slips_in_action.md @@ -31,7 +31,7 @@ First, Slips will start by updating all the remote TI feeds added in slips.yaml To make sure Slips is up to date with the most recent IoCs in all feeds, all feeds are loaded, parsed and updated periodically and automatically by -Slips every 24 hours by our [Update Manager](https://stratospherelinuxips.readthedocs.io/en/develop/detection_modules.html#update-manager-module), which requires no user interaction. +Slips every 24 hours by our [update_manager](https://stratospherelinuxips.readthedocs.io/en/develop/detection_modules.html#update-manager-module), which requires no user interaction. Afetr updating, slips modules start and print the PID of every successfully started module. diff --git a/docs/training.md b/docs/training.md index e2a86674be..7edf73c5a7 100644 --- a/docs/training.md +++ b/docs/training.md @@ -2,14 +2,14 @@ Slips has one machine learning module that can be retrained by users. This is done by puttin slips in training mode so you can re-train the machine learning models with your own traffic. By default Slips includes an already trained model with our data, but it is sometimes necessary to adapt it to your own circumstances. -Until Slips 0.7.3, there is only one module for now that can do this, the one called 'flowmldetection'. This module analyzes flows one by one, as formatted similarly as in a conn.log Zeek file. This module is enabled by default in testing mode. This module uses by default the SGDClassifier with a linear support vector machine (SVM). The decision to use SVM was done because is one of the few algorithms that can be used for online learning and that can extend a current model with new data. +Until Slips 0.7.3, there is only one module for now that can do this, the one called 'flow_ml_detection'. This module analyzes flows one by one, as formatted similarly as in a conn.log Zeek file. This module is enabled by default in testing mode. This module uses by default the SGDClassifier with a linear support vector machine (SVM). The decision to use SVM was done because is one of the few algorithms that can be used for online learning and that can extend a current model with new data. To re-train this machine learning algorithm, you need to do the following: -1- Edit the config/slips.yaml file to put Slips in train mode. Search the word __train__ in the section __[flowmldetection]__ and uncomment the __mode = train__ and comment __mode = test__. It should look like +1- Edit the config/slips.yaml file to put Slips in train mode. Search the word __train__ in the section __[flow_ml_detection]__ and uncomment the __mode = train__ and comment __mode = test__. It should look like - [flowmldetection] - # The mode 'train' should be used to tell the flowmldetection module that the flows received are all for training. + [flow_ml_detection] + # The mode 'train' should be used to tell the flow_ml_detection module that the flows received are all for training. # A label should be provided in the [Parameters] section mode = train @@ -51,8 +51,8 @@ You can also run slips in an interface and train it directly with your data 4- Finally to use the model, put back the __test__ mode in the configuration config/slips.yaml - [flowmldetection] - # The mode 'train' should be used to tell the flowmldetection module that the flows received are all for training. + [flow_ml_detection] + # The mode 'train' should be used to tell the flow_ml_detection module that the flows received are all for training. # A label should be provided in the [Parameters] section #mode = train diff --git a/docs/usage.md b/docs/usage.md index d4cd642f4d..357c04ce0d 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -523,6 +523,12 @@ In ```out``` mode, SLIPS still creates profiles for internal (A) and external (B This parameter allows you to tailor SLIPS's analysis focus based on your specific monitoring requirements, such as detecting potential data exfiltration attempts (```out``` mode) or performing comprehensive network monitoring in both directions (```all``` mode). +**Persistent runtime data** + +Use ```permanent_dir``` to choose where Slips stores databases and runtime-generated files that must persist across different Slips runs and should not be overwritten. + +This includes persistent artifacts such as ```p2p_trust_runtime/``` and shared module databases like the Fides cache. +
diff --git a/managers/process_manager.py b/managers/process_manager.py index 13e295f943..2c50143570 100644 --- a/managers/process_manager.py +++ b/managers/process_manager.py @@ -155,7 +155,7 @@ def start_evidence_process(self): 1, 0, ) - self.main.db.store_pid("EvidenceHandler", int(evidence_process.pid)) + self.main.db.store_pid("evidence_handler", int(evidence_process.pid)) return evidence_process def start_input_process(self): @@ -266,7 +266,7 @@ def is_bootstrapping_module(self, module_name: str) -> bool: return False def is_abstract_module(self, obj) -> bool: - return obj.name in ("IModule", "AsyncModule") + return obj.name in ("imodule", "iasync_module") def get_modules(self): """ @@ -356,11 +356,11 @@ def _load_valid_classes_from_module(self, module, plugins): def _prioritize_blocking_modules(self, plugins): """ - Changes the order of the blocking modules (ARP poisoner and - Blocking) to load them before the rest of the modules + Changes the order of the blocking modules (`arp_poisoner` and + `blocking`) to load them before the rest of the modules so they can receive msgs sent from other modules """ - blocking_modules = ("Blocking", "ARP Poisoner") + blocking_modules = ("blocking", "arp_poisoner") at_least_one_blocking_module_is_loaded = False for module in blocking_modules: @@ -428,7 +428,7 @@ def print_started_module( self, module_name: str, module_pid: int, module_description: str ) -> None: self.main.print( - f"\t\tStarting the module {green(module_name)} " + f"\t\tStarting {green(module_name)} module " f"({module_description}) " f"[PID {green(module_pid)}]", 1, @@ -541,9 +541,9 @@ def warn_about_pending_modules(self, pending_modules: List[Process]): ) # check if update manager is still alive - if "Update Manager" in pending_module_names: + if "update_manager" in pending_module_names: self.main.print( - "Update Manager may take several minutes " + "update_manager may take several minutes " "to finish updating 45+ TI files." ) @@ -563,16 +563,16 @@ def get_hitlist_in_order(self) -> Tuple[List[Process], List[Process]]: # slips won't reach this function unless they are done already. # so no need to kill them last pids_to_kill_last = [ - self.main.db.get_pid_of("EvidenceHandler"), + self.main.db.get_pid_of("evidence_handler"), ] if self.main.args.blocking: - pids_to_kill_last.append(self.main.db.get_pid_of("Blocking")) - pids_to_kill_last.append(self.main.db.get_pid_of("ARP Poisoner")) + pids_to_kill_last.append(self.main.db.get_pid_of("blocking")) + pids_to_kill_last.append(self.main.db.get_pid_of("arp_poisoner")) if "exporting_alerts" not in self.main.db.get_disabled_modules(): pids_to_kill_last.append( - self.main.db.get_pid_of("Exporting Alerts") + self.main.db.get_pid_of("exporting_alerts") ) # remove all None PIDs. this happens when a module in that list # isnt started in the current run. e.g. virustotal module starts then diff --git a/modules/anomaly_detection_https/anomaly_detection_https.py b/modules/anomaly_detection_https/anomaly_detection_https.py index 88e7fe12b4..726143f7f2 100644 --- a/modules/anomaly_detection_https/anomaly_detection_https.py +++ b/modules/anomaly_detection_https/anomaly_detection_https.py @@ -2,7 +2,6 @@ # SPDX-License-Identifier: GPL-2.0-only import json import math -import os import time from dataclasses import dataclass, field from datetime import datetime, timezone @@ -124,7 +123,9 @@ def update_min_std_floor(self): candidate = self.floor_scale * max(q10, sigma_mad, self.floor_min) candidate = min(self.floor_max, max(self.floor_min, candidate)) beta = min(1.0, max(0.0, self.floor_update_beta)) - self.min_std_floor = (1.0 - beta) * self.min_std_floor + beta * candidate + self.min_std_floor = ( + 1.0 - beta + ) * self.min_std_floor + beta * candidate def zscore(self, value: float) -> float: std = math.sqrt(max(self.var, self.min_std_floor * self.min_std_floor)) @@ -203,7 +204,7 @@ class HostState: class AnomalyDetectionHTTPS(IModule): - name = "Anomaly Detection HTTPS" + name = "anomaly_detection_https" description = ( "HTTPS anomaly detector with hourly adaptive baselines and " "flow-level checks." @@ -213,8 +214,8 @@ class AnomalyDetectionHTTPS(IModule): def init(self): self.classifier = FlowClassifier() self.read_configuration() - self.operational_log_path = os.path.join( - self.output_dir, "anomaly_detection_https.log" + self.operational_log_path = self.get_module_specific_output_path( + "anomaly_detection_https.log" ) self.host_states: Dict[str, HostState] = {} @@ -262,9 +263,7 @@ def subscribe_to_channels(self): def read_configuration(self): conf = ConfigParser() self.training_hours = conf.https_anomaly_training_hours() - self.training_fit_method = ( - conf.https_anomaly_training_fit_method() - ) + self.training_fit_method = conf.https_anomaly_training_fit_method() self.training_alpha = conf.https_anomaly_training_alpha() self.hourly_zscore_threshold = conf.https_anomaly_hourly_zscore_thr() self.flow_zscore_threshold = conf.https_anomaly_flow_zscore_thr() @@ -279,9 +278,7 @@ def read_configuration(self): self.ja3_min_variants_per_server = ( conf.https_anomaly_ja3_min_variants_per_server() ) - self.requested_use_adwin_drift = ( - conf.https_anomaly_use_adwin_drift() - ) + self.requested_use_adwin_drift = conf.https_anomaly_use_adwin_drift() self.adwin_delta = conf.https_anomaly_adwin_delta() self.adwin_clock = conf.https_anomaly_adwin_clock() self.adwin_grace_period = conf.https_anomaly_adwin_grace_period() @@ -379,9 +376,7 @@ def log_event( reset = "\033[0m" if self.log_colors else "" wall_clock = self._ts_to_iso() traffic_clock = ( - self._ts_to_iso(traffic_ts) - if traffic_ts is not None - else "n/a" + self._ts_to_iso(traffic_ts) if traffic_ts is not None else "n/a" ) metrics_json = json.dumps(metrics, sort_keys=True) line = ( @@ -390,7 +385,9 @@ def log_event( ) if color: line = f"{color}{line}{reset}" - with open(self.operational_log_path, "a", encoding="utf-8") as log_file: + with open( + self.operational_log_path, "a", encoding="utf-8" + ) as log_file: log_file.write(f"{line}\n") @staticmethod @@ -405,7 +402,9 @@ def to_float(value, default=0.0) -> float: except (TypeError, ValueError): return float(default) - def get_traffic_ts(self, flow, fallback_ts: Optional[float] = None) -> float: + def get_traffic_ts( + self, flow, fallback_ts: Optional[float] = None + ) -> float: """ Returns traffic timestamp from flow.starttime. Detection windows must use traffic time, not host wall-clock time. @@ -621,7 +620,9 @@ def evidence_ts_from_traffic_ts(ts: float) -> str: ) @staticmethod - def threat_level_from_confidence_level(confidence_level: str) -> ThreatLevel: + def threat_level_from_confidence_level( + confidence_level: str, + ) -> ThreatLevel: # Requested policy: # - confidence low/medium -> threat level low # - confidence high -> threat level medium @@ -694,9 +695,9 @@ def emit_anomaly_evidence( threshold = reason.get( "threshold", ( - self.flow_zscore_threshold - if feature == "bytes_to_known_server" - else self.hourly_zscore_threshold + self.flow_zscore_threshold + if feature == "bytes_to_known_server" + else self.hourly_zscore_threshold ), ) @@ -729,7 +730,11 @@ def emit_anomaly_evidence( f"reason={reason_name}; value={value}; why={why}" ) - reasons_text = " | ".join(reason_parts) if reason_parts else "reason=Unknown; value=; why=not provided" + reasons_text = ( + " | ".join(reason_parts) + if reason_parts + else "reason=Unknown; value=; why=not provided" + ) description = ( f"HTTPS anomaly: type={kind}; confidence={confidence.get('level')} " f"({confidence_score:.3f}); {reasons_text}." @@ -883,9 +888,8 @@ def finalize_hour_bucket(self, profileid: str, state: HostState): ssl_flows = float(bucket.ssl_flows) known_server_avg_bytes = 0.0 if bucket.known_servers_flow_count > 0: - known_server_avg_bytes = ( - bucket.known_servers_total_bytes - / float(bucket.known_servers_flow_count) + known_server_avg_bytes = bucket.known_servers_total_bytes / float( + bucket.known_servers_flow_count ) features = { @@ -1210,7 +1214,8 @@ def process_ssl_event( twid_number: int, ): ts = self.get_traffic_ts( - ssl_flow, fallback_ts=self.to_float(conn_info.get("starttime"), 0.0) + ssl_flow, + fallback_ts=self.to_float(conn_info.get("starttime"), 0.0), ) state = self.ensure_hour_bucket(profileid, ts) state.last_twid = twid_number @@ -1478,9 +1483,7 @@ def process_ssl_event( "flow_raw_signals": flow_raw_signals, "alpha": alpha, "fit_method": ( - flow_training_fit_method - if alpha is None - else "ewma" + flow_training_fit_method if alpha is None else "ewma" ), }, ) diff --git a/modules/arp/arp.py b/modules/arp/arp.py index f83bf94843..e661e1475d 100644 --- a/modules/arp/arp.py +++ b/modules/arp/arp.py @@ -29,7 +29,7 @@ class ARP(IModule): - name = "ARP" + name = "arp" description = "Detect ARP attacks" authors = ["Alya Gomaa"] diff --git a/modules/arp/filter.py b/modules/arp/filter.py index fa3a8449f3..51ae0bb18c 100644 --- a/modules/arp/filter.py +++ b/modules/arp/filter.py @@ -37,7 +37,7 @@ def is_self_defense(self, ip: str): return ( ip in self.our_ips and self.args.blocking - and "ARP Poisoner" in loaded_modules + and "arp_poisoner" in loaded_modules ) def is_slips_peer(self, ip: str) -> bool: diff --git a/modules/arp_poisoner/arp_poisoner.py b/modules/arp_poisoner/arp_poisoner.py index ed43242c3d..a8dae0a976 100644 --- a/modules/arp_poisoner/arp_poisoner.py +++ b/modules/arp_poisoner/arp_poisoner.py @@ -1,7 +1,6 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only import logging -import os import subprocess import time from threading import Lock @@ -26,14 +25,16 @@ def generate_fake_mac(): class ARPPoisoner(IModule): - name = "ARP Poisoner" + name = "arp_poisoner" description = "ARP poisons attackers to isolate them from the network." authors = ["Alya Gomaa"] def init(self): self._time_since_last_repoison = {} self._time_since_last_internet_cut = {} - self.log_file_path = os.path.join(self.output_dir, "arp_poisoning.log") + self.log_file_path = self.get_module_specific_output_path( + "arp_poisoning.log" + ) self.blocking_logfile_lock = Lock() # clear it try: diff --git a/modules/blocking/blocking.py b/modules/blocking/blocking.py index 02658c68e4..f6b4fbe69e 100644 --- a/modules/blocking/blocking.py +++ b/modules/blocking/blocking.py @@ -24,7 +24,7 @@ class Blocking(IModule): by default this module flushes all slipsBlocking chains before it starts""" # Name: short name of the module. Do not use spaces - name = "Blocking" + name = "blocking" description = "Block malicious IPs connecting to this device" authors = ["Sebastian Garcia, Alya Gomaa"] @@ -36,7 +36,9 @@ def init(self): self.firewall = self._determine_linux_firewall() self.sudo = utils.get_sudo_according_to_env() self._init_chains_in_firewall() - self.blocking_log_path = os.path.join(self.output_dir, "blocking.log") + self.blocking_log_path = self.get_module_specific_output_path( + "blocking.log" + ) self.blocking_logfile_lock = Lock() # clear it try: diff --git a/modules/fidesModule/evaluation/__init__.py b/modules/brute_force_detector/__init__.py similarity index 100% rename from modules/fidesModule/evaluation/__init__.py rename to modules/brute_force_detector/__init__.py diff --git a/modules/bruteforcing/bruteforcing.py b/modules/brute_force_detector/brute_force_detector.py similarity index 92% rename from modules/bruteforcing/bruteforcing.py rename to modules/brute_force_detector/brute_force_detector.py index 017026ef2d..2127004f68 100644 --- a/modules/bruteforcing/bruteforcing.py +++ b/modules/brute_force_detector/brute_force_detector.py @@ -61,12 +61,13 @@ class SSHBruteforceCampaign: reported_bucket: int = -1 -class Bruteforcing(IModule): - name = "Bruteforcing" +class BruteforceDetector(IModule): + name = "brute_force_detector" description = ( - "Detect SSH bruteforcing using ssh.log, software.log, and Zeek notices." + "Detect SSH brute forcing using ssh.log, software.log, and Zeek " + "notices." ) - authors = ["Sebastian Garcia", "OpenAI"] + authors = ["Sebastian Garcia"] ssh_full_confidence_attempts = 30 def init(self): @@ -93,10 +94,12 @@ def pre_main(self): def read_configuration(self): conf = ConfigParser() - self.ssh_attempt_threshold = conf.ssh_bruteforcing_threshold() + self.ssh_attempt_threshold = conf.ssh_brute_force_detector_threshold() @staticmethod - def _campaign_key(profileid: str, twid: str, daddr: str, dport: str) -> str: + def _campaign_key( + profileid: str, twid: str, daddr: str, dport: str + ) -> str: return f"{profileid}_{twid}:dst:{daddr}:dport:{dport}" @staticmethod @@ -150,7 +153,9 @@ def _get_reporting_bucket(self, attempts: int) -> int: if attempts < self.ssh_attempt_threshold: return -1 # Emit frequently near the threshold, then increasingly sparsely. - return int(math.log2(max(1, attempts - self.ssh_attempt_threshold + 1))) + return int( + math.log2(max(1, attempts - self.ssh_attempt_threshold + 1)) + ) def _get_banner_bonus(self, banner: str, source: str) -> float: if not banner: @@ -161,7 +166,9 @@ def _get_banner_bonus(self, banner: str, source: str) -> float: bonus += 0.03 normalized_banner = banner.lower() - if any(token in normalized_banner for token in AUTOMATION_BANNER_TOKENS): + if any( + token in normalized_banner for token in AUTOMATION_BANNER_TOKENS + ): bonus += 0.07 elif any( token in normalized_banner for token in KNOWN_CLIENT_BANNER_TOKENS @@ -239,11 +246,9 @@ def _build_campaign_description( ) -> str: port_label = self._get_port_label(campaign.dport) target = campaign.daddr or "an SSH server" - destination = ( - f"{target} on {port_label}" if port_label else target - ) + destination = f"{target} on {port_label}" if port_label else target description = ( - f"SSH bruteforcing from {campaign.saddr} to {destination}. " + f"SSH brute force detector from {campaign.saddr} to {destination}. " f"Attempts observed: {campaign.attempts}." ) if banner: @@ -285,13 +290,15 @@ def _set_campaign_evidence( ioc_type=IoCType.IP, value=campaign.saddr, ), - victim=Victim( - direction=Direction.DST, - ioc_type=IoCType.IP, - value=campaign.daddr, - ) - if utils.is_valid_ip(campaign.daddr) - else False, + victim=( + Victim( + direction=Direction.DST, + ioc_type=IoCType.IP, + value=campaign.daddr, + ) + if utils.is_valid_ip(campaign.daddr) + else False + ), threat_level=ThreatLevel.MEDIUM, confidence=confidence, description=description, @@ -307,7 +314,7 @@ def _set_campaign_evidence( def _set_notice_evidence(self, profileid: str, twid: str, flow): srcip = flow.saddr description = ( - f"SSH bruteforcing. {flow.msg}. " + f"SSH brute force detector. {flow.msg}. " f"Confirmed by Zeek notice.log. Confidence: 1.0. by Zeek" ) evidence = Evidence( @@ -378,7 +385,10 @@ def _handle_notice(self, profileid: str, twid: str, flow): ) confirmation["msg"] = flow.msg confirmation["attempts"] = str( - max(int(confirmation["attempts"]), self._parse_notice_attempts(flow.msg)) + max( + int(confirmation["attempts"]), + self._parse_notice_attempts(flow.msg), + ) ) if confirmation["reported"] != "true": diff --git a/modules/bruteforcing/__init__.py b/modules/bruteforcing/__init__.py deleted file mode 100644 index 8b13789179..0000000000 --- a/modules/bruteforcing/__init__.py +++ /dev/null @@ -1 +0,0 @@ - diff --git a/modules/cesnet/cesnet.py b/modules/cesnet/cesnet.py index 78ac5233d1..a1313daa31 100644 --- a/modules/cesnet/cesnet.py +++ b/modules/cesnet/cesnet.py @@ -23,7 +23,7 @@ class CESNET(IModule): - name = "CESNET" + name = "cesnet" description = "Send and receive alerts from warden servers." authors = ["Alya Gomaa"] diff --git a/modules/cyst/cyst.py b/modules/cyst/cyst.py index 76e5b0fffe..254d2ac1e2 100644 --- a/modules/cyst/cyst.py +++ b/modules/cyst/cyst.py @@ -17,7 +17,7 @@ class Module(IModule): # Name: short name of the module. Do not use spaces - name = "CYST" + name = "cyst" description = "Communicates with CYST simulation framework" authors = ["Alya Gomaa"] diff --git a/modules/exporting_alerts/exporting_alerts.py b/modules/exporting_alerts/exporting_alerts.py index d3fe47fa2c..de28c1943b 100644 --- a/modules/exporting_alerts/exporting_alerts.py +++ b/modules/exporting_alerts/exporting_alerts.py @@ -1,7 +1,6 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only import json -import os import sqlite3 import threading import time @@ -20,7 +19,7 @@ class ExportingAlerts(IModule): variables to use this module """ - name = "Exporting Alerts" + name = "exporting_alerts" description = "Export alerts to slack or STIX format" authors = ["Alya Gomaa"] @@ -42,9 +41,7 @@ def subscribe_to_channels(self): def _init_direct_export_queue(self): if self.queue_db: return - self.queue_db_path = os.path.join( - self.output_dir, "stix_export_queue.sqlite" - ) + self.queue_db_path = self.get_database_path("stix_export_queue.sqlite") self.queue_db = sqlite3.connect( self.queue_db_path, check_same_thread=False ) @@ -336,7 +333,7 @@ def pre_main(self): if not export_to_slack and not export_to_stix: self.print( - "Exporting Alerts module disabled (no export targets configured).", + "exporting_alerts module disabled (no export targets configured).", 0, 2, ) diff --git a/modules/exporting_alerts/stix_exporter.py b/modules/exporting_alerts/stix_exporter.py index 8998c25752..b2eef15a78 100644 --- a/modules/exporting_alerts/stix_exporter.py +++ b/modules/exporting_alerts/stix_exporter.py @@ -89,9 +89,7 @@ def init(self): name="stix_exporter_to_taxii_thread", ) else: - self._log_export( - f"Export disabled export_to={self.export_to}" - ) + self._log_export(f"Export disabled export_to={self.export_to}") def start_exporting_thread(self): # This thread is responsible for waiting n seconds before @@ -126,11 +124,7 @@ def _build_url(self, path: str) -> str: return urljoin(self._base_url(), adjusted) def _log_export(self, message: str) -> None: - timestamp = ( - datetime.utcnow() - .replace(tzinfo=timezone.utc) - .isoformat() - ) + timestamp = datetime.utcnow().replace(tzinfo=timezone.utc).isoformat() line = f"{timestamp} {message}\n" try: with open(self.export_log_path, "a", encoding="utf-8") as log_file: @@ -148,11 +142,7 @@ def _resolve_output_dir(self) -> str: Falls back to the current working directory if the DB does not have an output directory set yet. """ - output_dir = getattr(self.db, "output_dir", None) - if not output_dir: - output_dir = self.db.get_output_dir() - if isinstance(output_dir, bytes): - output_dir = output_dir.decode("utf-8") + output_dir = self.get_output_path(module_name="exporting_alerts") if not output_dir: output_dir = os.getcwd() output_dir = os.path.abspath(output_dir) @@ -312,9 +302,7 @@ def _resolve_objects_url(self) -> Optional[str]: timeout=self.taxii_timeout, ) except Exception as err: - self._log_export( - f"Direct export failed: discovery error {err}" - ) + self._log_export(f"Direct export failed: discovery error {err}") return None if not response.ok: self._log_export( @@ -341,7 +329,9 @@ def _resolve_objects_url(self) -> Optional[str]: root_url = ( root if str(root).startswith("http") else self._build_url(root) ) - collections_url = urljoin(root_url.rstrip("/") + "/", "collections/") + collections_url = urljoin( + root_url.rstrip("/") + "/", "collections/" + ) try: resp = requests.get( collections_url, @@ -412,9 +402,7 @@ def export(self) -> bool: STIX_data.json bundle as a TAXII envelope. """ if self.taxii_version == 1: - self._log_export( - "Export skipped: TAXII 1 requires direct_export." - ) + self._log_export("Export skipped: TAXII 1 requires direct_export.") return False if not self.should_export(): self._log_export("Export skipped: stix not enabled.") @@ -472,7 +460,9 @@ def export(self) -> bool: try: status = collection.add_objects(envelope) except Exception as err: - self.print(f"Failed to push bundle to TAXII collection: {err}", 0, 3) + self.print( + f"Failed to push bundle to TAXII collection: {err}", 0, 3 + ) response = getattr(err, "response", None) if response is not None: self._log_export( @@ -544,20 +534,22 @@ def read_configuration(self) -> bool: self.collection_name = conf.collection_name() self.taxii_username = conf.taxii_username() self.taxii_password = conf.taxii_password() - self.taxii_version = self._normalize_taxii_version(conf.taxii_version()) + self.taxii_version = self._normalize_taxii_version( + conf.taxii_version() + ) self.taxii_timeout = conf.taxii_timeout() self.direct_export = bool(conf.taxii_direct_export()) self.direct_export_workers = conf.taxii_direct_export_workers() self.direct_export_max_workers = conf.taxii_direct_export_max_workers() self.direct_export_retry_max = conf.taxii_direct_export_retry_max() - self.direct_export_retry_backoff = conf.taxii_direct_export_retry_backoff() + self.direct_export_retry_backoff = ( + conf.taxii_direct_export_retry_backoff() + ) self.direct_export_retry_max_delay = ( conf.taxii_direct_export_retry_max_delay() ) if self.taxii_version == 1 and not self.direct_export: - self._log_export( - "TAXII 1 selected; forcing direct_export=true." - ) + self._log_export("TAXII 1 selected; forcing direct_export=true.") self.direct_export = True # push delay exists -> create a thread that waits # push delay doesn't exist -> running using file not interface @@ -772,7 +764,9 @@ def _build_taxii1_cybox_properties( def _build_taxii1_package( self, evidence: dict, attacker: str, ioc_type: str ) -> Optional[str]: - properties_xml = self._build_taxii1_cybox_properties(attacker, ioc_type) + properties_xml = self._build_taxii1_cybox_properties( + attacker, ioc_type + ) if not properties_xml: return None @@ -927,11 +921,11 @@ def _export_taxii1( ) except Exception as err: self.direct_export_fail += 1 - self._log_export(f"Direct export failed: TAXII 1 request error {err}") - duration = time.time() - start_ts self._log_export( - f"Direct export duration_seconds={duration:.3f}" + f"Direct export failed: TAXII 1 request error {err}" ) + duration = time.time() - start_ts + self._log_export(f"Direct export duration_seconds={duration:.3f}") return False if not response.ok or 'status_type="FAILURE"' in response.text: @@ -941,9 +935,7 @@ def _export_taxii1( f"response={response.text[:2000]}" ) duration = time.time() - start_ts - self._log_export( - f"Direct export duration_seconds={duration:.3f}" - ) + self._log_export(f"Direct export duration_seconds={duration:.3f}") return False self.direct_export_success += 1 @@ -1042,9 +1034,7 @@ def export_evidence_direct(self, evidence: dict) -> bool: self.direct_export_fail += 1 self._log_export(f"Direct export failed: request error {err}") duration = time.time() - start - self._log_export( - f"Direct export duration_seconds={duration:.3f}" - ) + self._log_export(f"Direct export duration_seconds={duration:.3f}") return False if not response.ok: @@ -1054,9 +1044,7 @@ def export_evidence_direct(self, evidence: dict) -> bool: f"response={response.text[:2000]}" ) duration = time.time() - start - self._log_export( - f"Direct export duration_seconds={duration:.3f}" - ) + self._log_export(f"Direct export duration_seconds={duration:.3f}") return False success_count = None @@ -1074,9 +1062,7 @@ def export_evidence_direct(self, evidence: dict) -> bool: f"Direct export failed: status failures={failure_count}" ) duration = time.time() - start - self._log_export( - f"Direct export duration_seconds={duration:.3f}" - ) + self._log_export(f"Direct export duration_seconds={duration:.3f}") return False self.direct_export_success += 1 @@ -1098,9 +1084,7 @@ def schedule_sending_to_taxii_server(self): self.push_delay seconds when running on an interface only """ if self.direct_export: - self._log_export( - "Scheduler disabled: direct_export enabled." - ) + self._log_export("Scheduler disabled: direct_export enabled.") return while True: # on an interface, we use the push delay from slips.yaml diff --git a/modules/fidesModule/__init__.py b/modules/fides/__init__.py similarity index 100% rename from modules/fidesModule/__init__.py rename to modules/fides/__init__.py diff --git a/modules/fidesModule/config/fides.conf.yml b/modules/fides/config/fides.conf.yml similarity index 99% rename from modules/fidesModule/config/fides.conf.yml rename to modules/fides/config/fides.conf.yml index a83336ff53..27e1c7f057 100644 --- a/modules/fidesModule/config/fides.conf.yml +++ b/modules/fides/config/fides.conf.yml @@ -148,4 +148,3 @@ trust: # Threat Intelligence aggregation strategy # valid values - ['average', 'weightedAverage', 'stdevFromScore'] tiAggregationStrategy: 'average' - diff --git a/modules/fides/evaluation/README.md b/modules/fides/evaluation/README.md new file mode 100644 index 0000000000..459d186c0f --- /dev/null +++ b/modules/fides/evaluation/README.md @@ -0,0 +1 @@ +All algorithms in this package are based on SORT - see paper. diff --git a/modules/fidesModule/evaluation/recommendation/__init__.py b/modules/fides/evaluation/__init__.py similarity index 100% rename from modules/fidesModule/evaluation/recommendation/__init__.py rename to modules/fides/evaluation/__init__.py diff --git a/modules/fidesModule/evaluation/discount_factor.py b/modules/fides/evaluation/discount_factor.py similarity index 100% rename from modules/fidesModule/evaluation/discount_factor.py rename to modules/fides/evaluation/discount_factor.py diff --git a/modules/fidesModule/evaluation/service/__init__.py b/modules/fides/evaluation/recommendation/__init__.py similarity index 100% rename from modules/fidesModule/evaluation/service/__init__.py rename to modules/fides/evaluation/recommendation/__init__.py diff --git a/modules/fidesModule/evaluation/recommendation/new_history.py b/modules/fides/evaluation/recommendation/new_history.py similarity index 53% rename from modules/fidesModule/evaluation/recommendation/new_history.py rename to modules/fides/evaluation/recommendation/new_history.py index 387e70e0e0..37fad3dbae 100644 --- a/modules/fidesModule/evaluation/recommendation/new_history.py +++ b/modules/fides/evaluation/recommendation/new_history.py @@ -1,18 +1,21 @@ from ...model.configuration import TrustModelConfiguration from ...model.peer_trust_data import PeerTrustData from ...model.recommendation import Recommendation -from ...model.recommendation_history import RecommendationHistoryRecord, RecommendationHistory +from ...model.recommendation_history import ( + RecommendationHistoryRecord, + RecommendationHistory, +) from ...utils.time import now def create_recommendation_history_for_peer( - configuration: TrustModelConfiguration, - peer: PeerTrustData, - recommendation: Recommendation, - history_factor: float, - er_ij: float, - ecb_ij: float, - eib_ij: float + configuration: TrustModelConfiguration, + peer: PeerTrustData, + recommendation: Recommendation, + history_factor: float, + er_ij: float, + ecb_ij: float, + eib_ij: float, ) -> RecommendationHistory: """ Creates new recommendation_history for given peer and its recommendations. @@ -26,25 +29,31 @@ def create_recommendation_history_for_peer( :param eib_ij: estimation about integrity belief :return: """ - rs_ik = __compute_recommendation_satisfaction_parameter(recommendation, er_ij, ecb_ij, eib_ij) - rw_ik = __compute_weight_of_recommendation(configuration, recommendation, history_factor) + rs_ik = __compute_recommendation_satisfaction_parameter( + recommendation, er_ij, ecb_ij, eib_ij + ) + rw_ik = __compute_weight_of_recommendation( + configuration, recommendation, history_factor + ) - updated_history = peer.recommendation_history + [RecommendationHistoryRecord(satisfaction=rs_ik, - weight=rw_ik, - timestamp=now())] + updated_history = peer.recommendation_history + [ + RecommendationHistoryRecord( + satisfaction=rs_ik, weight=rw_ik, timestamp=now() + ) + ] # fix history len if we reached max size if len(updated_history) > configuration.recommendations.history_max_size: last_idx = len(updated_history) - updated_history = updated_history[last_idx - configuration.recommendations.history_max_size: last_idx] + updated_history = updated_history[ + last_idx + - configuration.recommendations.history_max_size : last_idx + ] return updated_history def __compute_recommendation_satisfaction_parameter( - recommendation: Recommendation, - er_ij: float, - ecb_ij: float, - eib_ij: float + recommendation: Recommendation, er_ij: float, ecb_ij: float, eib_ij: float ) -> float: """ Computes satisfaction parameter - how much was peer satisfied with provided data. @@ -55,16 +64,28 @@ def __compute_recommendation_satisfaction_parameter( :param eib_ij: estimation about integrity belief :return: recommendation satisfaction rs_ik """ - r_diff = (1 - abs(recommendation.recommendation - er_ij) / er_ij) if er_ij > 0 else 0 - cb_diff = (1 - abs(recommendation.competence_belief - ecb_ij) / ecb_ij) if ecb_ij > 0 else 0 - ib_diff = (1 - abs(recommendation.integrity_belief - eib_ij) / eib_ij) if eib_ij > 0 else 0 + r_diff = ( + (1 - abs(recommendation.recommendation - er_ij) / er_ij) + if er_ij > 0 + else 0 + ) + cb_diff = ( + (1 - abs(recommendation.competence_belief - ecb_ij) / ecb_ij) + if ecb_ij > 0 + else 0 + ) + ib_diff = ( + (1 - abs(recommendation.integrity_belief - eib_ij) / eib_ij) + if eib_ij > 0 + else 0 + ) return (r_diff + cb_diff + ib_diff) / 3 def __compute_weight_of_recommendation( - configuration: TrustModelConfiguration, - recommendation: Recommendation, - history_factor: float + configuration: TrustModelConfiguration, + recommendation: Recommendation, + history_factor: float, ) -> float: """ Computes weight of recommendation - in model's notation rw^z_ik. @@ -73,6 +94,12 @@ def __compute_weight_of_recommendation( :param history_factor: int(mean(size of history) / maximal history size) :return: recommendation weight rw^z_ik """ - service_history = recommendation.service_history_size / configuration.service_history_max_size - used_peers = recommendation.initial_reputation_provided_by_count / configuration.recommendations.peers_max_count + service_history = ( + recommendation.service_history_size + / configuration.service_history_max_size + ) + used_peers = ( + recommendation.initial_reputation_provided_by_count + / configuration.recommendations.peers_max_count + ) return history_factor * service_history + (1 - history_factor) * used_peers diff --git a/modules/fidesModule/evaluation/recommendation/peer_update.py b/modules/fides/evaluation/recommendation/peer_update.py similarity index 58% rename from modules/fidesModule/evaluation/recommendation/peer_update.py rename to modules/fides/evaluation/recommendation/peer_update.py index 9e6a7efac0..1a61135ab5 100644 --- a/modules/fidesModule/evaluation/recommendation/peer_update.py +++ b/modules/fides/evaluation/recommendation/peer_update.py @@ -11,9 +11,9 @@ # noinspection DuplicatedCode # TODO: [+] try to abstract this def update_recommendation_data_for_peer( - configuration: TrustModelConfiguration, - peer: PeerTrustData, - new_history: RecommendationHistory + configuration: TrustModelConfiguration, + peer: PeerTrustData, + new_history: RecommendationHistory, ) -> PeerTrustData: """ Computes and updates all recommendation data for given peer with new_history. @@ -28,28 +28,39 @@ def update_recommendation_data_for_peer( """ fading_factor = __compute_fading_factor(configuration, new_history) competence_belief = __compute_competence_belief(new_history, fading_factor) - integrity_belief = __compute_integrity_belief(new_history, fading_factor, competence_belief) + integrity_belief = __compute_integrity_belief( + new_history, fading_factor, competence_belief + ) integrity_discount = compute_discount_factor() - history_factor = len(new_history) / configuration.recommendations.history_max_size + history_factor = ( + len(new_history) / configuration.recommendations.history_max_size + ) # (rh_ik / rh_max) * (rcb_ik -0.5 * rib_ik) -> where -0.5 is discount factor - reputation_trust_own_experience = history_factor * (competence_belief + integrity_discount * integrity_belief) + reputation_trust_own_experience = history_factor * ( + competence_belief + integrity_discount * integrity_belief + ) # (1 - (rh_ik / rh_max)) * r_ik reputation_experience = (1 - history_factor) * peer.reputation # and now add both parts together - recommendation_trust = reputation_trust_own_experience + reputation_experience + recommendation_trust = ( + reputation_trust_own_experience + reputation_experience + ) - updated_trust = dataclasses.replace(peer, - recommendation_trust=recommendation_trust, - recommendation_history=new_history - ) + updated_trust = dataclasses.replace( + peer, + recommendation_trust=recommendation_trust, + recommendation_history=new_history, + ) return updated_trust -def __compute_fading_factor(configuration: TrustModelConfiguration, - recommendation_history: RecommendationHistory) -> List[float]: +def __compute_fading_factor( + configuration: TrustModelConfiguration, + recommendation_history: RecommendationHistory, +) -> List[float]: """ Computes fading factor for each record in recommendation history. @@ -70,7 +81,9 @@ def __compute_fading_factor(configuration: TrustModelConfiguration, return [1] * len(recommendation_history) -def __compute_competence_belief(recommendation_history: RecommendationHistory, fading_factor: List[float]) -> float: +def __compute_competence_belief( + recommendation_history: RecommendationHistory, fading_factor: List[float] +) -> float: """ Computes competence belief - rcb_ik. @@ -78,22 +91,34 @@ def __compute_competence_belief(recommendation_history: RecommendationHistory, f :param fading_factor: fading factors for given history :return: reputation competence belief for given data """ - assert len(recommendation_history) == len(fading_factor), \ - "Recommendation history must have same length as fading factors." + assert len(recommendation_history) == len( + fading_factor + ), "Recommendation history must have same length as fading factors." normalisation = sum( - [recommendation.weight * fading for recommendation, fading in zip(recommendation_history, fading_factor)]) - - belief = sum([service.satisfaction * service.weight * fading - for service, fading - in zip(recommendation_history, fading_factor)]) + [ + recommendation.weight * fading + for recommendation, fading in zip( + recommendation_history, fading_factor + ) + ] + ) + + belief = sum( + [ + service.satisfaction * service.weight * fading + for service, fading in zip(recommendation_history, fading_factor) + ] + ) return belief / normalisation if normalisation > 0 else 0 -def __compute_integrity_belief(recommendation_history: RecommendationHistory, - fading_factor: List[float], - recommendation_competence_belief: float) -> float: +def __compute_integrity_belief( + recommendation_history: RecommendationHistory, + fading_factor: List[float], + recommendation_competence_belief: float, +) -> float: """ Computes integrity belief - rib_ik. @@ -102,15 +127,24 @@ def __compute_integrity_belief(recommendation_history: RecommendationHistory, :param fading_factor: fading factors for given history :return: integrity belief for given data """ - assert len(recommendation_history) == len(fading_factor), \ - "Recommendation history must have same length as fading factors." + assert len(recommendation_history) == len( + fading_factor + ), "Recommendation history must have same length as fading factors." history_size = len(recommendation_history) - weight_mean = sum(service.weight for service in recommendation_history) / history_size + weight_mean = ( + sum(service.weight for service in recommendation_history) + / history_size + ) fading_mean = sum(fading_factor) / history_size - sat = sum((recommendation.satisfaction * weight_mean * fading_mean - recommendation_competence_belief) ** 2 - for recommendation - in recommendation_history) + sat = sum( + ( + recommendation.satisfaction * weight_mean * fading_mean + - recommendation_competence_belief + ) + ** 2 + for recommendation in recommendation_history + ) return sqrt(sat / history_size) diff --git a/modules/fidesModule/evaluation/recommendation/process.py b/modules/fides/evaluation/recommendation/process.py similarity index 67% rename from modules/fidesModule/evaluation/recommendation/process.py rename to modules/fides/evaluation/recommendation/process.py index d0368e2e82..ceedf91e89 100644 --- a/modules/fidesModule/evaluation/recommendation/process.py +++ b/modules/fides/evaluation/recommendation/process.py @@ -2,8 +2,12 @@ from typing import Dict from ...evaluation.discount_factor import compute_discount_factor -from ...evaluation.recommendation.new_history import create_recommendation_history_for_peer -from ...evaluation.recommendation.peer_update import update_recommendation_data_for_peer +from ...evaluation.recommendation.new_history import ( + create_recommendation_history_for_peer, +) +from ...evaluation.recommendation.peer_update import ( + update_recommendation_data_for_peer, +) from ...model.aliases import PeerId from ...model.configuration import TrustModelConfiguration from ...model.peer_trust_data import TrustMatrix, PeerTrustData @@ -11,10 +15,10 @@ def process_new_recommendations( - configuration: TrustModelConfiguration, - subject: PeerTrustData, - matrix: TrustMatrix, - recommendations: Dict[PeerId, Recommendation] + configuration: TrustModelConfiguration, + subject: PeerTrustData, + matrix: TrustMatrix, + recommendations: Dict[PeerId, Recommendation], ) -> TrustMatrix: """ Evaluates received recommendation, computing recommendations and recommendation @@ -35,10 +39,14 @@ def process_new_recommendations( """ # verify that peers with responses are in trust matrix for peer in recommendations.keys(): - assert matrix[peer] is not None, f"Peer {peer} is not present in peer matrix." + assert ( + matrix[peer] is not None + ), f"Peer {peer} is not present in peer matrix." er_ij = __estimate_recommendation(matrix, recommendations) - ecb_ij, eib_ij = __estimate_competence_integrity_belief(matrix, recommendations) + ecb_ij, eib_ij = __estimate_competence_integrity_belief( + matrix, recommendations + ) history_sizes = [r.service_history_size for r in recommendations.values()] history_mean = int(sum(history_sizes) / len(history_sizes)) @@ -54,35 +62,41 @@ def process_new_recommendations( # now update final trust for the subject with new reputation # we also trust the subject same with service as well as with recommendations # we also set service_trust if it is not set, because for the first interaction it is equal to reputation - updated_subject_trust = dataclasses \ - .replace(subject, - service_trust=max(subject.service_trust, reputation), - reputation=reputation, - recommendation_trust=reputation, - initial_reputation_provided_by_count=len(recommendations) - ) - peers_updated_matrix = {updated_subject_trust.peer_id: updated_subject_trust} + updated_subject_trust = dataclasses.replace( + subject, + service_trust=max(subject.service_trust, reputation), + reputation=reputation, + recommendation_trust=reputation, + initial_reputation_provided_by_count=len(recommendations), + ) + peers_updated_matrix = { + updated_subject_trust.peer_id: updated_subject_trust + } # now we need to reflect performed reputation query and update how much we trust other peers for peer_id, recommendation in recommendations.items(): peer = matrix[peer_id] # build new history new_history = create_recommendation_history_for_peer( - configuration=configuration, peer=peer, recommendation=recommendation, - history_factor=history_factor, er_ij=er_ij, ecb_ij=ecb_ij, eib_ij=eib_ij + configuration=configuration, + peer=peer, + recommendation=recommendation, + history_factor=history_factor, + er_ij=er_ij, + ecb_ij=ecb_ij, + eib_ij=eib_ij, ) # and update peer and its recommendation data - updated_peer = update_recommendation_data_for_peer(configuration=configuration, - peer=peer, - new_history=new_history) + updated_peer = update_recommendation_data_for_peer( + configuration=configuration, peer=peer, new_history=new_history + ) peers_updated_matrix[updated_peer.peer_id] = updated_peer return peers_updated_matrix def __estimate_recommendation( - matrix: TrustMatrix, - recommendations: Dict[PeerId, Recommendation] + matrix: TrustMatrix, recommendations: Dict[PeerId, Recommendation] ) -> float: """ Computes estimation about recommendation. @@ -93,23 +107,28 @@ def __estimate_recommendation( :param recommendations: responses from the peers :return: estimation about recommendation er_ij """ - normalisation = sum([ - matrix[peer].recommendation_trust * response.initial_reputation_provided_by_count - for peer, response - in recommendations.items()] + normalisation = sum( + [ + matrix[peer].recommendation_trust + * response.initial_reputation_provided_by_count + for peer, response in recommendations.items() + ] ) recommendations = sum( - [matrix[peer].recommendation_trust * response.initial_reputation_provided_by_count * response.recommendation - for peer, response - in recommendations.items()]) + [ + matrix[peer].recommendation_trust + * response.initial_reputation_provided_by_count + * response.recommendation + for peer, response in recommendations.items() + ] + ) return recommendations / normalisation if normalisation > 0 else 0 def __estimate_competence_integrity_belief( - matrix: TrustMatrix, - recommendations: Dict[PeerId, Recommendation] + matrix: TrustMatrix, recommendations: Dict[PeerId, Recommendation] ) -> [float, float]: """ Estimates about competence and integrity beliefs. @@ -126,7 +145,9 @@ def __estimate_competence_integrity_belief( # as we would need to iterate three times, it's just better to make for cycle for peer, response in recommendations.items(): - trust_history_size = matrix[peer].recommendation_trust * response.service_history_size + trust_history_size = ( + matrix[peer].recommendation_trust * response.service_history_size + ) # rt_ik * sh_kj normalisation += trust_history_size # rt_ik * sh_kj * cb_kj diff --git a/modules/fidesModule/evaluation/recommendation/selection.py b/modules/fides/evaluation/recommendation/selection.py similarity index 67% rename from modules/fidesModule/evaluation/recommendation/selection.py rename to modules/fides/evaluation/recommendation/selection.py index b38c789d2d..d07aa731b4 100644 --- a/modules/fidesModule/evaluation/recommendation/selection.py +++ b/modules/fides/evaluation/recommendation/selection.py @@ -5,8 +5,7 @@ def select_trustworthy_peers_for_recommendations( - data: Dict[PeerId, float], - max_peers: int + data: Dict[PeerId, float], max_peers: int ) -> List[PeerId]: """ Selects peers that can be asked for recommendation. @@ -18,8 +17,14 @@ def select_trustworthy_peers_for_recommendations( var = sqrt(sum((rt - mean) ** 2 for rt in data.values())) lowest_rt = mean - var # select only peers that have recommendation_trust higher than mean - variance - candidates = sorted([ - {'id': peer_id, 'rt': rt} for peer_id, rt in data.items() if rt >= lowest_rt - ], key=lambda x: x['rt'], reverse=True) + candidates = sorted( + [ + {"id": peer_id, "rt": rt} + for peer_id, rt in data.items() + if rt >= lowest_rt + ], + key=lambda x: x["rt"], + reverse=True, + ) # and now cut them at max - return [p['id'] for p in candidates[: max_peers]] + return [p["id"] for p in candidates[:max_peers]] diff --git a/modules/fidesModule/persistence/__init__.py b/modules/fides/evaluation/service/__init__.py similarity index 100% rename from modules/fidesModule/persistence/__init__.py rename to modules/fides/evaluation/service/__init__.py diff --git a/modules/fidesModule/evaluation/service/interaction.py b/modules/fides/evaluation/service/interaction.py similarity index 58% rename from modules/fidesModule/evaluation/service/interaction.py rename to modules/fides/evaluation/service/interaction.py index cec4b5ec28..d9548ee988 100644 --- a/modules/fidesModule/evaluation/service/interaction.py +++ b/modules/fides/evaluation/service/interaction.py @@ -1,9 +1,8 @@ from enum import Enum Satisfaction = float -"""Represents value how much was client satisfied with the interaction -0 <= satisfaction <= 1 where 0 is NOT satisfied and 1 is satisfied. -""" +# Represents value how much was client satisfied with the interaction. +# 0 <= satisfaction <= 1 where 0 is NOT satisfied and 1 is satisfied. class SatisfactionLevels: @@ -12,10 +11,10 @@ class SatisfactionLevels: class Weight(Enum): - """How much was the interaction important. - 0 <= weight <= 1 - where 0 is unimportant and 1 is important - """ + # How much was the interaction important. + # 0 <= weight <= 1 + # where 0 is unimportant and 1 is important + FIRST_ENCOUNTER = 0.1 PING = 0.2 INTELLIGENCE_NO_DATA_REPORT = 0.3 diff --git a/modules/fidesModule/evaluation/service/peer_update.py b/modules/fides/evaluation/service/peer_update.py similarity index 63% rename from modules/fidesModule/evaluation/service/peer_update.py rename to modules/fides/evaluation/service/peer_update.py index 732584a93d..0f49a0205b 100644 --- a/modules/fidesModule/evaluation/service/peer_update.py +++ b/modules/fides/evaluation/service/peer_update.py @@ -12,10 +12,11 @@ # noinspection DuplicatedCode # TODO: [+] try to abstract this + def update_service_data_for_peer( - configuration: TrustModelConfiguration, - peer: PeerTrustData, - new_history: ServiceHistory + configuration: TrustModelConfiguration, + peer: PeerTrustData, + new_history: ServiceHistory, ) -> PeerTrustData: """ Computes and updates PeerTrustData.service_trust - st_ij - for peer j - based on the given data. @@ -32,13 +33,17 @@ def update_service_data_for_peer( fading_factor = __compute_fading_factor(configuration, new_history) competence_belief = __compute_competence_belief(new_history, fading_factor) - integrity_belief = __compute_integrity_belief(new_history, fading_factor, competence_belief) + integrity_belief = __compute_integrity_belief( + new_history, fading_factor, competence_belief + ) integrity_discount = compute_discount_factor() history_factor = len(new_history) / configuration.service_history_max_size # (sh_ij / sh_max) * (cb_ij -0.5 * ib_ij) -> where -0.5 is discount factor - service_trust_own_experience = history_factor * (competence_belief + integrity_discount * integrity_belief) + service_trust_own_experience = history_factor * ( + competence_belief + integrity_discount * integrity_belief + ) # (1 - (sh_ij / sh_max)) * r_ij service_trust_reputation = (1 - history_factor) * peer.reputation # and now add both parts together @@ -47,17 +52,20 @@ def update_service_data_for_peer( # (case when the data do not follow normal distribution and ib is higher then mean) service_trust = bound(service_trust, 0, 1) - updated_trust = dataclasses.replace(peer, - service_trust=service_trust, - competence_belief=competence_belief, - integrity_belief=integrity_belief, - service_history=new_history - ) + updated_trust = dataclasses.replace( + peer, + service_trust=service_trust, + competence_belief=competence_belief, + integrity_belief=integrity_belief, + service_history=new_history, + ) return updated_trust -def __compute_fading_factor(configuration: TrustModelConfiguration, service_history: ServiceHistory) -> List[float]: +def __compute_fading_factor( + configuration: TrustModelConfiguration, service_history: ServiceHistory +) -> List[float]: """ Computes fading factor for each record in service history. @@ -79,7 +87,9 @@ def __compute_fading_factor(configuration: TrustModelConfiguration, service_hist return [1] * len(service_history) -def __compute_competence_belief(service_history: ServiceHistory, fading_factor: List[float]) -> float: +def __compute_competence_belief( + service_history: ServiceHistory, fading_factor: List[float] +) -> float: """ Computes competence belief - cb_ij. @@ -87,19 +97,31 @@ def __compute_competence_belief(service_history: ServiceHistory, fading_factor: :param fading_factor: fading factors for given history :return: competence belief for given data """ - assert len(service_history) == len(fading_factor), "Service history must have same length as fading factors." - - normalisation = sum([service.weight * fading for service, fading in zip(service_history, fading_factor)]) - belief = sum([service.satisfaction * service.weight * fading - for service, fading - in zip(service_history, fading_factor)]) + assert len(service_history) == len( + fading_factor + ), "Service history must have same length as fading factors." + + normalisation = sum( + [ + service.weight * fading + for service, fading in zip(service_history, fading_factor) + ] + ) + belief = sum( + [ + service.satisfaction * service.weight * fading + for service, fading in zip(service_history, fading_factor) + ] + ) return belief / normalisation -def __compute_integrity_belief(service_history: ServiceHistory, - fading_factor: List[float], - competence_belief: float) -> float: +def __compute_integrity_belief( + service_history: ServiceHistory, + fading_factor: List[float], + competence_belief: float, +) -> float: """ Computes integrity belief - ib_ij. @@ -108,15 +130,26 @@ def __compute_integrity_belief(service_history: ServiceHistory, :param fading_factor: fading factors for given history :return: integrity belief for given data """ - assert len(service_history) == len(fading_factor), "Service history must have same length as fading factors." + assert len(service_history) == len( + fading_factor + ), "Service history must have same length as fading factors." history_size = len(service_history) - weight_mean = sum([service.weight for service in service_history]) / history_size + weight_mean = ( + sum([service.weight for service in service_history]) / history_size + ) fading_mean = sum(fading_factor) / history_size - sat = sum([(service.satisfaction * weight_mean * fading_mean - competence_belief) ** 2 - for service - in service_history]) + sat = sum( + [ + ( + service.satisfaction * weight_mean * fading_mean + - competence_belief + ) + ** 2 + for service in service_history + ] + ) ib = sqrt(sat / history_size) return ib diff --git a/modules/fidesModule/evaluation/service/process.py b/modules/fides/evaluation/service/process.py similarity index 66% rename from modules/fidesModule/evaluation/service/process.py rename to modules/fides/evaluation/service/process.py index 159c382cf8..49ce97fb50 100644 --- a/modules/fidesModule/evaluation/service/process.py +++ b/modules/fides/evaluation/service/process.py @@ -12,21 +12,23 @@ def process_service_interaction( - configuration: TrustModelConfiguration, - peer: PeerTrustData, - satisfaction: Satisfaction, - weight: Weight + configuration: TrustModelConfiguration, + peer: PeerTrustData, + satisfaction: Satisfaction, + weight: Weight, ) -> PeerTrustData: """Processes given interaction and updates trust data.""" - new_history = peer.service_history + [ServiceHistoryRecord( - satisfaction=satisfaction, - weight=weight.value, - timestamp=now() - )] + new_history = peer.service_history + [ + ServiceHistoryRecord( + satisfaction=satisfaction, weight=weight.value, timestamp=now() + ) + ] # now restrict new history to max length if len(new_history) > configuration.service_history_max_size: last = len(new_history) - new_history = new_history[last - configuration.service_history_max_size: last] + new_history = new_history[ + last - configuration.service_history_max_size : last + ] # we don't update service trust for fixed trust peers if peer.has_fixed_trust: @@ -34,7 +36,5 @@ def process_service_interaction( return dataclasses.replace(peer, service_history=new_history) else: return update_service_data_for_peer( - configuration=configuration, - peer=peer, - new_history=new_history + configuration=configuration, peer=peer, new_history=new_history ) diff --git a/modules/fides/evaluation/ti_aggregation.py b/modules/fides/evaluation/ti_aggregation.py new file mode 100644 index 0000000000..5419e26dcc --- /dev/null +++ b/modules/fides/evaluation/ti_aggregation.py @@ -0,0 +1,118 @@ +from dataclasses import dataclass +from typing import List + +import numpy as np + +from ..model.peer_trust_data import PeerTrustData +from ..model.threat_intelligence import ThreatIntelligence +from ..utils import bound + + +@dataclass +class PeerReport: + report_ti: ThreatIntelligence + """Threat intelligence report.""" + + reporter_trust: PeerTrustData + """How much does Slips trust the reporter.""" + + +class TIAggregation: + + def assemble_peer_opinion( + self, data: List[PeerReport] + ) -> ThreatIntelligence: + """ + Assemble reports given by all peers and compute the overall network opinion. + + :param data: a list of peers and their reports, in the format given by TrustDB.get_opinion_on_ip() + :return: final score and final confidence + """ + raise NotImplementedError("") + + +class AverageConfidenceTIAggregation(TIAggregation): + + def assemble_peer_opinion( + self, data: List[PeerReport] + ) -> ThreatIntelligence: + """ + Uses average when computing final confidence. + """ + reports_ti = [d.report_ti for d in data] + reporters_trust = [d.reporter_trust.service_trust for d in data] + + normalize_net_trust_sum = sum(reporters_trust) + weighted_reporters = ( + [trust / normalize_net_trust_sum for trust in reporters_trust] + if normalize_net_trust_sum > 0 + else [0] * len(reporters_trust) + ) + + combined_score = sum( + r.score * w for r, w, in zip(reports_ti, weighted_reporters) + ) + combined_confidence = sum( + r.confidence * w for r, w, in zip(reports_ti, reporters_trust) + ) / len(reporters_trust) + + return ThreatIntelligence( + score=combined_score, confidence=combined_confidence + ) + + +class WeightedAverageConfidenceTIAggregation(TIAggregation): + + def assemble_peer_opinion( + self, data: List[PeerReport] + ) -> ThreatIntelligence: + reports_ti = [d.report_ti for d in data] + reporters_trust = [d.reporter_trust.service_trust for d in data] + + normalize_net_trust_sum = sum(reporters_trust) + weighted_reporters = [ + trust / normalize_net_trust_sum for trust in reporters_trust + ] + + combined_score = sum( + r.score * w for r, w, in zip(reports_ti, weighted_reporters) + ) + combined_confidence = sum( + r.confidence * w for r, w, in zip(reports_ti, weighted_reporters) + ) + + return ThreatIntelligence( + score=combined_score, confidence=combined_confidence + ) + + +class StdevFromScoreTIAggregation(TIAggregation): + + def assemble_peer_opinion( + self, data: List[PeerReport] + ) -> ThreatIntelligence: + reports_ti = [d.report_ti for d in data] + reporters_trust = [d.reporter_trust.service_trust for d in data] + + normalize_net_trust_sum = sum(reporters_trust) + weighted_reporters = [ + trust / normalize_net_trust_sum for trust in reporters_trust + ] + + merged_score = [ + r.score * r.confidence * w + for r, w, in zip(reports_ti, weighted_reporters) + ] + combined_score = sum(merged_score) + combined_confidence = bound(1 - np.std(merged_score), 0, 1) + + return ThreatIntelligence( + score=combined_score, confidence=combined_confidence + ) + + +TIAggregationStrategy = { + "average": AverageConfidenceTIAggregation, + "weightedAverage": WeightedAverageConfidenceTIAggregation, + "stdevFromScore": StdevFromScoreTIAggregation, +} diff --git a/modules/fides/evaluation/ti_evaluation.py b/modules/fides/evaluation/ti_evaluation.py new file mode 100644 index 0000000000..fd0b62ad77 --- /dev/null +++ b/modules/fides/evaluation/ti_evaluation.py @@ -0,0 +1,309 @@ +from collections import defaultdict +from typing import Dict, Tuple, Optional + +from ..evaluation.service.interaction import ( + Satisfaction, + Weight, + SatisfactionLevels, +) +from ..messaging.model import PeerIntelligenceResponse +from ..model.aliases import PeerId, Target +from ..model.peer_trust_data import PeerTrustData, TrustMatrix +from ..model.threat_intelligence import SlipsThreatIntelligence +from ..utils.logger import Logger + +logger = Logger(__name__) + + +class TIEvaluation: + def evaluate( + self, + aggregated_ti: SlipsThreatIntelligence, + responses: Dict[PeerId, PeerIntelligenceResponse], + trust_matrix: TrustMatrix, + **kwargs, + ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: + """Evaluate interaction with all peers that gave intelligence responses.""" + raise NotImplementedError("Use implementation rather then interface!") + + @staticmethod + def _weight() -> Weight: + return Weight.INTELLIGENCE_DATA_REPORT + + @staticmethod + def _assert_keys( + responses: Dict[PeerId, PeerIntelligenceResponse], + trust_matrix: TrustMatrix, + ): + assert trust_matrix.keys() == responses.keys() + + +class EvenTIEvaluation(TIEvaluation): + """Basic implementation for the TI evaluation, all responses are evaluated the same. + This implementation corresponds with Salinity botnet. + """ + + def __init__(self, **kwargs): + self.__kwargs = kwargs + self.__satisfaction = kwargs.get("satisfaction", SatisfactionLevels.Ok) + + def evaluate( + self, + aggregated_ti: SlipsThreatIntelligence, + responses: Dict[PeerId, PeerIntelligenceResponse], + trust_matrix: TrustMatrix, + **kwargs, + ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: + super()._assert_keys(responses, trust_matrix) + + return { + p.peer_id: (p, self.__satisfaction, self._weight()) + for p in trust_matrix.values() + } + + +class DistanceBasedTIEvaluation(TIEvaluation): + """Implementation that takes distance from the aggregated result and uses it as a penalisation.""" + + def __init__(self, **kwargs): + self.__kwargs = kwargs + + def evaluate( + self, + aggregated_ti: SlipsThreatIntelligence, + responses: Dict[PeerId, PeerIntelligenceResponse], + trust_matrix: TrustMatrix, + **kwargs, + ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: + super()._assert_keys(responses, trust_matrix) + return self._build_evaluation( + baseline_score=aggregated_ti.score, + baseline_confidence=aggregated_ti.confidence, + responses=responses, + trust_matrix=trust_matrix, + ) + + def _build_evaluation( + self, + baseline_score: float, + baseline_confidence: float, + responses: Dict[PeerId, PeerIntelligenceResponse], + trust_matrix: TrustMatrix, + ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: + satisfactions = { + peer_id: self._satisfaction( + baseline_score=baseline_score, + baseline_confidence=baseline_confidence, + report_score=ti.intelligence.score, + report_confidence=ti.intelligence.confidence, + ) + for peer_id, ti in responses.items() + } + + return { + p.peer_id: (p, satisfactions[p.peer_id], self._weight()) + for p in trust_matrix.values() + } + + @staticmethod + def _satisfaction( + baseline_score: float, + baseline_confidence: float, + report_score: float, + report_confidence: float, + ) -> Satisfaction: + return ( + 1 - (abs(baseline_score - report_score) / 2) * report_confidence + ) * baseline_confidence + + +class LocalCompareTIEvaluation(DistanceBasedTIEvaluation): + """This strategy compares received threat intelligence with the threat intelligence from local database. + + Uses the same penalisation system as DistanceBasedTIEvaluation with the difference that as a baseline, + it does not use aggregated value, but rather local intelligence. + + If it does not find threat intelligence for the target, it falls backs to DistanceBasedTIEvaluation. + """ + + def __init__(self, **kwargs): + super().__init__(**kwargs) + self.__default_ti_getter = kwargs.get("default_ti_getter", None) + + def get_local_ti( + self, + target: Target, + local_ti: Optional[SlipsThreatIntelligence] = None, + ) -> Optional[SlipsThreatIntelligence]: + if local_ti: + return local_ti + elif self.__default_ti_getter: + return self.__default_ti_getter(target) + else: + return None + + def evaluate( + self, + aggregated_ti: SlipsThreatIntelligence, + responses: Dict[PeerId, PeerIntelligenceResponse], + trust_matrix: TrustMatrix, + local_ti: Optional[SlipsThreatIntelligence] = None, + **kwargs, + ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: + super()._assert_keys(responses, trust_matrix) + + ti = self.get_local_ti(aggregated_ti.target, local_ti) + if not ti: + ti = aggregated_ti + logger.warn( + f"No local threat intelligence available for target {ti.target}! " + + "Falling back to DistanceBasedTIEvaluation." + ) + + return self._build_evaluation( + baseline_score=ti.score, + baseline_confidence=ti.confidence, + responses=responses, + trust_matrix=trust_matrix, + ) + + +class WeighedDistanceToLocalTIEvaluation(TIEvaluation): + """Strategy combines DistanceBasedTIEvaluation and LocalCompareTIEvaluation with the local weight parameter.""" + + def __init__(self, **kwargs): + super().__init__() + self.__distance = kwargs.get("distance", DistanceBasedTIEvaluation()) + self.__local = kwargs.get("localDistance", LocalCompareTIEvaluation()) + self.__local_weight = kwargs.get("localWeight", 0.5) + + def evaluate( + self, + aggregated_ti: SlipsThreatIntelligence, + responses: Dict[PeerId, PeerIntelligenceResponse], + trust_matrix: TrustMatrix, + **kwargs, + ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: + super()._assert_keys(responses, trust_matrix) + + distance_data = self.__distance.evaluate( + aggregated_ti, responses, trust_matrix, **kwargs + ) + local_data = self.__local.evaluate( + aggregated_ti, responses, trust_matrix, **kwargs + ) + + return { + p.peer_id: ( + p, + self.__local_weight * local_data[p.peer_id][1] + + (1 - self.__local_weight) * distance_data[p.peer_id][1], + self._weight(), + ) + for p in trust_matrix.values() + } + + +class MaxConfidenceTIEvaluation(TIEvaluation): + """Strategy combines DistanceBasedTIEvaluation, LocalCompareTIEvaluation and EvenTIEvaluation + in order to achieve maximal confidence when producing decision. + """ + + def __init__(self, **kwargs): + super().__init__() + self.__distance = kwargs.get("distance", DistanceBasedTIEvaluation()) + self.__local = kwargs.get("localDistance", LocalCompareTIEvaluation()) + self.__even = kwargs.get("even", EvenTIEvaluation()) + + def evaluate( + self, + aggregated_ti: SlipsThreatIntelligence, + responses: Dict[PeerId, PeerIntelligenceResponse], + trust_matrix: TrustMatrix, + **kwargs, + ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: + super()._assert_keys(responses, trust_matrix) + zero_dict = defaultdict(lambda: (None, 0, None)) + + # weight of the distance based evaluation + distance_weight = aggregated_ti.confidence + distance_data = ( + self.__distance.evaluate( + aggregated_ti, responses, trust_matrix, **kwargs + ) + if distance_weight > 0 + else zero_dict + ) + + # now we need to check if we even have some threat intelligence data + local_ti = self.__local.get_local_ti(aggregated_ti.target, **kwargs) + # weight of the local evaluation + local_weight = ( + min(1 - distance_weight, local_ti.confidence) if local_ti else 0 + ) + local_data = ( + self.__local.evaluate( + aggregated_ti, responses, trust_matrix, **kwargs + ) + if local_weight > 0 + else zero_dict + ) + + # weight of the same eval + even_weight = 1 - distance_weight - local_weight + even_data = ( + self.__even.evaluate( + aggregated_ti, responses, trust_matrix, **kwargs + ) + if even_weight > 0 + else zero_dict + ) + + def aggregate(peer: PeerId): + return ( + distance_weight * distance_data[peer][1] + + local_weight * local_data[peer][1] + + even_weight * even_data[peer][1] + ) + + return { + p.peer_id: (p, aggregate(p.peer_id), self._weight()) + for p in trust_matrix.values() + } + + +class ThresholdTIEvaluation(TIEvaluation): + """Employs DistanceBasedTIEvaluation when the confidence of the decision + is higher than given threshold. Otherwise, it uses even evaluation. + """ + + def __init__(self, **kwargs): + self.__kwargs = kwargs + self.__threshold = kwargs.get("threshold", 0.5) + self.__lower = kwargs.get("lower", EvenTIEvaluation()) + self.__higher = kwargs.get("higher", DistanceBasedTIEvaluation()) + + def evaluate( + self, + aggregated_ti: SlipsThreatIntelligence, + responses: Dict[PeerId, PeerIntelligenceResponse], + trust_matrix: TrustMatrix, + **kwargs, + ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: + super()._assert_keys(responses, trust_matrix) + + return ( + self.__higher.evaluate(aggregated_ti, responses, trust_matrix) + if self.__threshold <= aggregated_ti.confidence + else self.__lower.evaluate(aggregated_ti, responses, trust_matrix) + ) + + +EvaluationStrategy = { + "even": EvenTIEvaluation, + "distance": DistanceBasedTIEvaluation, + "localDistance": LocalCompareTIEvaluation, + "threshold": ThresholdTIEvaluation, + "maxConfidence": MaxConfidenceTIEvaluation, + "weighedDistance": WeighedDistanceToLocalTIEvaluation, +} diff --git a/modules/fidesModule/fidesModule.py b/modules/fides/fides.py similarity index 86% rename from modules/fidesModule/fidesModule.py rename to modules/fides/fides.py index 64e24df0b7..9e5cc07ae7 100644 --- a/modules/fidesModule/fidesModule.py +++ b/modules/fides/fides.py @@ -2,6 +2,9 @@ import json from pathlib import Path +from slips_files.common.output_paths import ( + get_this_filepath_inside_permanent_dir, +) from slips_files.common.slips_utils import utils from slips_files.common.abstracts.imodule import IModule from slips_files.common.parsers.config_parser import ( @@ -12,24 +15,24 @@ Alert, ) from .persistence.fides_sqlite_db import FidesSQLiteDB -from ..fidesModule.messaging.message_handler import MessageHandler -from ..fidesModule.messaging.network_bridge import NetworkBridge -from ..fidesModule.model.configuration import load_configuration -from ..fidesModule.model.threat_intelligence import SlipsThreatIntelligence -from ..fidesModule.protocols.alert import AlertProtocol -from ..fidesModule.protocols.initial_trusl import InitialTrustProtocol -from ..fidesModule.protocols.opinion import OpinionAggregator -from ..fidesModule.protocols.peer_list import PeerListUpdateProtocol -from ..fidesModule.protocols.recommendation import RecommendationProtocol -from ..fidesModule.protocols.threat_intelligence import ( +from .messaging.message_handler import MessageHandler +from .messaging.network_bridge import NetworkBridge +from .model.configuration import load_configuration +from .model.threat_intelligence import SlipsThreatIntelligence +from .protocols.alert import AlertProtocol +from .protocols.initial_trusl import InitialTrustProtocol +from .protocols.opinion import OpinionAggregator +from .protocols.peer_list import PeerListUpdateProtocol +from .protocols.recommendation import RecommendationProtocol +from .protocols.threat_intelligence import ( ThreatIntelligenceProtocol, ) -from ..fidesModule.utils.logger import LoggerPrintCallbacks -from ..fidesModule.messaging.redis_simplex_queue import RedisSimplexQueue -from ..fidesModule.persistence.threat_intelligence_db import ( +from .utils.logger import LoggerPrintCallbacks +from .messaging.redis_simplex_queue import RedisSimplexQueue +from .persistence.threat_intelligence_db import ( SlipsThreatIntelligenceDatabase, ) -from ..fidesModule.persistence.trust_db import SlipsTrustDatabase +from .persistence.trust_db import SlipsTrustDatabase class FidesModule(IModule): @@ -37,7 +40,7 @@ class FidesModule(IModule): This module ony runs when slips is running on an interface """ - name = "Fides" + name = "fides" description = "Trust computation module for P2P interactions." authors = ["David Otta", "Lukáš Forst"] @@ -58,12 +61,12 @@ def init(self): self.__intelligence: ThreatIntelligenceProtocol self.__alerts: AlertProtocol + db_name = os.path.basename(self.__trust_model_config.database) # this sqlite is shared between all runs, like a cache, - # so it shouldnt be stored in the current output dir, it should be - # in the main slips dir + # so it should not be stored in the current output dir. self.sqlite = FidesSQLiteDB( self.logger, - os.path.join(os.getcwd(), self.__trust_model_config.database), + get_this_filepath_inside_permanent_dir(db_name), ) def subscribe_to_channels(self): diff --git a/modules/fidesModule/messaging/__init__.py b/modules/fides/messaging/__init__.py similarity index 100% rename from modules/fidesModule/messaging/__init__.py rename to modules/fides/messaging/__init__.py diff --git a/modules/fidesModule/messaging/dacite/__init__.py b/modules/fides/messaging/dacite/__init__.py similarity index 100% rename from modules/fidesModule/messaging/dacite/__init__.py rename to modules/fides/messaging/dacite/__init__.py diff --git a/modules/fidesModule/messaging/dacite/cache.py b/modules/fides/messaging/dacite/cache.py similarity index 100% rename from modules/fidesModule/messaging/dacite/cache.py rename to modules/fides/messaging/dacite/cache.py diff --git a/modules/fidesModule/messaging/dacite/config.py b/modules/fides/messaging/dacite/config.py similarity index 85% rename from modules/fidesModule/messaging/dacite/config.py rename to modules/fides/messaging/dacite/config.py index 4832b84bf0..e33c4977fa 100644 --- a/modules/fidesModule/messaging/dacite/config.py +++ b/modules/fides/messaging/dacite/config.py @@ -22,4 +22,8 @@ class Config: @cached_property def hashable_forward_references(self) -> Optional[FrozenDict]: - return FrozenDict(self.forward_references) if self.forward_references else None + return ( + FrozenDict(self.forward_references) + if self.forward_references + else None + ) diff --git a/modules/fidesModule/messaging/dacite/core.py b/modules/fides/messaging/dacite/core.py similarity index 76% rename from modules/fidesModule/messaging/dacite/core.py rename to modules/fides/messaging/dacite/core.py index 71697aebc0..c56f2216c2 100644 --- a/modules/fidesModule/messaging/dacite/core.py +++ b/modules/fides/messaging/dacite/core.py @@ -1,6 +1,15 @@ from dataclasses import is_dataclass from itertools import zip_longest -from typing import TypeVar, Type, Optional, get_type_hints, Mapping, Any, Collection, MutableMapping +from typing import ( + TypeVar, + Type, + Optional, + get_type_hints, + Mapping, + Any, + Collection, + MutableMapping, +) from ..dacite.cache import cache from ..dacite.config import Config @@ -33,13 +42,13 @@ is_subclass, ) -from dataclasses import dataclass -from typing import List, Optional T = TypeVar("T") -def from_dict(data_class: Type[T], data: Data, config: Optional[Config] = None) -> T: +def from_dict( + data_class: Type[T], data: Data, config: Optional[Config] = None +) -> T: """Create a data class instance from a dictionary. :param data_class: a data class type @@ -51,7 +60,9 @@ def from_dict(data_class: Type[T], data: Data, config: Optional[Config] = None) post_init_values: MutableMapping[str, Any] = {} config = config or Config() try: - data_class_hints = cache(get_type_hints)(data_class, localns=config.hashable_forward_references) + data_class_hints = cache(get_type_hints)( + data_class, localns=config.hashable_forward_references + ) except NameError as error: raise ForwardReferenceError(str(error)) data_class_fields = cache(get_fields)(data_class) @@ -64,12 +75,16 @@ def from_dict(data_class: Type[T], data: Data, config: Optional[Config] = None) if field.name in data: try: field_data = data[field.name] - value = _build_value(type_=field_type, data=field_data, config=config) + value = _build_value( + type_=field_type, data=field_data, config=config + ) except DaciteFieldError as error: error.update_path(field.name) raise if config.check_types and not is_instance(value, field_type): - raise WrongTypeError(field_path=field.name, field_type=field_type, value=value) + raise WrongTypeError( + field_path=field.name, field_type=field_type, value=value + ) else: try: value = get_default_value_for_field(field, field_type) @@ -97,7 +112,9 @@ def _build_value(type_: Type, data: Any, config: Config) -> Any: if is_union(type_): data = _build_value_for_union(union=type_, data=data, config=config) elif is_generic_collection(type_): - data = _build_value_for_collection(collection=type_, data=data, config=config) + data = _build_value_for_collection( + collection=type_, data=data, config=config + ) elif cache(is_dataclass)(type_) and isinstance(data, Mapping): data = from_dict(data_class=type_, data=data, config=config) for cast_type in config.cast: @@ -119,7 +136,9 @@ def _build_value_for_union(union: Type, data: Any, config: Config) -> Any: try: # noinspection PyBroadException try: - value = _build_value(type_=inner_type, data=data, config=config) + value = _build_value( + type_=inner_type, data=data, config=config + ) except Exception: # pylint: disable=broad-except continue if is_instance(value, inner_type): @@ -138,21 +157,33 @@ def _build_value_for_union(union: Type, data: Any, config: Config) -> Any: raise UnionMatchError(field_type=union, value=data) -def _build_value_for_collection(collection: Type, data: Any, config: Config) -> Any: +def _build_value_for_collection( + collection: Type, data: Any, config: Config +) -> Any: data_type = data.__class__ if isinstance(data, Mapping) and is_subclass(collection, Mapping): item_type = extract_generic(collection, defaults=(Any, Any))[1] - return data_type((key, _build_value(type_=item_type, data=value, config=config)) for key, value in data.items()) + return data_type( + (key, _build_value(type_=item_type, data=value, config=config)) + for key, value in data.items() + ) elif isinstance(data, tuple) and is_subclass(collection, tuple): if not data: return data_type() types = extract_generic(collection) if len(types) == 2 and types[1] == Ellipsis: - return data_type(_build_value(type_=types[0], data=item, config=config) for item in data) + return data_type( + _build_value(type_=types[0], data=item, config=config) + for item in data + ) return data_type( - _build_value(type_=type_, data=item, config=config) for item, type_ in zip_longest(data, types) + _build_value(type_=type_, data=item, config=config) + for item, type_ in zip_longest(data, types) ) elif isinstance(data, Collection) and is_subclass(collection, Collection): item_type = extract_generic(collection, defaults=(Any,))[0] - return data_type(_build_value(type_=item_type, data=item, config=config) for item in data) + return data_type( + _build_value(type_=item_type, data=item, config=config) + for item in data + ) return data diff --git a/modules/fidesModule/messaging/dacite/data.py b/modules/fides/messaging/dacite/data.py similarity index 100% rename from modules/fidesModule/messaging/dacite/data.py rename to modules/fides/messaging/dacite/data.py diff --git a/modules/fidesModule/messaging/dacite/dataclasses.py b/modules/fides/messaging/dacite/dataclasses.py similarity index 86% rename from modules/fidesModule/messaging/dacite/dataclasses.py rename to modules/fides/messaging/dacite/dataclasses.py index 8f976d8fea..1debc38d3c 100644 --- a/modules/fidesModule/messaging/dacite/dataclasses.py +++ b/modules/fides/messaging/dacite/dataclasses.py @@ -24,7 +24,11 @@ def get_default_value_for_field(field: Field, type_: Type) -> Any: @cache def get_fields(data_class: Type[T]) -> List[Field]: fields = getattr(data_class, _FIELDS) - return [f for f in fields.values() if f._field_type is _FIELD or f._field_type is _FIELD_INITVAR] + return [ + f + for f in fields.values() + if f._field_type is _FIELD or f._field_type is _FIELD_INITVAR + ] @cache diff --git a/modules/fidesModule/messaging/dacite/exceptions.py b/modules/fides/messaging/dacite/exceptions.py similarity index 83% rename from modules/fidesModule/messaging/dacite/exceptions.py rename to modules/fides/messaging/dacite/exceptions.py index de96d0bd72..2b5e1fc7b9 100644 --- a/modules/fidesModule/messaging/dacite/exceptions.py +++ b/modules/fides/messaging/dacite/exceptions.py @@ -3,7 +3,11 @@ def _name(type_: Type) -> str: - return type_.__name__ if hasattr(type_, "__name__") and not is_union(type_) else str(type_) + return ( + type_.__name__ + if hasattr(type_, "__name__") and not is_union(type_) + else str(type_) + ) class DaciteError(Exception): @@ -23,7 +27,9 @@ def update_path(self, parent_field_path: str) -> None: class WrongTypeError(DaciteFieldError): - def __init__(self, field_type: Type, value: Any, field_path: Optional[str] = None) -> None: + def __init__( + self, field_type: Type, value: Any, field_path: Optional[str] = None + ) -> None: super().__init__(field_path=field_path) self.field_type = field_type self.value = value @@ -52,12 +58,16 @@ def __str__(self) -> str: class StrictUnionMatchError(DaciteFieldError): - def __init__(self, union_matches: Dict[Type, Any], field_path: Optional[str] = None) -> None: + def __init__( + self, union_matches: Dict[Type, Any], field_path: Optional[str] = None + ) -> None: super().__init__(field_path=field_path) self.union_matches = union_matches def __str__(self) -> str: - conflicting_types = ", ".join(_name(type_) for type_ in self.union_matches) + conflicting_types = ", ".join( + _name(type_) for type_ in self.union_matches + ) return f'can not choose between possible Union matches for field "{self.field_path}": {conflicting_types}' diff --git a/modules/fidesModule/messaging/dacite/frozen_dict.py b/modules/fides/messaging/dacite/frozen_dict.py similarity index 100% rename from modules/fidesModule/messaging/dacite/frozen_dict.py rename to modules/fides/messaging/dacite/frozen_dict.py diff --git a/modules/fidesModule/messaging/dacite/py.typed b/modules/fides/messaging/dacite/py.typed similarity index 100% rename from modules/fidesModule/messaging/dacite/py.typed rename to modules/fides/messaging/dacite/py.typed diff --git a/modules/fidesModule/messaging/dacite/types.py b/modules/fides/messaging/dacite/types.py similarity index 88% rename from modules/fidesModule/messaging/dacite/types.py rename to modules/fides/messaging/dacite/types.py index 4a96fa43f7..fa49773c4a 100644 --- a/modules/fidesModule/messaging/dacite/types.py +++ b/modules/fides/messaging/dacite/types.py @@ -31,7 +31,11 @@ def is_optional(type_: Type) -> bool: @cache def extract_optional(optional: Type[Optional[T]]) -> T: - other_members = [member for member in extract_generic(optional) if member is not type(None)] + other_members = [ + member + for member in extract_generic(optional) + if member is not type(None) + ] if other_members: return typing_cast(T, Union[tuple(other_members)]) else: @@ -97,7 +101,9 @@ def extract_init_var(type_: Type) -> Union[Type, Any]: def is_instance(value: Any, type_: Type) -> bool: try: # As described in PEP 484 - section: "The numeric tower" - if (type_ in [float, complex] and isinstance(value, (int, float))) or isinstance(value, type_): + if ( + type_ in [float, complex] and isinstance(value, (int, float)) + ) or isinstance(value, type_): return True except TypeError: pass @@ -120,14 +126,22 @@ def is_instance(value: Any, type_: Type) -> bool: else: if len(tuple_types) != len(value): return False - return all(is_instance(item, item_type) for item, item_type in zip(value, tuple_types)) + return all( + is_instance(item, item_type) + for item, item_type in zip(value, tuple_types) + ) if isinstance(value, Mapping): key_type, val_type = extract_generic(type_, defaults=(Any, Any)) for key, val in value.items(): - if not is_instance(key, key_type) or not is_instance(val, val_type): + if not is_instance(key, key_type) or not is_instance( + val, val_type + ): return False return True - return all(is_instance(item, extract_generic(type_, defaults=(Any,))[0]) for item in value) + return all( + is_instance(item, extract_generic(type_, defaults=(Any,))[0]) + for item in value + ) elif is_new_type(type_): return is_instance(value, extract_new_type(type_)) elif is_literal(type_): diff --git a/modules/fides/messaging/message_handler.py b/modules/fides/messaging/message_handler.py new file mode 100644 index 0000000000..24a2f7fd87 --- /dev/null +++ b/modules/fides/messaging/message_handler.py @@ -0,0 +1,225 @@ +from typing import Dict, List, Callable, Optional, Union + + +from ..messaging.dacite import from_dict + +from ..messaging.model import ( + NetworkMessage, + PeerInfo, + PeerIntelligenceResponse, + PeerRecommendationResponse, +) +from ..model.alert import Alert +from ..model.aliases import PeerId, Target +from ..model.recommendation import Recommendation +from ..model.threat_intelligence import ThreatIntelligence +from ..utils.logger import Logger + +logger = Logger(__name__) + + +class MessageHandler: + """ + Class responsible for parsing messages and handling requests coming from the queue. + + The entrypoint is on_message. + """ + + # def print(self, *args, **kwargs): + # return self.printer.print(*args, **kwargs) + + version = 1 + + def __init__( + self, + on_peer_list_update: Callable[[List[PeerInfo]], None], + on_recommendation_request: Callable[[str, PeerInfo, PeerId], None], + on_recommendation_response: Callable[ + [List[PeerRecommendationResponse]], None + ], + on_alert: Callable[[PeerInfo, Alert], None], + on_intelligence_request: Callable[[str, PeerInfo, Target], None], + on_intelligence_response: Callable[ + [List[PeerIntelligenceResponse]], None + ], + on_unknown: Optional[Callable[[NetworkMessage], None]] = None, + on_error: Optional[ + Callable[[Union[str, NetworkMessage], Exception], None] + ] = None, + ): + # self.logger = None + self.__on_peer_list_update_callback = on_peer_list_update + self.__on_recommendation_request_callback = on_recommendation_request + self.__on_recommendation_response_callback = on_recommendation_response + self.__on_alert_callback = on_alert + self.__on_intelligence_request_callback = on_intelligence_request + self.__on_intelligence_response_callback = on_intelligence_response + self.__on_unknown_callback = on_unknown + self.__on_error = on_error + # self.printer = Printer(self.logger, self.name) + + def on_message(self, message: NetworkMessage): + """ + Entry point for generic messages coming from the queue. + This method parses the message and then executes correct procedure from event. + :param message: message from the queue + :return: value from the underlining function from the constructor + """ + if message.version != self.version: + logger.warn( + f"Unknown message version! This handler supports {self.version}.", + message, + ) + return self.__on_unknown_message(message) + + execution_map = { + "nl2tl_peers_list": self.__on_nl2tl_peer_list, + "nl2tl_recommendation_request": self.__on_nl2tl_recommendation_request, + "nl2tl_recommendation_response": self.__on_nl2tl_recommendation_response, + "nl2tl_alert": self.__on_nl2tl_alert, + "nl2tl_intelligence_request": self.__on_nl2tl_intelligence_request, + "nl2tl_intelligence_response": self.__on_nl2tl_intelligence_response, + } + func = execution_map.get( + message.type, lambda data: self.__on_unknown_message(message) + ) + # we want to handle everything + # noinspection PyBroadException + try: + # we know that the functions can handle that, and if not, there's always error handling + # noinspection PyArgumentList + return func(message.data) + except Exception as ex: + logger.error( + f"Error when executing handler for message: {message.type}.", + ex, + ) + if self.__on_error: + return self.__on_error(message, ex) + + def on_error( + self, original_data: str, exception: Optional[Exception] = None + ): + """ + Should be executed when it was not possible to parse the message. + :param original_data: string received from the queue + :param exception: exception that occurred during handling + :return: + """ + logger.error(f"Unknown data received: {original_data}.") + if self.__on_error: + self.__on_error( + original_data, + exception if exception else Exception("Unknown data type!"), + ) + + def __on_unknown_message(self, message: NetworkMessage): + logger.warn("Unknown message handler executed!") + logger.debug("Message:", message) + + if self.__on_unknown_callback is not None: + self.__on_unknown_callback(message) + + def __on_nl2tl_peer_list(self, data: Dict): + logger.debug("nl2tl_peer_list message") + + peers = [ + from_dict(data_class=PeerInfo, data=peer) for peer in data["peers"] + ] + return self.__on_peer_list_update(peers) + + def __on_peer_list_update(self, peers: List[PeerInfo]): + return self.__on_peer_list_update_callback(peers) + + def __on_nl2tl_recommendation_request(self, data: Dict): + logger.debug("nl2tl_recommendation_request message") + + request_id = data["request_id"] + sender = from_dict(data_class=PeerInfo, data=data["sender"]) + subject = data["payload"] + return self.__on_recommendation_request(request_id, sender, subject) + + def __on_recommendation_request( + self, request_id: str, sender: PeerInfo, subject: PeerId + ): + return self.__on_recommendation_request_callback( + request_id, sender, subject + ) + + def __on_nl2tl_recommendation_response(self, data: List[Dict]): + logger.debug("nl2tl_recommendation_response message") + + responses = [ + PeerRecommendationResponse( + sender=from_dict(data_class=PeerInfo, data=single["sender"]), + subject=single["payload"]["subject"], + recommendation=from_dict( + data_class=Recommendation, + data=single["payload"]["recommendation"], + ), + ) + for single in data + ] + return self.__on_recommendation_response(responses) + + def __on_recommendation_response( + self, recommendations: List[PeerRecommendationResponse] + ): + return self.__on_recommendation_response_callback(recommendations) + + def __on_nl2tl_alert(self, data: Dict): + logger.debug("nl2tl_alert message") + + sender = from_dict(data_class=PeerInfo, data=data["sender"]) + alert = from_dict(data_class=Alert, data=data["payload"]) + return self.__on_alert(sender, alert) + + def __on_alert(self, sender: PeerInfo, alert: Alert): + return self.__on_alert_callback(sender, alert) + + def __on_nl2tl_intelligence_request(self, data: Dict): + logger.debug("nl2tl_intelligence_request message") + + request_id = data["request_id"] + sender = from_dict(data_class=PeerInfo, data=data["sender"]) + target = data["payload"] + return self.__on_intelligence_request(request_id, sender, target) + + def __on_intelligence_request( + self, request_id: str, sender: PeerInfo, target: Target + ): + return self.__on_intelligence_request_callback( + request_id, sender, target + ) + + def __on_nl2tl_intelligence_response(self, data: Dict): + logger.debug("nl2tl_intelligence_response message") + + responses = [] + + try: + responses = [ + PeerIntelligenceResponse( + sender=from_dict( + data_class=PeerInfo, data=single["sender"] + ), + intelligence=from_dict( + data_class=ThreatIntelligence, + data=single["payload"]["intelligence"], + ), + target=single["payload"]["target"], + ) + for single in data + ] + except Exception as e: + print( + "Error in Fides message_handler.py __on_nl2tl_intelligence_response(): ", + e.__str__(), + ) + # self.print("Error in Fides message_handler.py __on_nl2tl_intelligence_response(): ") + return self.__on_intelligence_response(responses) + + def __on_intelligence_response( + self, responses: List[PeerIntelligenceResponse] + ): + return self.__on_intelligence_response_callback(responses) diff --git a/modules/fidesModule/messaging/model.py b/modules/fides/messaging/model.py similarity index 84% rename from modules/fidesModule/messaging/model.py rename to modules/fides/messaging/model.py index e36b6c0a04..f8131caab4 100644 --- a/modules/fidesModule/messaging/model.py +++ b/modules/fides/messaging/model.py @@ -6,10 +6,8 @@ from ..model.recommendation import Recommendation from ..model.threat_intelligence import ThreatIntelligence -""" -Model data coming from the Redis queue - -communication layer between network and trust layer. -""" +# Model data coming from the Redis queue - +# communication layer between network and trust layer. @dataclass diff --git a/modules/fidesModule/messaging/network_bridge.py b/modules/fides/messaging/network_bridge.py similarity index 100% rename from modules/fidesModule/messaging/network_bridge.py rename to modules/fides/messaging/network_bridge.py diff --git a/modules/fidesModule/messaging/queue.py b/modules/fides/messaging/queue.py similarity index 77% rename from modules/fidesModule/messaging/queue.py rename to modules/fides/messaging/queue.py index 1ea8728f78..b21d3b26ee 100644 --- a/modules/fidesModule/messaging/queue.py +++ b/modules/fides/messaging/queue.py @@ -10,11 +10,11 @@ class Queue: def send(self, serialized_data: str, **argv): """Sends serialized data to the queue.""" - raise NotImplemented('This is interface. Use implementation.') + raise NotImplementedError("This is interface. Use implementation.") def listen(self, on_message: Callable[[str], None], **argv): """Starts listening, executes :param: on_message when new message arrives. Depending on the implementation, this method might be blocking. """ - raise NotImplemented('This is interface. Use implementation.') + raise NotImplementedError("This is interface. Use implementation.") diff --git a/modules/fidesModule/messaging/queue_in_memory.py b/modules/fides/messaging/queue_in_memory.py similarity index 66% rename from modules/fidesModule/messaging/queue_in_memory.py rename to modules/fides/messaging/queue_in_memory.py index ae08db2f81..06c2e2724c 100644 --- a/modules/fidesModule/messaging/queue_in_memory.py +++ b/modules/fides/messaging/queue_in_memory.py @@ -17,15 +17,21 @@ def __init__(self, on_message: Optional[Callable[[str], None]] = None): def default_on_message(data: str): InMemoryQueue.__exception(data) - self.__on_message: Callable[[str], None] = on_message if on_message else default_on_message + self.__on_message: Callable[[str], None] = ( + on_message if on_message else default_on_message + ) - def send(self, serialized_data: str, should_wait_for_join: bool = False, **argv): + def send( + self, serialized_data: str, should_wait_for_join: bool = False, **argv + ): """Sends serialized data to the queue.""" - logger.debug('New data received for send.') + logger.debug("New data received for send.") if self.__on_message is None: self.__exception(serialized_data) - th = threading.Thread(target=lambda: self.__on_message(serialized_data)) + th = threading.Thread( + target=lambda: self.__on_message(serialized_data) + ) th.start() if should_wait_for_join: th.join() @@ -40,4 +46,6 @@ def listen(self, on_message: Callable[[str], None], **argv): @staticmethod def __exception(data: str): - raise Exception(f'No on_message set! Call listen before calling send! Data: {data}') + raise Exception( + f"No on_message set! Call listen before calling send! Data: {data}" + ) diff --git a/modules/fidesModule/messaging/redis_simplex_queue.py b/modules/fides/messaging/redis_simplex_queue.py similarity index 98% rename from modules/fidesModule/messaging/redis_simplex_queue.py rename to modules/fides/messaging/redis_simplex_queue.py index 4fd1524169..a1c449f0f6 100644 --- a/modules/fidesModule/messaging/redis_simplex_queue.py +++ b/modules/fides/messaging/redis_simplex_queue.py @@ -5,8 +5,6 @@ from slips_files.core.database.database_manager import DBManager from ..messaging.queue import Queue from ..utils.logger import Logger -from dataclasses import dataclass -from typing import List, Optional logger = Logger(__name__) diff --git a/modules/fidesModule/model/__init__.py b/modules/fides/model/__init__.py similarity index 100% rename from modules/fidesModule/model/__init__.py rename to modules/fides/model/__init__.py diff --git a/modules/fidesModule/model/alert.py b/modules/fides/model/alert.py similarity index 100% rename from modules/fidesModule/model/alert.py rename to modules/fides/model/alert.py diff --git a/modules/fides/model/aliases.py b/modules/fides/model/aliases.py new file mode 100644 index 0000000000..3c9762d33f --- /dev/null +++ b/modules/fides/model/aliases.py @@ -0,0 +1,26 @@ +IP = str +# IPv4, IPv6 in string representation. + +Domain = str +# Host Name, Domain. + +PeerId = str +# String representation of peer's public key. + +OrganisationId = str +# String representation of organisation ID. + +Target = str +# Intelligence Target - domain or IP. + +ConfidentialityLevel = float +# Confidentiality level for threat intelligence. +# If an entity needs to have access to any data, it must mean +# entity.confidentiality_level >= data.confidentiality_level +# thus level 0 means accessible for everybody + +Score = float +# Score for the target, -1 <= score <= 1 + +Confidence = float +# Confidence in score, 0 <= confidence <= 1 diff --git a/modules/fidesModule/model/configuration.py b/modules/fides/model/configuration.py similarity index 57% rename from modules/fidesModule/model/configuration.py rename to modules/fides/model/configuration.py index 99bfe71e0d..685e91db48 100644 --- a/modules/fidesModule/model/configuration.py +++ b/modules/fides/model/configuration.py @@ -13,9 +13,9 @@ class PrivacyLevel: """Name of the level.""" value: float """Value used for comparison. - + 0 <= value <= 1 - + (there can be a case where value > 1 but that means the data won't be ever send) """ @@ -43,8 +43,8 @@ class TrustedEntity: """Initial trust for the entity. If, "enforce_trust = false" this value will change during time as the instance has more interactions with - organisation nodes. If "enforce_trust = true", the trust for all peers from this entity will remain - the same. + organisation nodes. If "enforce_trust = true", the trust for all peers from this entity will remain + the same. """ enforce_trust: bool @@ -100,7 +100,7 @@ class TrustModelConfiguration: service_history_max_size: int """Maximal size of Service History. - + In model's notation sh_max. """ @@ -135,70 +135,102 @@ def load_configuration(file_path: str) -> TrustModelConfiguration: with open(file_path, "r") as stream: try: import yaml + return __parse_config(yaml.safe_load(stream)) except Exception as exc: - Logger('config_loader').error(f"It was not possible to load file! {exc}.") + Logger("config_loader").error( + f"It was not possible to load file! {exc}." + ) raise exc def __parse_config(data: dict) -> TrustModelConfiguration: return TrustModelConfiguration( - privacy_levels=[PrivacyLevel(name=level['name'], - value=level['value']) - for level in data['confidentiality']['levels']], - confidentiality_thresholds=[ConfidentialityThreshold(level=threshold['level'], - required_trust=threshold['requiredTrust']) - for threshold in data['confidentiality']['thresholds']], - data_default_level=data['confidentiality']['defaultLevel'], - initial_reputation=data['trust']['service']['initialReputation'], - service_history_max_size=data['trust']['service']['historyMaxSize'], + privacy_levels=[ + PrivacyLevel(name=level["name"], value=level["value"]) + for level in data["confidentiality"]["levels"] + ], + confidentiality_thresholds=[ + ConfidentialityThreshold( + level=threshold["level"], + required_trust=threshold["requiredTrust"], + ) + for threshold in data["confidentiality"]["thresholds"] + ], + data_default_level=data["confidentiality"]["defaultLevel"], + initial_reputation=data["trust"]["service"]["initialReputation"], + service_history_max_size=data["trust"]["service"]["historyMaxSize"], recommendations=RecommendationsConfiguration( - enabled=data['trust']['recommendations']['enabled'], - only_connected=data['trust']['recommendations']['useOnlyConnected'], - only_preconfigured=data['trust']['recommendations']['useOnlyPreconfigured'], - required_trusted_peers_count=data['trust']['recommendations']['requiredTrustedPeersCount'], - trusted_peer_threshold=data['trust']['recommendations']['trustedPeerThreshold'], - peers_max_count=data['trust']['recommendations']['peersMaxCount'], - history_max_size=data['trust']['recommendations']['historyMaxSize'] + enabled=data["trust"]["recommendations"]["enabled"], + only_connected=data["trust"]["recommendations"][ + "useOnlyConnected" + ], + only_preconfigured=data["trust"]["recommendations"][ + "useOnlyPreconfigured" + ], + required_trusted_peers_count=data["trust"]["recommendations"][ + "requiredTrustedPeersCount" + ], + trusted_peer_threshold=data["trust"]["recommendations"][ + "trustedPeerThreshold" + ], + peers_max_count=data["trust"]["recommendations"]["peersMaxCount"], + history_max_size=data["trust"]["recommendations"][ + "historyMaxSize" + ], ), - alert_trust_from_unknown=data['trust']['alert']['defaultTrust'], - trusted_peers=[TrustedEntity(id=e['id'], - name=e['name'], - trust=e['trust'], - enforce_trust=e['enforceTrust'], - confidentiality_level=e['confidentialityLevel']) - for e in data['trust']['peers']], - trusted_organisations=[TrustedEntity(id=e['id'], - name=e['name'], - trust=e['trust'], - enforce_trust=e['enforceTrust'], - confidentiality_level=e['confidentialityLevel']) - for e in data['trust']['organisations']], - network_opinion_cache_valid_seconds=data['trust']['networkOpinionCacheValidSeconds'], + alert_trust_from_unknown=data["trust"]["alert"]["defaultTrust"], + trusted_peers=[ + TrustedEntity( + id=e["id"], + name=e["name"], + trust=e["trust"], + enforce_trust=e["enforceTrust"], + confidentiality_level=e["confidentialityLevel"], + ) + for e in data["trust"]["peers"] + ], + trusted_organisations=[ + TrustedEntity( + id=e["id"], + name=e["name"], + trust=e["trust"], + enforce_trust=e["enforceTrust"], + confidentiality_level=e["confidentialityLevel"], + ) + for e in data["trust"]["organisations"] + ], + network_opinion_cache_valid_seconds=data["trust"][ + "networkOpinionCacheValidSeconds" + ], interaction_evaluation_strategy=__parse_evaluation_strategy(data), - ti_aggregation_strategy=TIAggregationStrategy[data['trust']['tiAggregationStrategy']](), - database=data['database'] if 'database' in data else "fides_p2p_db.sqlite", + ti_aggregation_strategy=TIAggregationStrategy[ + data["trust"]["tiAggregationStrategy"] + ](), + database=( + data["database"] if "database" in data else "fides_p2p_db.sqlite" + ), ) def __parse_evaluation_strategy(data: dict) -> TIEvaluation: - strategies = data['trust']['interactionEvaluationStrategies'] + strategies = data["trust"]["interactionEvaluationStrategies"] def get_strategy_for_key(key: str) -> TIEvaluation: kwargs = strategies[key] kwargs = kwargs if kwargs else {} # there's special handling as this one combines multiple of them - if key == 'threshold': - kwargs['lower'] = get_strategy_for_key(kwargs['lower']) - kwargs['higher'] = get_strategy_for_key(kwargs['higher']) - elif key == 'maxConfidence': - kwargs['distance'] = get_strategy_for_key('distance') - kwargs['localDistance'] = get_strategy_for_key('localDistance') - kwargs['even'] = get_strategy_for_key('even') - elif key == 'weighedDistance': - kwargs['distance'] = get_strategy_for_key('distance') - kwargs['localDistance'] = get_strategy_for_key('localDistance') + if key == "threshold": + kwargs["lower"] = get_strategy_for_key(kwargs["lower"]) + kwargs["higher"] = get_strategy_for_key(kwargs["higher"]) + elif key == "maxConfidence": + kwargs["distance"] = get_strategy_for_key("distance") + kwargs["localDistance"] = get_strategy_for_key("localDistance") + kwargs["even"] = get_strategy_for_key("even") + elif key == "weighedDistance": + kwargs["distance"] = get_strategy_for_key("distance") + kwargs["localDistance"] = get_strategy_for_key("localDistance") return EvaluationStrategy[key](**kwargs) - return get_strategy_for_key(strategies['used']) + return get_strategy_for_key(strategies["used"]) diff --git a/modules/fidesModule/model/peer.py b/modules/fides/model/peer.py similarity index 86% rename from modules/fidesModule/model/peer.py rename to modules/fides/model/peer.py index bb7dcb3372..58cd4e10fd 100644 --- a/modules/fidesModule/model/peer.py +++ b/modules/fides/model/peer.py @@ -18,16 +18,16 @@ class PeerInfo: ip: Optional[IP] = None """Ip address of the peer, if we know it. - There are cases when we don't know the IP of the peer - when running behind NAT + There are cases when we don't know the IP of the peer - when running behind NAT or when the peers used TURN server to connect to each other. """ def to_dict(self): """Convert to dictionary for serialization.""" return { - 'id': self.id, - 'organisations': [org for org in self.organisations], - 'ip': self.ip, + "id": self.id, + "organisations": [org for org in self.organisations], + "ip": self.ip, } @classmethod diff --git a/modules/fidesModule/model/peer_trust_data.py b/modules/fides/model/peer_trust_data.py similarity index 79% rename from modules/fidesModule/model/peer_trust_data.py rename to modules/fides/model/peer_trust_data.py index c2032826e3..b78a27c687 100644 --- a/modules/fidesModule/model/peer_trust_data.py +++ b/modules/fides/model/peer_trust_data.py @@ -19,28 +19,28 @@ class PeerTrustData: service_trust: float """Service Trust Metric. - + Semantic meaning is basically "trust" - how much does current peer trust peer "j" about quality of service. In model's notation st_ij. - + 0 <= service_trust <= 1 """ reputation: float """Reputation Metric. - + The reputation metric measures a stranger’s trustworthiness based on recommendations. In model's notation r_ij. - + 0 <= reputation <= 1 """ recommendation_trust: float """Recommendation Trust Metric. - + How much does the peer trust that any recommendation received from this peer is correct. In model's notation rt_ij. - + 0 <= recommendation_trust <= 1 """ @@ -64,7 +64,7 @@ class PeerTrustData: initial_reputation_provided_by_count: int """How many peers provided recommendation during initial calculation of reputation. - + In model's notation η_ij. """ @@ -104,8 +104,12 @@ def to_dict(self, remove_histories: bool = False): "competence_belief": self.competence_belief, "integrity_belief": self.integrity_belief, "initial_reputation_provided_by_count": self.initial_reputation_provided_by_count, - "service_history": [sh.to_dict() for sh in self.service_history], # Assuming ServiceHistory has to_dict - "recommendation_history": [rh.to_dict() for rh in self.recommendation_history] # Assuming RecommendationHistory has to_dict + "service_history": [ + sh.to_dict() for sh in self.service_history + ], # Assuming ServiceHistory has to_dict + "recommendation_history": [ + rh.to_dict() for rh in self.recommendation_history + ], # Assuming RecommendationHistory has to_dict } if remove_histories: @@ -118,27 +122,38 @@ def to_dict(self, remove_histories: bool = False): @classmethod def from_dict(cls, data): return cls( - info=PeerInfo.from_dict(data["info"]), # Assuming PeerInfo has from_dict method + info=PeerInfo.from_dict( + data["info"] + ), # Assuming PeerInfo has from_dict method has_fixed_trust=data["has_fixed_trust"], service_trust=data["service_trust"], reputation=data["reputation"], recommendation_trust=data["recommendation_trust"], competence_belief=data["competence_belief"], integrity_belief=data["integrity_belief"], - initial_reputation_provided_by_count=data["initial_reputation_provided_by_count"], - service_history=[ServiceHistory.from_dict(sh) for sh in data["service_history"]], + initial_reputation_provided_by_count=data[ + "initial_reputation_provided_by_count" + ], + service_history=[ + ServiceHistory.from_dict(sh) for sh in data["service_history"] + ], # Assuming ServiceHistory has from_dict - recommendation_history=[RecommendationHistory.from_dict(rh) for rh in data["recommendation_history"]] + recommendation_history=[ + RecommendationHistory.from_dict(rh) + for rh in data["recommendation_history"] + ], # Assuming RecommendationHistory has from_dict ) TrustMatrix = Dict[PeerId, PeerTrustData] -"""Matrix that have PeerId as a key and then value is data about trust we have.""" +# Matrix that have PeerId as a key and then value is data about trust we have. -def trust_data_prototype(peer: PeerInfo, has_fixed_trust: bool = False) -> PeerTrustData: - """Creates clear trust object with 0 values and given peer info.""" +def trust_data_prototype( + peer: PeerInfo, has_fixed_trust: bool = False +) -> PeerTrustData: + # Creates clear trust object with 0 values and given peer info. return PeerTrustData( info=peer, has_fixed_trust=has_fixed_trust, @@ -149,5 +164,5 @@ def trust_data_prototype(peer: PeerInfo, has_fixed_trust: bool = False) -> PeerT integrity_belief=0, initial_reputation_provided_by_count=0, service_history=[], - recommendation_history=[] + recommendation_history=[], ) diff --git a/modules/fidesModule/model/recommendation.py b/modules/fides/model/recommendation.py similarity index 96% rename from modules/fidesModule/model/recommendation.py rename to modules/fides/model/recommendation.py index 6b6c9d9371..9a7d64d24d 100644 --- a/modules/fidesModule/model/recommendation.py +++ b/modules/fides/model/recommendation.py @@ -7,38 +7,38 @@ class Recommendation: competence_belief: float """How much is peer satisfied with historical service interactions. - + In general, this is expected mean behavior of the peer. In model's notation cb_kj. - + 0 <= competence_belief <= 1 """ integrity_belief: float """How much is peer consistent in its behavior. - + In general, this is standard deviation from the mean behavior. In model's notation ib_kj. - + 0 <= integrity_belief <= 1 """ service_history_size: int """Size of service interaction history. - + In model's notation sh_kj. """ recommendation: float """Recommendation about reputation. - + In model's notation r_kj. - + 0 <= recommendation <= 1 """ initial_reputation_provided_by_count: int """How many peers which provided recommendation during the initial calculation of r_kj. - + In model's notation η_kj. """ diff --git a/modules/fides/model/recommendation_history.py b/modules/fides/model/recommendation_history.py new file mode 100644 index 0000000000..6ee1b02f12 --- /dev/null +++ b/modules/fides/model/recommendation_history.py @@ -0,0 +1,42 @@ +from dataclasses import dataclass +from typing import List + +from ..utils.time import Time + + +@dataclass +class RecommendationHistoryRecord: + # Represents an evaluation of a single recommendation interaction between peer i and peer j. + + satisfaction: float + # Peer's satisfaction with the recommendation. In model's notation rs_ij. + # 0 <= satisfaction <= 1 + + weight: float + # Weight of the recommendation. In model's notation rw_ij. + # 0 <= weight <= 1 + + timestamp: Time + # Date time when this recommendation happened. + + def to_dict(self): + # Convert the instance to a dictionary. + return { + "satisfaction": self.satisfaction, + "weight": self.weight, + "timestamp": self.timestamp, # Keep as float + } + + @classmethod + def from_dict(cls, dict_obj): + # Create an instance of RecommendationHistoryRecord from a dictionary. + return cls( + satisfaction=dict_obj["satisfaction"], + weight=dict_obj["weight"], + timestamp=dict_obj["timestamp"], # Keep as float + ) + + +RecommendationHistory = List[RecommendationHistoryRecord] +# Ordered list with history of recommendation interactions. +# First element in the list is the oldest one. diff --git a/modules/fides/model/service_history.py b/modules/fides/model/service_history.py new file mode 100644 index 0000000000..190674e9c8 --- /dev/null +++ b/modules/fides/model/service_history.py @@ -0,0 +1,44 @@ +from dataclasses import dataclass +from typing import List + +from ..utils.time import Time + + +@dataclass +class ServiceHistoryRecord: + # Represents an evaluation of a single service interaction between peer i and peer j. + + satisfaction: float + # Peer's satisfaction with the service. In model's notation s_ij. + # 0 <= satisfaction <= 1 + + weight: float + # Weight of the service interaction. In model's notation w_ij. + # 0 <= weight <= 1 + + timestamp: Time + # Date time when this interaction happened. + + def to_dict(self): + # Convert the instance to a dictionary. + return { + "satisfaction": self.satisfaction, + "weight": self.weight, + "timestamp": self.timestamp, + } + + @classmethod + def from_dict(cls, dict_obj): + # Create an instance of ServiceHistoryRecord from a dictionary. + return cls( + satisfaction=dict_obj["satisfaction"], + weight=dict_obj["weight"], + timestamp=dict_obj[ + "timestamp" + ], # Convert ISO format back to datetime + ) + + +ServiceHistory = List[ServiceHistoryRecord] +# Ordered list with history of service interactions. +# First element in the list is the oldest one. diff --git a/modules/fidesModule/model/threat_intelligence.py b/modules/fides/model/threat_intelligence.py similarity index 81% rename from modules/fidesModule/model/threat_intelligence.py rename to modules/fides/model/threat_intelligence.py index 3f439c054f..511c967835 100644 --- a/modules/fidesModule/model/threat_intelligence.py +++ b/modules/fides/model/threat_intelligence.py @@ -10,13 +10,13 @@ class ThreatIntelligence: score: Score """How much is subject malicious or benign. - + -1 <= score <= 1 """ confidence: Confidence """How much does peer trust, that score is correct. - + 0 <= confidence <= 1 """ @@ -44,7 +44,13 @@ def to_dict(self): def from_dict(cls, data: dict): return cls( target=data["target"], - confidentiality=float(data["confidentiality"]) if data.get("confidentiality") else None, + confidentiality=( + float(data["confidentiality"]) + if data.get("confidentiality") + else None + ), score=float(data["score"]) if data.get("score") else None, - confidence=float(data["confidence"]) if data.get("confidence") else None + confidence=( + float(data["confidence"]) if data.get("confidence") else None + ), ) diff --git a/modules/fidesModule/originals/__init__.py b/modules/fides/originals/__init__.py similarity index 100% rename from modules/fidesModule/originals/__init__.py rename to modules/fides/originals/__init__.py diff --git a/modules/fidesModule/originals/abstracts.py b/modules/fides/originals/abstracts.py similarity index 72% rename from modules/fidesModule/originals/abstracts.py rename to modules/fides/originals/abstracts.py index 699575d32a..2c43188af4 100644 --- a/modules/fidesModule/originals/abstracts.py +++ b/modules/fides/originals/abstracts.py @@ -8,22 +8,22 @@ # This is the abstract Module class to check against. Do not modify class Module(object): - name = '' - description = 'Template abstract originals' - authors = ['Template abstract Author'] + name = "" + description = "Template abstract originals" + authors = ["Template abstract Author"] output = [] def __init__(self): pass def usage(self): - print('Usage') + print("Usage") def help(self): - print('Help') + print("Help") def run(self): try: - print('test') - except Exception as e: - print('error') + print("test") + except Exception: + print("error") diff --git a/modules/fidesModule/originals/database.py b/modules/fides/originals/database.py similarity index 82% rename from modules/fidesModule/originals/database.py rename to modules/fides/originals/database.py index fab26689cd..f7bddc3e49 100644 --- a/modules/fidesModule/originals/database.py +++ b/modules/fides/originals/database.py @@ -6,13 +6,13 @@ class Database(object): - """ Database object management """ + """Database object management""" def __init__(self): self.r: Redis def start(self, slip_conf): - raise NotImplemented('Use real implementation for Slips!') + raise NotImplementedError("Use real implementation for Slips!") __database__ = Database() diff --git a/modules/fidesModule/protocols/__init__.py b/modules/fides/persistence/__init__.py similarity index 100% rename from modules/fidesModule/protocols/__init__.py rename to modules/fides/persistence/__init__.py diff --git a/modules/fidesModule/persistence/fides_sqlite_db.py b/modules/fides/persistence/fides_sqlite_db.py similarity index 93% rename from modules/fidesModule/persistence/fides_sqlite_db.py rename to modules/fides/persistence/fides_sqlite_db.py index 5906741012..d4a222775c 100644 --- a/modules/fidesModule/persistence/fides_sqlite_db.py +++ b/modules/fides/persistence/fides_sqlite_db.py @@ -20,7 +20,7 @@ class FidesSQLiteDB: _lock = threading.RLock() - name = "Fides SQLiteDB" + name = "fides_sqlite_db" def __init__(self, logger: Output, db_path: str) -> None: """ @@ -104,6 +104,37 @@ def store_slips_threat_intelligence( def store_peer_trust_data(self, peer_trust_data: PeerTrustData) -> None: with FidesSQLiteDB._lock: + self.__execute_query( + """ + DELETE FROM PeerTrustServiceHistory + WHERE peer_trust_data_id IN ( + SELECT id FROM PeerTrustData WHERE peerID = ? + ); + """, + [peer_trust_data.info.id], + ) + self.__execute_query( + """ + DELETE FROM PeerTrustRecommendationHistory + WHERE peer_trust_data_id IN ( + SELECT id FROM PeerTrustData WHERE peerID = ? + ); + """, + [peer_trust_data.info.id], + ) + self.__execute_query( + "DELETE FROM ServiceHistory WHERE peerID = ?;", + [peer_trust_data.info.id], + ) + self.__execute_query( + "DELETE FROM RecommendationHistory WHERE peerID = ?;", + [peer_trust_data.info.id], + ) + self.__execute_query( + "DELETE FROM PeerTrustData WHERE peerID = ?;", + [peer_trust_data.info.id], + ) + # Insert PeerInfo first to ensure the peer exists self.__execute_query( """ @@ -143,6 +174,9 @@ def store_peer_trust_data(self, peer_trust_data: PeerTrustData) -> None: peer_trust_data.initial_reputation_provided_by_count, ), ) + peer_trust_data_id = self.__execute_query( + "SELECT last_insert_rowid();" + )[0][0] # Prepare to insert service history and link to PeerTrustData for sh in peer_trust_data.service_history: @@ -163,8 +197,14 @@ def store_peer_trust_data(self, peer_trust_data: PeerTrustData) -> None: self.__execute_query( """ INSERT INTO PeerTrustServiceHistory (peer_trust_data_id, service_history_id) - VALUES (last_insert_rowid(), last_insert_rowid()); - """ + VALUES (?, ?); + """, + ( + peer_trust_data_id, + self.__execute_query("SELECT last_insert_rowid();")[0][ + 0 + ], + ), ) # Prepare to insert recommendation history and link to PeerTrustData @@ -186,8 +226,14 @@ def store_peer_trust_data(self, peer_trust_data: PeerTrustData) -> None: self.__execute_query( """ INSERT INTO PeerTrustRecommendationHistory (peer_trust_data_id, recommendation_history_id) - VALUES (last_insert_rowid(), last_insert_rowid()); - """ + VALUES (?, ?); + """, + ( + peer_trust_data_id, + self.__execute_query("SELECT last_insert_rowid();")[0][ + 0 + ], + ), ) def get_peers_by_minimal_recommendation_trust( diff --git a/modules/fidesModule/persistence/threat_intelligence.py b/modules/fides/persistence/threat_intelligence.py similarity index 63% rename from modules/fidesModule/persistence/threat_intelligence.py rename to modules/fides/persistence/threat_intelligence.py index f8ce520e28..51ea99f25e 100644 --- a/modules/fidesModule/persistence/threat_intelligence.py +++ b/modules/fides/persistence/threat_intelligence.py @@ -1,7 +1,9 @@ from typing import Optional -from modules.fidesModule.model.aliases import Target -from modules.fidesModule.model.threat_intelligence import SlipsThreatIntelligence +from modules.fides.model.aliases import Target +from modules.fides.model.threat_intelligence import ( + SlipsThreatIntelligence, +) class ThreatIntelligenceDatabase: @@ -9,4 +11,4 @@ class ThreatIntelligenceDatabase: def get_for(self, target: Target) -> Optional[SlipsThreatIntelligence]: """Returns threat intelligence for given target or None if there are no data.""" - raise NotImplemented() + raise NotImplementedError() diff --git a/modules/fidesModule/persistence/threat_intelligence_db.py b/modules/fides/persistence/threat_intelligence_db.py similarity index 95% rename from modules/fidesModule/persistence/threat_intelligence_db.py rename to modules/fides/persistence/threat_intelligence_db.py index 6b450f46e2..1317c433db 100644 --- a/modules/fidesModule/persistence/threat_intelligence_db.py +++ b/modules/fides/persistence/threat_intelligence_db.py @@ -4,7 +4,7 @@ from ..model.aliases import Target from ..model.configuration import TrustModelConfiguration from ..model.threat_intelligence import SlipsThreatIntelligence -from modules.fidesModule.persistence.threat_intelligence import ( +from modules.fides.persistence.threat_intelligence import ( ThreatIntelligenceDatabase, ) diff --git a/modules/fidesModule/persistence/trust.py b/modules/fides/persistence/trust.py similarity index 57% rename from modules/fidesModule/persistence/trust.py rename to modules/fides/persistence/trust.py index d9efe379ec..26625b7e19 100644 --- a/modules/fidesModule/persistence/trust.py +++ b/modules/fides/persistence/trust.py @@ -1,10 +1,15 @@ from typing import List, Optional, Union -from modules.fidesModule.messaging.model import PeerInfo -from modules.fidesModule.model.aliases import PeerId, Target, OrganisationId -from modules.fidesModule.model.configuration import TrustModelConfiguration -from modules.fidesModule.model.peer_trust_data import PeerTrustData, TrustMatrix -from modules.fidesModule.model.threat_intelligence import SlipsThreatIntelligence +from modules.fides.messaging.model import PeerInfo +from modules.fides.model.aliases import PeerId, Target, OrganisationId +from modules.fides.model.configuration import TrustModelConfiguration +from modules.fides.model.peer_trust_data import ( + PeerTrustData, + TrustMatrix, +) +from modules.fides.model.threat_intelligence import ( + SlipsThreatIntelligence, +) class TrustDatabase: @@ -19,50 +24,62 @@ def get_model_configuration(self) -> TrustModelConfiguration: def store_connected_peers_list(self, current_peers: List[PeerInfo]): """Stores list of peers that are directly connected to the Slips.""" - raise NotImplemented() + raise NotImplementedError() def get_connected_peers(self) -> List[PeerInfo]: """Returns list of peers that are directly connected to the Slips.""" - raise NotImplemented() + raise NotImplementedError() def get_peers_info(self, peer_ids: List[PeerId]) -> List[PeerInfo]: """Returns list of peer infos for given ids.""" - raise NotImplemented() + raise NotImplementedError() - def get_peers_with_organisations(self, organisations: List[OrganisationId]) -> List[PeerInfo]: + def get_peers_with_organisations( + self, organisations: List[OrganisationId] + ) -> List[PeerInfo]: """Returns list of peers that have one of given organisations.""" - raise NotImplemented() + raise NotImplementedError() - def get_peers_with_geq_recommendation_trust(self, minimal_recommendation_trust: float) -> List[PeerInfo]: + def get_peers_with_geq_recommendation_trust( + self, minimal_recommendation_trust: float + ) -> List[PeerInfo]: """Returns peers that have >= recommendation_trust then the minimal.""" - raise NotImplemented() + raise NotImplementedError() - def get_peers_with_geq_service_trust(self, minimal_service_trust: float) -> List[PeerInfo]: + def get_peers_with_geq_service_trust( + self, minimal_service_trust: float + ) -> List[PeerInfo]: """Returns peers that have >= service_trust then the minimal.""" - raise NotImplemented() + raise NotImplementedError() def store_peer_trust_data(self, trust_data: PeerTrustData): """Stores trust data for given peer - overwrites any data if existed.""" - raise NotImplemented() + raise NotImplementedError() def store_peer_trust_matrix(self, trust_matrix: TrustMatrix): """Stores trust matrix.""" for peer in trust_matrix.values(): self.store_peer_trust_data(peer) - def get_peer_trust_data(self, peer: Union[PeerId, PeerInfo]) -> Optional[PeerTrustData]: + def get_peer_trust_data( + self, peer: Union[PeerId, PeerInfo] + ) -> Optional[PeerTrustData]: """Returns trust data for given peer ID, if no data are found, returns None.""" - raise NotImplemented() + raise NotImplementedError() - def get_peers_trust_data(self, peer_ids: List[Union[PeerId, PeerInfo]]) -> TrustMatrix: + def get_peers_trust_data( + self, peer_ids: List[Union[PeerId, PeerInfo]] + ) -> TrustMatrix: """Return trust data for each peer from peer_ids.""" data = [self.get_peer_trust_data(peer_id) for peer_id in peer_ids] return {peer.peer_id: peer for peer in data if peer} def cache_network_opinion(self, ti: SlipsThreatIntelligence): """Caches aggregated opinion on given target.""" - raise NotImplemented() + raise NotImplementedError() - def get_cached_network_opinion(self, target: Target) -> Optional[SlipsThreatIntelligence]: + def get_cached_network_opinion( + self, target: Target + ) -> Optional[SlipsThreatIntelligence]: """Returns cached network opinion. Checks cache time and returns None if data expired.""" - raise NotImplemented() + raise NotImplementedError() diff --git a/modules/fidesModule/persistence/trust_db.py b/modules/fides/persistence/trust_db.py similarity index 98% rename from modules/fidesModule/persistence/trust_db.py rename to modules/fides/persistence/trust_db.py index b11974fb73..189b9257fd 100644 --- a/modules/fidesModule/persistence/trust_db.py +++ b/modules/fides/persistence/trust_db.py @@ -5,7 +5,7 @@ from ..model.configuration import TrustModelConfiguration from ..model.peer_trust_data import PeerTrustData, TrustMatrix from ..model.threat_intelligence import SlipsThreatIntelligence -from modules.fidesModule.persistence.trust import TrustDatabase +from modules.fides.persistence.trust import TrustDatabase from .fides_sqlite_db import FidesSQLiteDB from slips_files.core.database.database_manager import DBManager diff --git a/modules/fides/protocols/__init__.py b/modules/fides/protocols/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/modules/fidesModule/protocols/alert.py b/modules/fides/protocols/alert.py similarity index 100% rename from modules/fidesModule/protocols/alert.py rename to modules/fides/protocols/alert.py diff --git a/modules/fidesModule/protocols/initial_trusl.py b/modules/fides/protocols/initial_trusl.py similarity index 69% rename from modules/fidesModule/protocols/initial_trusl.py rename to modules/fides/protocols/initial_trusl.py index 5e088ba003..1a497cd218 100644 --- a/modules/fidesModule/protocols/initial_trusl.py +++ b/modules/fides/protocols/initial_trusl.py @@ -11,16 +11,19 @@ class InitialTrustProtocol: - def __init__(self, - trust_db: SlipsTrustDatabase, - configuration: TrustModelConfiguration, - recommendation_protocol: RecommendationProtocol - ): + def __init__( + self, + trust_db: SlipsTrustDatabase, + configuration: TrustModelConfiguration, + recommendation_protocol: RecommendationProtocol, + ): self.__trust_db = trust_db self.__configuration = configuration self.__recommendation_protocol = recommendation_protocol - def determine_and_store_initial_trust(self, peer: PeerInfo, get_recommendations: bool = False) -> PeerTrustData: + def determine_and_store_initial_trust( + self, peer: PeerInfo, get_recommendations: bool = False + ) -> PeerTrustData: """Determines initial trust and stores that value in database. Returns trust data before the recommendation protocol is executed. @@ -29,7 +32,9 @@ def determine_and_store_initial_trust(self, peer: PeerInfo, get_recommendations: existing_trust = self.__trust_db.get_peer_trust_data(peer.id) if existing_trust is not None: - logger.debug(f"There's an existing trust for peer {peer.id}: ST: {existing_trust.service_trust}") + logger.debug( + f"There's an existing trust for peer {peer.id}: ST: {existing_trust.service_trust}" + ) return existing_trust # now we know that this is a new peer @@ -40,29 +45,43 @@ def determine_and_store_initial_trust(self, peer: PeerInfo, get_recommendations: trust.initial_reputation_provided_by_count = 1 # check if this is pre-trusted peer - pre_trusted_peer = [p for p in self.__configuration.trusted_peers if trust.peer_id == p.id] + pre_trusted_peer = [ + p + for p in self.__configuration.trusted_peers + if trust.peer_id == p.id + ] if len(pre_trusted_peer) == 1: configured_peer = pre_trusted_peer[0] self.__inherit_trust(trust, configured_peer) trust.initial_reputation_provided_by_count += 1 # add values that are inherited from the organisations - peers_orgs = [org for org in self.__configuration.trusted_organisations if org.id in peer.organisations] + peers_orgs = [ + org + for org in self.__configuration.trusted_organisations + if org.id in peer.organisations + ] if peers_orgs: - logger.debug(f"Peer {peer.id} has known organisations.", peers_orgs) + logger.debug( + f"Peer {peer.id} has known organisations.", peers_orgs + ) trust.initial_reputation_provided_by_count += len(peers_orgs) # select organisation that has the highest trust leading_organisation = max(peers_orgs, key=lambda org: org.trust) - logger.debug(f"Main organisation selected, computing trust", leading_organisation) + logger.debug( + "Main organisation selected, computing trust", + leading_organisation, + ) # now set all other stuff from the organisation self.__inherit_trust(trust, leading_organisation) # process interaction and assign all others values - trust = process_service_interaction(configuration=self.__configuration, - peer=trust, - satisfaction=SatisfactionLevels.Ok, - weight=Weight.FIRST_ENCOUNTER - ) + trust = process_service_interaction( + configuration=self.__configuration, + peer=trust, + satisfaction=SatisfactionLevels.Ok, + weight=Weight.FIRST_ENCOUNTER, + ) logger.debug(f"New trust for peer: {trust.peer_id}", trust) # determine if it is necessary to get recommendations from the network @@ -76,7 +95,9 @@ def determine_and_store_initial_trust(self, peer: PeerInfo, get_recommendations: return trust @staticmethod - def __inherit_trust(trust: PeerTrustData, parent: TrustedEntity) -> PeerTrustData: + def __inherit_trust( + trust: PeerTrustData, parent: TrustedEntity + ) -> PeerTrustData: # TODO [?] check which believes / trust metrics can we set as well trust.reputation = max(trust.reputation, parent.trust) trust.recommendation_trust = trust.reputation @@ -88,6 +109,8 @@ def __inherit_trust(trust: PeerTrustData, parent: TrustedEntity) -> PeerTrustDat # and we will be satisfied with all interactions equally trust.integrity_belief = 1 trust.competence_belief = 1 - logger.debug(f"Enforced trust, leaving service trust to: {trust.service_trust}.") + logger.debug( + f"Enforced trust, leaving service trust to: {trust.service_trust}." + ) return trust diff --git a/modules/fides/protocols/opinion.py b/modules/fides/protocols/opinion.py new file mode 100644 index 0000000000..81d8d81deb --- /dev/null +++ b/modules/fides/protocols/opinion.py @@ -0,0 +1,61 @@ +from typing import Dict + +from ..evaluation.ti_aggregation import TIAggregation, PeerReport +from ..messaging.model import PeerIntelligenceResponse +from ..model.alert import Alert +from ..model.aliases import PeerId, Target +from ..model.configuration import TrustModelConfiguration +from ..model.peer_trust_data import PeerTrustData, TrustMatrix +from ..model.threat_intelligence import SlipsThreatIntelligence +from ..persistence.threat_intelligence_db import ( + SlipsThreatIntelligenceDatabase, +) + + +class OpinionAggregator: + """ + Class responsible for evaluation of the intelligence received from the network. + """ + + def __init__( + self, + configuration: TrustModelConfiguration, + ti_db: SlipsThreatIntelligenceDatabase, + ti_aggregation: TIAggregation, + ): + self.__configuration = configuration + self.__ti_db = ti_db + self.__ti_aggregation = ti_aggregation + + def evaluate_alert( + self, peer_trust: PeerTrustData, alert: Alert + ) -> SlipsThreatIntelligence: + """Evaluates given data about alert and produces aggregated intelligence for Slips.""" + + alert_trust = max( + self.__configuration.alert_trust_from_unknown, + peer_trust.service_trust, + ) + score = alert.score + confidence = alert.confidence * alert_trust + return SlipsThreatIntelligence( + score=score, confidence=confidence, target=alert.target + ) + + def evaluate_intelligence_response( + self, + target: Target, + data: Dict[PeerId, PeerIntelligenceResponse], + trust_matrix: TrustMatrix, + ) -> SlipsThreatIntelligence: + """Evaluates given threat intelligence report from the network.""" + reports = [ + PeerReport( + report_ti=ti.intelligence, reporter_trust=trust_matrix[peer_id] + ) + for peer_id, ti in data.items() + ] + ti = self.__ti_aggregation.assemble_peer_opinion(data=reports) + return SlipsThreatIntelligence( + score=ti.score, confidence=ti.confidence, target=target + ) diff --git a/modules/fidesModule/protocols/peer_list.py b/modules/fides/protocols/peer_list.py similarity index 65% rename from modules/fidesModule/protocols/peer_list.py rename to modules/fides/protocols/peer_list.py index 6e6fcc554e..2f0f480477 100644 --- a/modules/fidesModule/protocols/peer_list.py +++ b/modules/fides/protocols/peer_list.py @@ -10,12 +10,13 @@ class PeerListUpdateProtocol: """Protocol handling situations when peer list was updated.""" - def __init__(self, - trust_db: SlipsTrustDatabase, - bridge: NetworkBridge, - recommendation_protocol: RecommendationProtocol, - trust_protocol: InitialTrustProtocol - ): + def __init__( + self, + trust_db: SlipsTrustDatabase, + bridge: NetworkBridge, + recommendation_protocol: RecommendationProtocol, + trust_protocol: InitialTrustProtocol, + ): self.__trust_db = trust_db self.__bridge = bridge self.__recommendation_protocol = recommendation_protocol @@ -26,8 +27,14 @@ def handle_peer_list_updated(self, peers: List[PeerInfo]): # first store them in the database self.__trust_db.store_connected_peers_list(peers) # and now find their trust metrics to send it to the network module - trust_data = self.__trust_db.get_peers_trust_data([p.id for p in peers]) - known_peers = {peer_id for peer_id, trust in trust_data.items() if trust is not None} + trust_data = self.__trust_db.get_peers_trust_data( + [p.id for p in peers] + ) + known_peers = { + peer_id + for peer_id, trust in trust_data.items() + if trust is not None + } # if we don't have data for all peers that means that there are some new peers # we need to establish initial trust for them if len(known_peers) != len(peers): @@ -35,11 +42,19 @@ def handle_peer_list_updated(self, peers: List[PeerInfo]): for peer in [p for p in peers if p.id not in known_peers]: # this stores trust in database as well, do not get recommendations because at this point # we don't have correct peer list in database - peer_trust = self.__trust_protocol.determine_and_store_initial_trust(peer, get_recommendations=False) + peer_trust = ( + self.__trust_protocol.determine_and_store_initial_trust( + peer, get_recommendations=False + ) + ) new_trusts.append(peer_trust) # get recommendations for this peer - self.__recommendation_protocol.get_recommendation_for(peer, connected_peers=list(known_peers)) + self.__recommendation_protocol.get_recommendation_for( + peer, connected_peers=list(known_peers) + ) # send only updated trusts to the network layer - self.__bridge.send_peers_reliability({p.peer_id: p.service_trust for p in new_trusts}) + self.__bridge.send_peers_reliability( + {p.peer_id: p.service_trust for p in new_trusts} + ) # now set update peer list in database self.__trust_db.store_connected_peers_list(peers) diff --git a/modules/fidesModule/protocols/protocol.py b/modules/fides/protocols/protocol.py similarity index 52% rename from modules/fidesModule/protocols/protocol.py rename to modules/fides/protocols/protocol.py index b9ec4b6143..b484eb9c75 100644 --- a/modules/fidesModule/protocols/protocol.py +++ b/modules/fides/protocols/protocol.py @@ -6,37 +6,44 @@ from ..model.aliases import PeerId from ..model.configuration import TrustModelConfiguration from ..model.peer_trust_data import PeerTrustData, TrustMatrix -from modules.fidesModule.persistence.trust import TrustDatabase +from modules.fides.persistence.trust import TrustDatabase class Protocol: - def __init__(self, - configuration: TrustModelConfiguration, - trust_db: TrustDatabase, - bridge: NetworkBridge): + def __init__( + self, + configuration: TrustModelConfiguration, + trust_db: TrustDatabase, + bridge: NetworkBridge, + ): self._configuration = configuration self._trust_db = trust_db self._bridge = bridge - def _evaluate_interaction(self, - peer: PeerTrustData, - satisfaction: Satisfaction, - weight: Weight - ) -> PeerTrustData: + def _evaluate_interaction( + self, peer: PeerTrustData, satisfaction: Satisfaction, weight: Weight + ) -> PeerTrustData: """Callback to evaluate and save new trust data for given peer.""" - return self._evaluate_interactions({peer.peer_id: (peer, satisfaction, weight)})[peer.peer_id] + return self._evaluate_interactions( + {peer.peer_id: (peer, satisfaction, weight)} + )[peer.peer_id] - def _evaluate_interactions(self, - data: Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]) -> TrustMatrix: + def _evaluate_interactions( + self, data: Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]] + ) -> TrustMatrix: """Callback to evaluate and save new trust data for given peer matrix.""" trust_matrix: TrustMatrix = {} # first process all interactions for _, (peer_trust, satisfaction, weight) in data.items(): - updated_trust = process_service_interaction(self._configuration, peer_trust, satisfaction, weight) + updated_trust = process_service_interaction( + self._configuration, peer_trust, satisfaction, weight + ) trust_matrix[updated_trust.peer_id] = updated_trust # then store matrix self._trust_db.store_peer_trust_matrix(trust_matrix) # and dispatch this update to the network layer - self._bridge.send_peers_reliability({p.peer_id: p.service_trust for p in trust_matrix.values()}) + self._bridge.send_peers_reliability( + {p.peer_id: p.service_trust for p in trust_matrix.values()} + ) return trust_matrix diff --git a/modules/fidesModule/protocols/recommendation.py b/modules/fides/protocols/recommendation.py similarity index 53% rename from modules/fidesModule/protocols/recommendation.py rename to modules/fides/protocols/recommendation.py index a9b732fdc1..3672183925 100644 --- a/modules/fidesModule/protocols/recommendation.py +++ b/modules/fides/protocols/recommendation.py @@ -19,29 +19,50 @@ class RecommendationProtocol(Protocol): """Protocol that is responsible for getting and updating recommendation data.""" - def __init__(self, configuration: TrustModelConfiguration, trust_db: SlipsTrustDatabase, bridge: NetworkBridge): + def __init__( + self, + configuration: TrustModelConfiguration, + trust_db: SlipsTrustDatabase, + bridge: NetworkBridge, + ): super().__init__(configuration, trust_db, bridge) self.__rec_conf = configuration.recommendations self.__trust_db = trust_db self.__bridge = bridge - def get_recommendation_for(self, peer: PeerInfo, connected_peers: Optional[List[PeerId]] = None): + def get_recommendation_for( + self, peer: PeerInfo, connected_peers: Optional[List[PeerId]] = None + ): """Dispatches recommendation request from the network. connected_peers - new peer list if the one from database is not accurate """ if not self.__rec_conf.enabled: - logger.debug(f"Recommendation protocol is disabled. NOT getting recommendations for Peer {peer.id}.") + logger.debug( + f"Recommendation protocol is disabled. NOT getting recommendations for Peer {peer.id}." + ) return - connected_peers = connected_peers if connected_peers is not None else self.__trust_db.get_connected_peers() - recipients = self.__get_recommendation_request_recipients(peer, connected_peers) + connected_peers = ( + connected_peers + if connected_peers is not None + else self.__trust_db.get_connected_peers() + ) + recipients = self.__get_recommendation_request_recipients( + peer, connected_peers + ) if recipients: - self.__bridge.send_recommendation_request(recipients=recipients, peer=peer.id) + self.__bridge.send_recommendation_request( + recipients=recipients, peer=peer.id + ) else: - logger.debug(f"No peers are trusted enough to ask them for recommendation!") + logger.debug( + "No peers are trusted enough to ask them for recommendation!" + ) - def handle_recommendation_request(self, request_id: str, sender: PeerInfo, subject: PeerId): + def handle_recommendation_request( + self, request_id: str, sender: PeerInfo, subject: PeerId + ): """Handle request for recommendation on given subject.""" sender_trust = self.__trust_db.get_peer_trust_data(sender) # TODO: [+] implement data filtering based on the sender @@ -53,7 +74,7 @@ def handle_recommendation_request(self, request_id: str, sender: PeerInfo, subje integrity_belief=trust.integrity_belief, service_history_size=trust.service_history_size, recommendation=trust.reputation, - initial_reputation_provided_by_count=trust.initial_reputation_provided_by_count + initial_reputation_provided_by_count=trust.initial_reputation_provided_by_count, ) else: recommendation = Recommendation( @@ -61,106 +82,160 @@ def handle_recommendation_request(self, request_id: str, sender: PeerInfo, subje integrity_belief=0, service_history_size=0, recommendation=0, - initial_reputation_provided_by_count=0 + initial_reputation_provided_by_count=0, ) - self.__bridge.send_recommendation_response(request_id, sender.id, subject, recommendation) + self.__bridge.send_recommendation_response( + request_id, sender.id, subject, recommendation + ) # it is possible that we saw sender for the first time # TODO: [+] initialise peer if we saw it for the first time if sender_trust: - self._evaluate_interaction(sender_trust, SatisfactionLevels.Ok, Weight.INTELLIGENCE_REQUEST) + self._evaluate_interaction( + sender_trust, + SatisfactionLevels.Ok, + Weight.INTELLIGENCE_REQUEST, + ) - def handle_recommendation_response(self, responses: List[PeerRecommendationResponse]): + def handle_recommendation_response( + self, responses: List[PeerRecommendationResponse] + ): """Handles response from peers with recommendations. Updates all necessary values in db.""" if len(responses) == 0: return # TODO: [+] handle cases with multiple subjects - assert all(responses[0].subject == r.subject for r in responses), \ - "Responses are not for the same subject!" + assert all( + responses[0].subject == r.subject for r in responses + ), "Responses are not for the same subject!" subject = self.__trust_db.get_peer_trust_data(responses[0].subject) if subject is None: - logger.warn(f'Received recommendation for subject {responses[0].subject} that does not exist!') + logger.warn( + f"Received recommendation for subject {responses[0].subject} that does not exist!" + ) return recommendations = {r.sender.id: r.recommendation for r in responses} - trust_matrix = self.__trust_db.get_peers_trust_data(list(recommendations.keys())) + trust_matrix = self.__trust_db.get_peers_trust_data( + list(recommendations.keys()) + ) # check that the data are consistent - assert len(trust_matrix) == len(responses) == len(recommendations), \ - f'Data are not consistent: TM: {len(trust_matrix)}, RES: {len(responses)}, REC: {len(recommendations)}!' + assert ( + len(trust_matrix) == len(responses) == len(recommendations) + ), f"Data are not consistent: TM: {len(trust_matrix)}, RES: {len(responses)}, REC: {len(recommendations)}!" # update all recommendations updated_matrix = process_new_recommendations( configuration=self._configuration, subject=subject, matrix=trust_matrix, - recommendations=recommendations + recommendations=recommendations, ) # now store updated matrix self.__trust_db.store_peer_trust_matrix(updated_matrix) # and dispatch event - self.__bridge.send_peers_reliability({p.peer_id: p.service_trust for p in updated_matrix.values()}) + self.__bridge.send_peers_reliability( + {p.peer_id: p.service_trust for p in updated_matrix.values()} + ) # TODO: [+] optionally employ same thing as when receiving TI - interaction_matrix = {p.peer_id: (p, SatisfactionLevels.Ok, Weight.RECOMMENDATION_RESPONSE) - for p in trust_matrix.values()} + interaction_matrix = { + p.peer_id: ( + p, + SatisfactionLevels.Ok, + Weight.RECOMMENDATION_RESPONSE, + ) + for p in trust_matrix.values() + } self._evaluate_interactions(interaction_matrix) @staticmethod def __is_zero_recommendation(recommendation: Recommendation) -> bool: - return recommendation.competence_belief == 0 and \ - recommendation.integrity_belief == 0 and \ - recommendation.service_history_size == 0 and \ - recommendation.recommendation == 0 and \ - recommendation.initial_reputation_provided_by_count == 0 - - def __get_recommendation_request_recipients(self, - subject: PeerInfo, - connected_peers: List[PeerInfo]) -> List[PeerId]: + return ( + recommendation.competence_belief == 0 + and recommendation.integrity_belief == 0 + and recommendation.service_history_size == 0 + and recommendation.recommendation == 0 + and recommendation.initial_reputation_provided_by_count == 0 + ) + + def __get_recommendation_request_recipients( + self, subject: PeerInfo, connected_peers: List[PeerInfo] + ) -> List[PeerId]: recommenders: List[PeerInfo] = [] - require_trusted_peer_count = self.__rec_conf.required_trusted_peers_count + require_trusted_peer_count = ( + self.__rec_conf.required_trusted_peers_count + ) trusted_peer_threshold = self.__rec_conf.trusted_peer_threshold if self.__rec_conf.only_connected: recommenders = connected_peers if self.__rec_conf.only_preconfigured: - preconfigured_peers = set(p.id for p in self._configuration.trusted_peers) - preconfigured_organisations = set(p.id for p in self._configuration.trusted_organisations) + preconfigured_peers = set( + p.id for p in self._configuration.trusted_peers + ) + preconfigured_organisations = set( + p.id for p in self._configuration.trusted_organisations + ) if len(recommenders) > 0: # if there are already some recommenders it means that only_connected filter is enabled # in that case we need to filter those peers and see if they either are on preconfigured # list or if they have any organisation - recommenders = [p for p in recommenders - if p.id in preconfigured_peers - or preconfigured_organisations.intersection(p.organisations)] + recommenders = [ + p + for p in recommenders + if p.id in preconfigured_peers + or preconfigured_organisations.intersection( + p.organisations + ) + ] else: # if there are no recommenders, only_preconfigured is disabled, so we select all preconfigured # peers and all peers from database that have the organisation - recommenders = self.__trust_db.get_peers_info(list(preconfigured_peers)) \ - + self.__trust_db.get_peers_with_organisations(list(preconfigured_organisations)) + recommenders = self.__trust_db.get_peers_info( + list(preconfigured_peers) + ) + self.__trust_db.get_peers_with_organisations( + list(preconfigured_organisations) + ) # if we have only_preconfigured, we do not need to care about minimal trust because we're safe enough require_trusted_peer_count = -math.inf elif not self.__rec_conf.only_connected: # in this case there's no restriction, and we can freely select any peers # select peers that hev at least trusted_peer_threshold recommendation trust - recommenders = self.__trust_db.get_peers_with_geq_recommendation_trust(trusted_peer_threshold) + recommenders = ( + self.__trust_db.get_peers_with_geq_recommendation_trust( + trusted_peer_threshold + ) + ) # if there's not enough peers like that, select some more with this service trust if len(recommenders) <= self.__rec_conf.peers_max_count: # TODO: [+] maybe add higher trusted_peer_threshold for this one - recommenders += self.__trust_db.get_peers_with_geq_service_trust(trusted_peer_threshold) + recommenders += ( + self.__trust_db.get_peers_with_geq_service_trust( + trusted_peer_threshold + ) + ) # now we need to get all trust data and sort them by recommendation trust - candidates = list(self.__trust_db.get_peers_trust_data(recommenders).values()) + candidates = list( + self.__trust_db.get_peers_trust_data(recommenders).values() + ) candidates = [c for c in candidates if c.peer_id != subject.id] # check if we can proceed - if len(candidates) == 0 or len(candidates) < require_trusted_peer_count: + if ( + len(candidates) == 0 + or len(candidates) < require_trusted_peer_count + ): logger.debug( - f"Not enough trusted peers! Candidates: {len(candidates)}, requirement: {require_trusted_peer_count}.") + f"Not enough trusted peers! Candidates: {len(candidates)}, requirement: {require_trusted_peer_count}." + ) return [] # now sort them candidates.sort(key=lambda c: c.service_trust, reverse=True) # and take only top __rec_conf.peers_max_count peers to ask for recommendations - return [p.peer_id for p in candidates][:self.__rec_conf.peers_max_count] + return [p.peer_id for p in candidates][ + : self.__rec_conf.peers_max_count + ] diff --git a/modules/fidesModule/protocols/threat_intelligence.py b/modules/fides/protocols/threat_intelligence.py similarity index 56% rename from modules/fidesModule/protocols/threat_intelligence.py rename to modules/fides/protocols/threat_intelligence.py index 6ae9234d91..cd1e2a2bfb 100644 --- a/modules/fidesModule/protocols/threat_intelligence.py +++ b/modules/fides/protocols/threat_intelligence.py @@ -8,8 +8,13 @@ from ..model.configuration import TrustModelConfiguration from ..model.peer import PeerInfo from ..model.peer_trust_data import PeerTrustData -from ..model.threat_intelligence import ThreatIntelligence, SlipsThreatIntelligence -from ..persistence.threat_intelligence_db import SlipsThreatIntelligenceDatabase +from ..model.threat_intelligence import ( + ThreatIntelligence, + SlipsThreatIntelligence, +) +from ..persistence.threat_intelligence_db import ( + SlipsThreatIntelligenceDatabase, +) from ..persistence.trust_db import SlipsTrustDatabase from ..protocols.initial_trusl import InitialTrustProtocol from ..protocols.opinion import OpinionAggregator @@ -22,16 +27,17 @@ class ThreatIntelligenceProtocol(Protocol): """Class handling threat intelligence requests and responses.""" - def __init__(self, - trust_db: SlipsTrustDatabase, - ti_db: SlipsThreatIntelligenceDatabase, - bridge: NetworkBridge, - configuration: TrustModelConfiguration, - aggregator: OpinionAggregator, - trust_protocol: InitialTrustProtocol, - ti_evaluation_strategy: TIEvaluation, - network_opinion_callback: Callable[[SlipsThreatIntelligence], None] - ): + def __init__( + self, + trust_db: SlipsTrustDatabase, + ti_db: SlipsThreatIntelligenceDatabase, + bridge: NetworkBridge, + configuration: TrustModelConfiguration, + aggregator: OpinionAggregator, + trust_protocol: InitialTrustProtocol, + ti_evaluation_strategy: TIEvaluation, + network_opinion_callback: Callable[[SlipsThreatIntelligence], None], + ): super().__init__(configuration, trust_db, bridge) self.__ti_db = ti_db self.__aggregator = aggregator @@ -43,18 +49,22 @@ def request_data(self, target: Target): """Requests network opinion on given target.""" cached = self._trust_db.get_cached_network_opinion(target) if cached: - logger.debug(f'TI for target {target} found in cache.') + logger.debug(f"TI for target {target} found in cache.") return self.__network_opinion_callback(cached) else: - logger.debug(f'Requesting data for target {target} from network.') + logger.debug(f"Requesting data for target {target} from network.") self._bridge.send_intelligence_request(target) - def handle_intelligence_request(self, request_id: str, sender: PeerInfo, target: Target): + def handle_intelligence_request( + self, request_id: str, sender: PeerInfo, target: Target + ): """Handles intelligence request.""" peer_trust = self._trust_db.get_peer_trust_data(sender.id) if not peer_trust: - logger.debug(f'We don\'t have any trust data for peer {sender.id}!') - peer_trust = self.__trust_protocol.determine_and_store_initial_trust(sender) + logger.debug(f"We don't have any trust data for peer {sender.id}!") + peer_trust = ( + self.__trust_protocol.determine_and_store_initial_trust(sender) + ) ti = self.__filter_ti(self.__ti_db.get_for(target), peer_trust) if ti is None: @@ -63,50 +73,64 @@ def handle_intelligence_request(self, request_id: str, sender: PeerInfo, target: # and respond with data we have self._bridge.send_intelligence_response(request_id, target, ti) - self._evaluate_interaction(peer_trust, - SatisfactionLevels.Ok, - Weight.INTELLIGENCE_REQUEST) + self._evaluate_interaction( + peer_trust, SatisfactionLevels.Ok, Weight.INTELLIGENCE_REQUEST + ) - def handle_intelligence_response(self, responses: List[PeerIntelligenceResponse]): + def handle_intelligence_response( + self, responses: List[PeerIntelligenceResponse] + ): """Handles intelligence responses.""" - trust_matrix = self._trust_db.get_peers_trust_data([r.sender.id for r in responses]) - assert len(trust_matrix) == len(responses), 'We need to have trust data for all peers that sent the response.' + trust_matrix = self._trust_db.get_peers_trust_data( + [r.sender.id for r in responses] + ) + assert len(trust_matrix) == len( + responses + ), "We need to have trust data for all peers that sent the response." target = {r.target for r in responses} - assert len(target) == 1, 'Responses should be for a single target.' + assert len(target) == 1, "Responses should be for a single target." target = target.pop() # now everything is checked, so we aggregate it and get the threat intelligence r = {r.sender.id: r for r in responses} - ti = self.__aggregator.evaluate_intelligence_response(target, r, trust_matrix) + ti = self.__aggregator.evaluate_intelligence_response( + target, r, trust_matrix + ) # cache data for further retrieval self._trust_db.cache_network_opinion(ti) - #test = self._trust_db.get_cached_network_opinion(target) + # test = self._trust_db.get_cached_network_opinion(target) interaction_matrix = self.__ti_evaluation_strategy.evaluate( aggregated_ti=ti, responses=r, trust_matrix=trust_matrix, - local_ti=self.__ti_db.get_for(target) + local_ti=self.__ti_db.get_for(target), ) self._evaluate_interactions(interaction_matrix) return self.__network_opinion_callback(ti) - def __filter_ti(self, - ti: Optional[SlipsThreatIntelligence], - peer_trust: PeerTrustData) -> Optional[SlipsThreatIntelligence]: + def __filter_ti( + self, ti: Optional[SlipsThreatIntelligence], peer_trust: PeerTrustData + ) -> Optional[SlipsThreatIntelligence]: if ti is None: return None - peers_allowed_levels = [p.confidentiality_level - for p in self._configuration.trusted_organisations if - p.id in peer_trust.organisations] + peers_allowed_levels = [ + p.confidentiality_level + for p in self._configuration.trusted_organisations + if p.id in peer_trust.organisations + ] peers_allowed_levels.append(peer_trust.service_trust) # select maximum allowed level allowed_level = max(peers_allowed_levels) # set correct confidentiality - ti.confidentiality = ti.confidentiality if ti.confidentiality else self._configuration.data_default_level + ti.confidentiality = ( + ti.confidentiality + if ti.confidentiality + else self._configuration.data_default_level + ) # check if data confidentiality is lower than allowed level for the peer return ti if ti.confidentiality <= allowed_level else None diff --git a/modules/fidesModule/utils/__init__.py b/modules/fides/utils/__init__.py similarity index 100% rename from modules/fidesModule/utils/__init__.py rename to modules/fides/utils/__init__.py diff --git a/modules/fidesModule/utils/logger.py b/modules/fides/utils/logger.py similarity index 66% rename from modules/fidesModule/utils/logger.py rename to modules/fides/utils/logger.py index 9fbb14e836..5997b3b679 100644 --- a/modules/fidesModule/utils/logger.py +++ b/modules/fides/utils/logger.py @@ -1,18 +1,21 @@ import json import threading from dataclasses import is_dataclass, asdict -from tabnanny import verbose from typing import Optional, List, Callable -LoggerPrintCallbacks: List[Callable[[str, Optional[str], Optional[int], Optional[int], Optional[bool]], None]] = [ +LoggerPrintCallbacks: List[ + Callable[ + [str, Optional[str], Optional[int], Optional[int], Optional[bool]], + None, + ] +] = [ lambda msg, level=None, verbose=1, debug=0, log_to_logfiles_only=False: print( - f'{level}: {msg}' if level is not None else f'UNSPECIFIED_LEVEL: {msg}' + f"{level}: {msg}" if level is not None else f"UNSPECIFIED_LEVEL: {msg}" ) ] -"""Set this to custom callback that should be executed when there's new log message. -First parameter is level ('DEBUG', 'INFO', 'WARN', 'ERROR'), second is message to be logged. -""" +# Set this to custom callback that should be executed when there's new log message. +# First parameter is level ('DEBUG', 'INFO', 'WARN', 'ERROR'), second is message to be logged. class Logger: @@ -27,12 +30,7 @@ def __init__(self, name: Optional[str] = None): if name is None: name = self.__try_to_guess_name() self.__name = name - self.log_levels = log_levels = { - 'INFO': 1, - 'WARN': 2, - 'ERROR': 3 - } - + self.log_levels = {"INFO": 1, "WARN": 2, "ERROR": 3} # this whole method is a hack # noinspection PyBroadException @@ -41,29 +39,31 @@ def __try_to_guess_name() -> str: # noinspection PyPep8 try: import sys + # noinspection PyUnresolvedReferences,PyProtectedMember name = sys._getframe().f_back.f_code.co_name if name is None: import inspect + inspect.currentframe() frame = inspect.currentframe() frame = inspect.getouterframes(frame, 2) name = frame[1][3] - except: + except Exception: name = "logger" return name def debug(self, message: str, params=None): - return self.__print('DEBUG', message) + return self.__print("DEBUG", message) def info(self, message: str, params=None): - return self.__print('INFO', message) + return self.__print("INFO", message) def warn(self, message: str, params=None): - return self.__print('WARN', message) + return self.__print("WARN", message) def error(self, message: str, params=None): - return self.__print('ERROR', message) + return self.__print("ERROR", message) def __format(self, message: str, params=None): thread = threading.get_ident() @@ -76,8 +76,11 @@ def __format(self, message: str, params=None): def __print(self, level: str, message: str, params=None): formatted_message = self.__format(message, params) for print_callback in LoggerPrintCallbacks: - if level == 'DEBUG': - print_callback(formatted_message, verbose=0) # automatically verbose = 1 - print, debug = 0 - do not print + if level == "DEBUG": + print_callback( + formatted_message, verbose=0 + ) # automatically verbose = 1 - print, debug = 0 - do not print else: - print_callback(formatted_message, verbose=self.log_levels[level]) - + print_callback( + formatted_message, verbose=self.log_levels[level] + ) diff --git a/modules/fides/utils/time.py b/modules/fides/utils/time.py new file mode 100644 index 0000000000..ad51861f38 --- /dev/null +++ b/modules/fides/utils/time.py @@ -0,0 +1,11 @@ +import time + +Time = float +# Type for time used across the whole module. +# Represents the current time in seconds since the Epoch. Can have frictions of seconds. +# We have it as alias so we can easily change that in the future. + + +def now() -> Time: + # Returns current Time. + return time.time() diff --git a/modules/fidesModule/evaluation/README.md b/modules/fidesModule/evaluation/README.md deleted file mode 100644 index ee22d10295..0000000000 --- a/modules/fidesModule/evaluation/README.md +++ /dev/null @@ -1 +0,0 @@ -All algorithms in this package are based on SORT - see paper. \ No newline at end of file diff --git a/modules/fidesModule/evaluation/ti_aggregation.py b/modules/fidesModule/evaluation/ti_aggregation.py deleted file mode 100644 index 14aae9be73..0000000000 --- a/modules/fidesModule/evaluation/ti_aggregation.py +++ /dev/null @@ -1,86 +0,0 @@ -from dataclasses import dataclass -from typing import List - -import numpy as np - -from ..model.peer_trust_data import PeerTrustData -from ..model.threat_intelligence import ThreatIntelligence -from ..utils import bound - - -@dataclass -class PeerReport: - report_ti: ThreatIntelligence - """Threat intelligence report.""" - - reporter_trust: PeerTrustData - """How much does Slips trust the reporter.""" - - -class TIAggregation: - - def assemble_peer_opinion(self, data: List[PeerReport]) -> ThreatIntelligence: - """ - Assemble reports given by all peers and compute the overall network opinion. - - :param data: a list of peers and their reports, in the format given by TrustDB.get_opinion_on_ip() - :return: final score and final confidence - """ - raise NotImplemented('') - - -class AverageConfidenceTIAggregation(TIAggregation): - - def assemble_peer_opinion(self, data: List[PeerReport]) -> ThreatIntelligence: - """ - Uses average when computing final confidence. - """ - reports_ti = [d.report_ti for d in data] - reporters_trust = [d.reporter_trust.service_trust for d in data] - - normalize_net_trust_sum = sum(reporters_trust) - weighted_reporters = [trust / normalize_net_trust_sum for trust in reporters_trust] \ - if normalize_net_trust_sum > 0 else [0] * len(reporters_trust) - - combined_score = sum(r.score * w for r, w, in zip(reports_ti, weighted_reporters)) - combined_confidence = sum(r.confidence * w for r, w, in zip(reports_ti, reporters_trust)) / len(reporters_trust) - - return ThreatIntelligence(score=combined_score, confidence=combined_confidence) - - -class WeightedAverageConfidenceTIAggregation(TIAggregation): - - def assemble_peer_opinion(self, data: List[PeerReport]) -> ThreatIntelligence: - reports_ti = [d.report_ti for d in data] - reporters_trust = [d.reporter_trust.service_trust for d in data] - - normalize_net_trust_sum = sum(reporters_trust) - weighted_reporters = [trust / normalize_net_trust_sum for trust in reporters_trust] - - combined_score = sum(r.score * w for r, w, in zip(reports_ti, weighted_reporters)) - combined_confidence = sum(r.confidence * w for r, w, in zip(reports_ti, weighted_reporters)) - - return ThreatIntelligence(score=combined_score, confidence=combined_confidence) - - -class StdevFromScoreTIAggregation(TIAggregation): - - def assemble_peer_opinion(self, data: List[PeerReport]) -> ThreatIntelligence: - reports_ti = [d.report_ti for d in data] - reporters_trust = [d.reporter_trust.service_trust for d in data] - - normalize_net_trust_sum = sum(reporters_trust) - weighted_reporters = [trust / normalize_net_trust_sum for trust in reporters_trust] - - merged_score = [r.score * r.confidence * w for r, w, in zip(reports_ti, weighted_reporters)] - combined_score = sum(merged_score) - combined_confidence = bound(1 - np.std(merged_score), 0, 1) - - return ThreatIntelligence(score=combined_score, confidence=combined_confidence) - - -TIAggregationStrategy = { - 'average': AverageConfidenceTIAggregation, - 'weightedAverage': WeightedAverageConfidenceTIAggregation, - 'stdevFromScore': StdevFromScoreTIAggregation, -} diff --git a/modules/fidesModule/evaluation/ti_evaluation.py b/modules/fidesModule/evaluation/ti_evaluation.py deleted file mode 100644 index a2bf0f00d2..0000000000 --- a/modules/fidesModule/evaluation/ti_evaluation.py +++ /dev/null @@ -1,255 +0,0 @@ -from collections import defaultdict -from typing import Dict, Tuple, Optional - -from ..evaluation.service.interaction import Satisfaction, Weight, SatisfactionLevels -from ..messaging.model import PeerIntelligenceResponse -from ..model.aliases import PeerId, Target -from ..model.peer_trust_data import PeerTrustData, TrustMatrix -from ..model.threat_intelligence import SlipsThreatIntelligence -from ..utils.logger import Logger - -logger = Logger(__name__) - - -class TIEvaluation: - def evaluate(self, - aggregated_ti: SlipsThreatIntelligence, - responses: Dict[PeerId, PeerIntelligenceResponse], - trust_matrix: TrustMatrix, - **kwargs, - ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: - """Evaluate interaction with all peers that gave intelligence responses.""" - raise NotImplemented('Use implementation rather then interface!') - - @staticmethod - def _weight() -> Weight: - return Weight.INTELLIGENCE_DATA_REPORT - - @staticmethod - def _assert_keys(responses: Dict[PeerId, PeerIntelligenceResponse], trust_matrix: TrustMatrix): - assert trust_matrix.keys() == responses.keys() - - -class EvenTIEvaluation(TIEvaluation): - """Basic implementation for the TI evaluation, all responses are evaluated the same. - This implementation corresponds with Salinity botnet. - """ - - def __init__(self, **kwargs): - self.__kwargs = kwargs - self.__satisfaction = kwargs.get('satisfaction', SatisfactionLevels.Ok) - - def evaluate(self, - aggregated_ti: SlipsThreatIntelligence, - responses: Dict[PeerId, PeerIntelligenceResponse], - trust_matrix: TrustMatrix, - **kwargs, - ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: - super()._assert_keys(responses, trust_matrix) - - return {p.peer_id: (p, self.__satisfaction, self._weight()) for p in - trust_matrix.values()} - - -class DistanceBasedTIEvaluation(TIEvaluation): - """Implementation that takes distance from the aggregated result and uses it as a penalisation.""" - - def __init__(self, **kwargs): - self.__kwargs = kwargs - - def evaluate(self, - aggregated_ti: SlipsThreatIntelligence, - responses: Dict[PeerId, PeerIntelligenceResponse], - trust_matrix: TrustMatrix, - **kwargs, - ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: - super()._assert_keys(responses, trust_matrix) - return self._build_evaluation( - baseline_score=aggregated_ti.score, - baseline_confidence=aggregated_ti.confidence, - responses=responses, - trust_matrix=trust_matrix - ) - - def _build_evaluation( - self, - baseline_score: float, - baseline_confidence: float, - responses: Dict[PeerId, PeerIntelligenceResponse], - trust_matrix: TrustMatrix, - ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: - satisfactions = { - peer_id: self._satisfaction( - baseline_score=baseline_score, - baseline_confidence=baseline_confidence, - report_score=ti.intelligence.score, - report_confidence=ti.intelligence.confidence - ) - for peer_id, ti in responses.items() - } - - return {p.peer_id: (p, satisfactions[p.peer_id], self._weight()) for p in - trust_matrix.values()} - - @staticmethod - def _satisfaction(baseline_score: float, - baseline_confidence: float, - report_score: float, - report_confidence: float) -> Satisfaction: - return (1 - (abs(baseline_score - report_score) / 2) * report_confidence) * baseline_confidence - - -class LocalCompareTIEvaluation(DistanceBasedTIEvaluation): - """This strategy compares received threat intelligence with the threat intelligence from local database. - - Uses the same penalisation system as DistanceBasedTIEvaluation with the difference that as a baseline, - it does not use aggregated value, but rather local intelligence. - - If it does not find threat intelligence for the target, it falls backs to DistanceBasedTIEvaluation. - """ - - def __init__(self, **kwargs): - super().__init__(**kwargs) - self.__default_ti_getter = kwargs.get('default_ti_getter', None) - - def get_local_ti(self, - target: Target, - local_ti: Optional[SlipsThreatIntelligence] = None) -> Optional[SlipsThreatIntelligence]: - if local_ti: - return local_ti - elif self.__default_ti_getter: - return self.__default_ti_getter(target) - else: - return None - - def evaluate(self, - aggregated_ti: SlipsThreatIntelligence, - responses: Dict[PeerId, PeerIntelligenceResponse], - trust_matrix: TrustMatrix, - local_ti: Optional[SlipsThreatIntelligence] = None, - **kwargs, - ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: - super()._assert_keys(responses, trust_matrix) - - ti = self.get_local_ti(aggregated_ti.target, local_ti) - if not ti: - ti = aggregated_ti - logger.warn(f'No local threat intelligence available for target {ti.target}! ' + - 'Falling back to DistanceBasedTIEvaluation.') - - return self._build_evaluation( - baseline_score=ti.score, - baseline_confidence=ti.confidence, - responses=responses, - trust_matrix=trust_matrix - ) - - -class WeighedDistanceToLocalTIEvaluation(TIEvaluation): - """Strategy combines DistanceBasedTIEvaluation and LocalCompareTIEvaluation with the local weight parameter.""" - - def __init__(self, **kwargs): - super().__init__() - self.__distance = kwargs.get('distance', DistanceBasedTIEvaluation()) - self.__local = kwargs.get('localDistance', LocalCompareTIEvaluation()) - self.__local_weight = kwargs.get('localWeight', 0.5) - - def evaluate(self, - aggregated_ti: SlipsThreatIntelligence, - responses: Dict[PeerId, PeerIntelligenceResponse], - trust_matrix: TrustMatrix, - **kwargs, - ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: - super()._assert_keys(responses, trust_matrix) - - distance_data = self.__distance.evaluate(aggregated_ti, responses, trust_matrix, **kwargs) - local_data = self.__local.evaluate(aggregated_ti, responses, trust_matrix, **kwargs) - - return {p.peer_id: (p, - self.__local_weight * local_data[p.peer_id][1] + - (1 - self.__local_weight) * distance_data[p.peer_id][1], - self._weight() - ) for p in trust_matrix.values()} - - -class MaxConfidenceTIEvaluation(TIEvaluation): - """Strategy combines DistanceBasedTIEvaluation, LocalCompareTIEvaluation and EvenTIEvaluation - in order to achieve maximal confidence when producing decision. - """ - - def __init__(self, **kwargs): - super().__init__() - self.__distance = kwargs.get('distance', DistanceBasedTIEvaluation()) - self.__local = kwargs.get('localDistance', LocalCompareTIEvaluation()) - self.__even = kwargs.get('even', EvenTIEvaluation()) - - def evaluate(self, - aggregated_ti: SlipsThreatIntelligence, - responses: Dict[PeerId, PeerIntelligenceResponse], - trust_matrix: TrustMatrix, - **kwargs, - ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: - super()._assert_keys(responses, trust_matrix) - zero_dict = defaultdict(lambda: (None, 0, None)) - - # weight of the distance based evaluation - distance_weight = aggregated_ti.confidence - distance_data = self.__distance.evaluate(aggregated_ti, responses, trust_matrix, **kwargs) \ - if distance_weight > 0 \ - else zero_dict - - # now we need to check if we even have some threat intelligence data - local_ti = self.__local.get_local_ti(aggregated_ti.target, **kwargs) - # weight of the local evaluation - local_weight = min(1 - distance_weight, local_ti.confidence) if local_ti else 0 - local_data = self.__local.evaluate(aggregated_ti, responses, trust_matrix, **kwargs) \ - if local_weight > 0 \ - else zero_dict - - # weight of the same eval - even_weight = 1 - distance_weight - local_weight - even_data = self.__even.evaluate(aggregated_ti, responses, trust_matrix, **kwargs) \ - if even_weight > 0 \ - else zero_dict - - def aggregate(peer: PeerId): - return distance_weight * distance_data[peer][1] + \ - local_weight * local_data[peer][1] + \ - even_weight * even_data[peer][1] - - return {p.peer_id: (p, aggregate(p.peer_id), self._weight()) for p in - trust_matrix.values()} - - -class ThresholdTIEvaluation(TIEvaluation): - """Employs DistanceBasedTIEvaluation when the confidence of the decision - is higher than given threshold. Otherwise, it uses even evaluation. - """ - - def __init__(self, **kwargs): - self.__kwargs = kwargs - self.__threshold = kwargs.get('threshold', 0.5) - self.__lower = kwargs.get('lower', EvenTIEvaluation()) - self.__higher = kwargs.get('higher', DistanceBasedTIEvaluation()) - - def evaluate(self, - aggregated_ti: SlipsThreatIntelligence, - responses: Dict[PeerId, PeerIntelligenceResponse], - trust_matrix: TrustMatrix, - **kwargs, - ) -> Dict[PeerId, Tuple[PeerTrustData, Satisfaction, Weight]]: - super()._assert_keys(responses, trust_matrix) - - return self.__higher.evaluate(aggregated_ti, responses, trust_matrix) \ - if self.__threshold <= aggregated_ti.confidence \ - else self.__lower.evaluate(aggregated_ti, responses, trust_matrix) - - -EvaluationStrategy = { - 'even': EvenTIEvaluation, - 'distance': DistanceBasedTIEvaluation, - 'localDistance': LocalCompareTIEvaluation, - 'threshold': ThresholdTIEvaluation, - 'maxConfidence': MaxConfidenceTIEvaluation, - 'weighedDistance': WeighedDistanceToLocalTIEvaluation -} diff --git a/modules/fidesModule/messaging/message_handler.py b/modules/fidesModule/messaging/message_handler.py deleted file mode 100644 index 41f235e3cb..0000000000 --- a/modules/fidesModule/messaging/message_handler.py +++ /dev/null @@ -1,176 +0,0 @@ -from http.client import responses -from typing import Dict, List, Callable, Optional, Union, Any - -from absl.logging import debug - -from slips_files.common.printer import Printer -from ..messaging.dacite import from_dict - -from ..messaging.model import NetworkMessage, PeerInfo, \ - PeerIntelligenceResponse, PeerRecommendationResponse -from ..model.alert import Alert -from ..model.aliases import PeerId, Target -from ..model.recommendation import Recommendation -from ..model.threat_intelligence import ThreatIntelligence -from ..utils.logger import Logger - -logger = Logger(__name__) - - - -class MessageHandler: - """ - Class responsible for parsing messages and handling requests coming from the queue. - - The entrypoint is on_message. - """ - - - - #def print(self, *args, **kwargs): - # return self.printer.print(*args, **kwargs) - - version = 1 - - def __init__(self, - on_peer_list_update: Callable[[List[PeerInfo]], None], - on_recommendation_request: Callable[[str, PeerInfo, PeerId], None], - on_recommendation_response: Callable[[List[PeerRecommendationResponse]], None], - on_alert: Callable[[PeerInfo, Alert], None], - on_intelligence_request: Callable[[str, PeerInfo, Target], None], - on_intelligence_response: Callable[[List[PeerIntelligenceResponse]], None], - on_unknown: Optional[Callable[[NetworkMessage], None]] = None, - on_error: Optional[Callable[[Union[str, NetworkMessage], Exception], None]] = None - ): - #self.logger = None - self.__on_peer_list_update_callback = on_peer_list_update - self.__on_recommendation_request_callback = on_recommendation_request - self.__on_recommendation_response_callback = on_recommendation_response - self.__on_alert_callback = on_alert - self.__on_intelligence_request_callback = on_intelligence_request - self.__on_intelligence_response_callback = on_intelligence_response - self.__on_unknown_callback = on_unknown - self.__on_error = on_error - #self.printer = Printer(self.logger, self.name) - - def on_message(self, message: NetworkMessage): - """ - Entry point for generic messages coming from the queue. - This method parses the message and then executes correct procedure from event. - :param message: message from the queue - :return: value from the underlining function from the constructor - """ - if message.version != self.version: - logger.warn(f'Unknown message version! This handler supports {self.version}.', message) - return self.__on_unknown_message(message) - - execution_map = { - 'nl2tl_peers_list': self.__on_nl2tl_peer_list, - 'nl2tl_recommendation_request': self.__on_nl2tl_recommendation_request, - 'nl2tl_recommendation_response': self.__on_nl2tl_recommendation_response, - 'nl2tl_alert': self.__on_nl2tl_alert, - 'nl2tl_intelligence_request': self.__on_nl2tl_intelligence_request, - 'nl2tl_intelligence_response': self.__on_nl2tl_intelligence_response - } - func = execution_map.get(message.type, lambda data: self.__on_unknown_message(message)) - # we want to handle everything - # noinspection PyBroadException - try: - # we know that the functions can handle that, and if not, there's always error handling - # noinspection PyArgumentList - return func(message.data) - except Exception as ex: - logger.error(f"Error when executing handler for message: {message.type}.", ex) - if self.__on_error: - return self.__on_error(message, ex) - - def on_error(self, original_data: str, exception: Optional[Exception] = None): - """ - Should be executed when it was not possible to parse the message. - :param original_data: string received from the queue - :param exception: exception that occurred during handling - :return: - """ - logger.error(f'Unknown data received: {original_data}.') - if self.__on_error: - self.__on_error(original_data, exception if exception else Exception('Unknown data type!')) - - def __on_unknown_message(self, message: NetworkMessage): - logger.warn(f'Unknown message handler executed!') - logger.debug(f'Message:', message) - - if self.__on_unknown_callback is not None: - self.__on_unknown_callback(message) - - def __on_nl2tl_peer_list(self, data: Dict): - logger.debug('nl2tl_peer_list message') - - peers = [from_dict(data_class=PeerInfo, data=peer) for peer in data['peers']] - return self.__on_peer_list_update(peers) - - def __on_peer_list_update(self, peers: List[PeerInfo]): - return self.__on_peer_list_update_callback(peers) - - def __on_nl2tl_recommendation_request(self, data: Dict): - logger.debug('nl2tl_recommendation_request message') - - request_id = data['request_id'] - sender = from_dict(data_class=PeerInfo, data=data['sender']) - subject = data['payload'] - return self.__on_recommendation_request(request_id, sender, subject) - - def __on_recommendation_request(self, request_id: str, sender: PeerInfo, subject: PeerId): - return self.__on_recommendation_request_callback(request_id, sender, subject) - - def __on_nl2tl_recommendation_response(self, data: List[Dict]): - logger.debug('nl2tl_recommendation_response message') - - responses = [PeerRecommendationResponse( - sender=from_dict(data_class=PeerInfo, data=single['sender']), - subject=single['payload']['subject'], - recommendation=from_dict(data_class=Recommendation, data=single['payload']['recommendation']) - ) for single in data] - return self.__on_recommendation_response(responses) - - def __on_recommendation_response(self, recommendations: List[PeerRecommendationResponse]): - return self.__on_recommendation_response_callback(recommendations) - - def __on_nl2tl_alert(self, data: Dict): - logger.debug('nl2tl_alert message') - - sender = from_dict(data_class=PeerInfo, data=data['sender']) - alert = from_dict(data_class=Alert, data=data['payload']) - return self.__on_alert(sender, alert) - - def __on_alert(self, sender: PeerInfo, alert: Alert): - return self.__on_alert_callback(sender, alert) - - def __on_nl2tl_intelligence_request(self, data: Dict): - logger.debug('nl2tl_intelligence_request message') - - request_id = data['request_id'] - sender = from_dict(data_class=PeerInfo, data=data['sender']) - target = data['payload'] - return self.__on_intelligence_request(request_id, sender, target) - - def __on_intelligence_request(self, request_id: str, sender: PeerInfo, target: Target): - return self.__on_intelligence_request_callback(request_id, sender, target) - - def __on_nl2tl_intelligence_response(self, data: Dict): - logger.debug('nl2tl_intelligence_response message') - - responses = [] - - try: - responses = [PeerIntelligenceResponse( - sender=from_dict(data_class=PeerInfo, data=single['sender']), - intelligence=from_dict(data_class=ThreatIntelligence, data=single['payload']['intelligence']), - target=single['payload']['target'] - ) for single in data] - except Exception as e: - print("Error in Fides message_handler.py __on_nl2tl_intelligence_response(): ", e.__str__()) - #self.print("Error in Fides message_handler.py __on_nl2tl_intelligence_response(): ") - return self.__on_intelligence_response(responses) - - def __on_intelligence_response(self, responses: List[PeerIntelligenceResponse]): - return self.__on_intelligence_response_callback(responses) diff --git a/modules/fidesModule/model/aliases.py b/modules/fidesModule/model/aliases.py deleted file mode 100644 index fed80418e5..0000000000 --- a/modules/fidesModule/model/aliases.py +++ /dev/null @@ -1,30 +0,0 @@ -IP = str -"""IPv4, IPv6 in string representation.""" - -Domain = str -"""Host Name, Domain.""" - -PeerId = str -"""String representation of peer's public key. """ - -OrganisationId = str -"""String representation of organisation ID.""" - -Target = str -"""Intelligence Target - domain or IP.""" - -ConfidentialityLevel = float -"""Confidentiality level for threat intelligence. - -If an entity needs to have access to any data, it must mean - -entity.confidentiality_level >= data.confidentiality_level - -thus level 0 means accessible for everybody -""" - -Score = float -"""Score for the target, -1 <= score <= 1""" - -Confidence = float -"""Confidence in score, 0 <= confidence <= 1""" diff --git a/modules/fidesModule/model/recommendation_history.py b/modules/fidesModule/model/recommendation_history.py deleted file mode 100644 index 340d82aa07..0000000000 --- a/modules/fidesModule/model/recommendation_history.py +++ /dev/null @@ -1,49 +0,0 @@ -from dataclasses import dataclass -from typing import List - -from ..utils.time import Time - - -@dataclass -class RecommendationHistoryRecord: - """Represents an evaluation of a single recommendation interaction between peer i and peer j.""" - - satisfaction: float - """Peer's satisfaction with the recommendation. In model's notation rs_ij. - - 0 <= satisfaction <= 1 - """ - - weight: float - """Weight of the recommendation. In model's notation rw_ij. - - 0 <= weight <= 1 - """ - - timestamp: Time - """Date time when this recommendation happened.""" - - - def to_dict(self): - """Convert the instance to a dictionary.""" - return { - 'satisfaction': self.satisfaction, - 'weight': self.weight, - 'timestamp': self.timestamp # Keep as float - } - - @classmethod - def from_dict(cls, dict_obj): - """Create an instance of RecommendationHistoryRecord from a dictionary.""" - return cls( - satisfaction=dict_obj['satisfaction'], - weight=dict_obj['weight'], - timestamp=dict_obj['timestamp'] # Keep as float - ) - - -RecommendationHistory = List[RecommendationHistoryRecord] -"""Ordered list with history of recommendation interactions. - -First element in the list is the oldest one. -""" diff --git a/modules/fidesModule/model/service_history.py b/modules/fidesModule/model/service_history.py deleted file mode 100644 index d9526a63a4..0000000000 --- a/modules/fidesModule/model/service_history.py +++ /dev/null @@ -1,48 +0,0 @@ -from dataclasses import dataclass -from typing import List - -from ..utils.time import Time - - -@dataclass -class ServiceHistoryRecord: - """Represents an evaluation of a single service interaction between peer i and peer j.""" - - satisfaction: float - """Peer's satisfaction with the service. In model's notation s_ij. - - 0 <= satisfaction <= 1 - """ - - weight: float - """Weight of the service interaction. In model's notation w_ij. - - 0 <= weight <= 1 - """ - - timestamp: Time - """Date time when this interaction happened.""" - - def to_dict(self): - """Convert the instance to a dictionary.""" - return { - 'satisfaction': self.satisfaction, - 'weight': self.weight, - 'timestamp': self.timestamp - } - - @classmethod - def from_dict(cls, dict_obj): - """Create an instance of ServiceHistoryRecord from a dictionary.""" - return cls( - satisfaction=dict_obj['satisfaction'], - weight=dict_obj['weight'], - timestamp=dict_obj['timestamp'] # Convert ISO format back to datetime - ) - - -ServiceHistory = List[ServiceHistoryRecord] -"""Ordered list with history of service interactions. - -First element in the list is the oldest one. -""" diff --git a/modules/fidesModule/protocols/opinion.py b/modules/fidesModule/protocols/opinion.py deleted file mode 100644 index 79cb89b30c..0000000000 --- a/modules/fidesModule/protocols/opinion.py +++ /dev/null @@ -1,43 +0,0 @@ -from typing import Dict - -from ..evaluation.ti_aggregation import TIAggregation, PeerReport -from ..messaging.model import PeerIntelligenceResponse -from ..model.alert import Alert -from ..model.aliases import PeerId, Target -from ..model.configuration import TrustModelConfiguration -from ..model.peer_trust_data import PeerTrustData, TrustMatrix -from ..model.threat_intelligence import SlipsThreatIntelligence -from ..persistence.threat_intelligence_db import SlipsThreatIntelligenceDatabase - - -class OpinionAggregator: - """ - Class responsible for evaluation of the intelligence received from the network. - """ - - def __init__(self, - configuration: TrustModelConfiguration, - ti_db: SlipsThreatIntelligenceDatabase, - ti_aggregation: TIAggregation): - self.__configuration = configuration - self.__ti_db = ti_db - self.__ti_aggregation = ti_aggregation - - def evaluate_alert(self, peer_trust: PeerTrustData, alert: Alert) -> SlipsThreatIntelligence: - """Evaluates given data about alert and produces aggregated intelligence for Slips.""" - - alert_trust = max(self.__configuration.alert_trust_from_unknown, peer_trust.service_trust) - score = alert.score - confidence = alert.confidence * alert_trust - return SlipsThreatIntelligence(score=score, confidence=confidence, target=alert.target) - - def evaluate_intelligence_response(self, - target: Target, - data: Dict[PeerId, PeerIntelligenceResponse], - trust_matrix: TrustMatrix) -> SlipsThreatIntelligence: - """Evaluates given threat intelligence report from the network.""" - reports = [PeerReport(report_ti=ti.intelligence, - reporter_trust=trust_matrix[peer_id] - ) for peer_id, ti in data.items()] - ti = self.__ti_aggregation.assemble_peer_opinion(data=reports) - return SlipsThreatIntelligence(score=ti.score, confidence=ti.confidence, target=target) diff --git a/modules/fidesModule/utils/time.py b/modules/fidesModule/utils/time.py deleted file mode 100644 index e802070f6b..0000000000 --- a/modules/fidesModule/utils/time.py +++ /dev/null @@ -1,14 +0,0 @@ -import time - -Time = float -"""Type for time used across the whole module. - -Represents the current time in seconds since the Epoch. Can have frictions of seconds. - -We have it as alias so we can easily change that in the future. -""" - - -def now() -> Time: - """Returns current Time.""" - return time.time() diff --git a/modules/flowalerts/__init__.py b/modules/flow_alerts/__init__.py similarity index 100% rename from modules/flowalerts/__init__.py rename to modules/flow_alerts/__init__.py diff --git a/modules/flowalerts/conn.py b/modules/flow_alerts/conn.py similarity index 99% rename from modules/flowalerts/conn.py rename to modules/flow_alerts/conn.py index ba021112e5..1ad44ca04d 100644 --- a/modules/flowalerts/conn.py +++ b/modules/flow_alerts/conn.py @@ -8,7 +8,7 @@ from typing import Tuple, List, Dict import validators -from modules.flowalerts.dns import DNS +from modules.flow_alerts.dns import DNS from slips_files.common.abstracts.iflowalerts_analyzer import ( IFlowalertsAnalyzer, ) diff --git a/modules/flowalerts/dns.py b/modules/flow_alerts/dns.py similarity index 99% rename from modules/flowalerts/dns.py rename to modules/flow_alerts/dns.py index 297d3512a1..8bfd523f56 100644 --- a/modules/flowalerts/dns.py +++ b/modules/flow_alerts/dns.py @@ -61,7 +61,7 @@ def init(self): # the reason why we can just use .get_msg() there is because once # the msg is handled here, it wont be passed to other analyzers the # should analyze it anymore. - # meaning, only flowalerts.py is allowed to do a get_msg because it + # meaning, only flow_alerts.py is allowed to do a get_msg because it # manages all the analyzers the msg should be passed to self.dns_msgs = Queue() self.priv_ips_doing_dns_outside_of_localnet = {} @@ -752,7 +752,7 @@ def pre_analyze(self): async def analyze(self, msg): """ - is only used by flowalerts.py + is only used by flow_alerts.py runs whenever we get a new_dns message """ if not utils.is_msg_intended_for(msg, "new_dns"): diff --git a/modules/flowalerts/downloaded_file.py b/modules/flow_alerts/downloaded_file.py similarity index 100% rename from modules/flowalerts/downloaded_file.py rename to modules/flow_alerts/downloaded_file.py diff --git a/modules/flowalerts/flowalerts.py b/modules/flow_alerts/flow_alerts.py similarity index 95% rename from modules/flowalerts/flowalerts.py rename to modules/flow_alerts/flow_alerts.py index abf4cf5a8e..f41dc96083 100644 --- a/modules/flowalerts/flowalerts.py +++ b/modules/flow_alerts/flow_alerts.py @@ -20,7 +20,7 @@ class FlowAlerts(AsyncModule): - name = "Flow Alerts" + name = "flow_alerts" description = ( "Alerts about flows: long connection, successful ssh, " "password guessing, self-signed certificate, data exfiltration, etc." @@ -41,7 +41,7 @@ def init(self): self.downloaded_file = DownloadedFile(self.db, flowalerts=self) self.tunnel = Tunnel(self.db, flowalerts=self) self.conn = Conn(self.db, flowalerts=self) - # list of async functions to await before flowalerts shuts down + # list of async functions to await before flow_alerts shuts down self.tasks: List[Task] = [] def subscribe_to_channels(self): @@ -100,7 +100,7 @@ async def main(self): task = loop.create_task(analyzer.analyze(msg)) # because Async Tasks swallow exceptions. task.add_done_callback(self.handle_task_exception) - # to wait for these functions before flowalerts shuts down + # to wait for these functions before flow_alerts shuts down self.tasks.append(task) # Allow the event loop to run the scheduled task await asyncio.sleep(0) diff --git a/modules/flowalerts/notice.py b/modules/flow_alerts/notice.py similarity index 100% rename from modules/flowalerts/notice.py rename to modules/flow_alerts/notice.py diff --git a/modules/flowalerts/set_evidence.py b/modules/flow_alerts/set_evidence.py similarity index 100% rename from modules/flowalerts/set_evidence.py rename to modules/flow_alerts/set_evidence.py diff --git a/modules/flowalerts/smtp.py b/modules/flow_alerts/smtp.py similarity index 100% rename from modules/flowalerts/smtp.py rename to modules/flow_alerts/smtp.py diff --git a/modules/flowalerts/software.py b/modules/flow_alerts/software.py similarity index 100% rename from modules/flowalerts/software.py rename to modules/flow_alerts/software.py diff --git a/modules/flowalerts/ssh.py b/modules/flow_alerts/ssh.py similarity index 100% rename from modules/flowalerts/ssh.py rename to modules/flow_alerts/ssh.py diff --git a/modules/flowalerts/ssl.py b/modules/flow_alerts/ssl.py similarity index 100% rename from modules/flowalerts/ssl.py rename to modules/flow_alerts/ssl.py diff --git a/modules/flowalerts/timer_thread.py b/modules/flow_alerts/timer_thread.py similarity index 100% rename from modules/flowalerts/timer_thread.py rename to modules/flow_alerts/timer_thread.py diff --git a/modules/flowalerts/tunnel.py b/modules/flow_alerts/tunnel.py similarity index 100% rename from modules/flowalerts/tunnel.py rename to modules/flow_alerts/tunnel.py diff --git a/modules/flowmldetection/__init__.py b/modules/flow_ml_detection/__init__.py similarity index 100% rename from modules/flowmldetection/__init__.py rename to modules/flow_ml_detection/__init__.py diff --git a/modules/flowmldetection/flowmldetection.py b/modules/flow_ml_detection/flow_ml_detection.py similarity index 98% rename from modules/flowmldetection/flowmldetection.py rename to modules/flow_ml_detection/flow_ml_detection.py index 21fd7d4030..bdda4e0ab3 100644 --- a/modules/flowmldetection/flowmldetection.py +++ b/modules/flow_ml_detection/flow_ml_detection.py @@ -41,7 +41,7 @@ def warn(*args, **kwargs): class FlowMLDetection(IModule): # Name: short name of the module. Do not use spaces - name = "Flow ML Detection" + name = "flow_ml_detection" description = ( "Train or test a Machine Learning model to detect malicious flows" ) @@ -58,8 +58,8 @@ def init(self): # self.scores = [] # The scaler trained during training and to use during testing self.scaler = StandardScaler() - self.model_path = "./modules/flowmldetection/model.bin" - self.scaler_path = "./modules/flowmldetection/scaler.bin" + self.model_path = "./modules/flow_ml_detection/model.bin" + self.scaler_path = "./modules/flow_ml_detection/scaler.bin" def subscribe_to_channels(self): # Subscribe to the channel @@ -250,7 +250,7 @@ def process_flows(self): "appproto": "ssl", "label": "Malware", "module_labels": { - "flowalerts-long-connection": "Malware" + "flow_alerts-long-connection": "Malware" }, } ) @@ -270,7 +270,7 @@ def process_flows(self): "appproto": "http", "label": "Normal", "module_labels": { - "flowalerts-long-connection": "Normal" + "flow_alerts-long-connection": "Normal" }, } ) diff --git a/modules/flowmldetection/model.bin b/modules/flow_ml_detection/model.bin similarity index 100% rename from modules/flowmldetection/model.bin rename to modules/flow_ml_detection/model.bin diff --git a/modules/flowmldetection/model.license b/modules/flow_ml_detection/model.license similarity index 100% rename from modules/flowmldetection/model.license rename to modules/flow_ml_detection/model.license diff --git a/modules/flowmldetection/scaler.bin b/modules/flow_ml_detection/scaler.bin similarity index 100% rename from modules/flowmldetection/scaler.bin rename to modules/flow_ml_detection/scaler.bin diff --git a/modules/flowmldetection/scaler.license b/modules/flow_ml_detection/scaler.license similarity index 100% rename from modules/flowmldetection/scaler.license rename to modules/flow_ml_detection/scaler.license diff --git a/modules/http_analyzer/http_analyzer.py b/modules/http_analyzer/http_analyzer.py index 8f1e29bd26..c5df15aa93 100644 --- a/modules/http_analyzer/http_analyzer.py +++ b/modules/http_analyzer/http_analyzer.py @@ -21,7 +21,7 @@ class HTTPAnalyzer(AsyncModule): # Name: short name of the module. Do not use spaces - name = "HTTP Analyzer" + name = "http_analyzer" description = "Analyze HTTP flows" authors = ["Alya Gomaa"] diff --git a/modules/ip_info/ip_info.py b/modules/ip_info/ip_info.py index fe50b2cbd5..f0ced354e8 100644 --- a/modules/ip_info/ip_info.py +++ b/modules/ip_info/ip_info.py @@ -44,7 +44,7 @@ class IPInfo(AsyncModule): # Name: short name of the module. Do not use spaces - name = "IP Info" + name = "ip_info" description = "Get different info about an IP/MAC address" authors = ["Alya Gomaa", "Sebastian Garcia"] diff --git a/modules/irisModule/__init__.py b/modules/iris/__init__.py similarity index 100% rename from modules/irisModule/__init__.py rename to modules/iris/__init__.py diff --git a/modules/irisModule/iris b/modules/iris/iris similarity index 100% rename from modules/irisModule/iris rename to modules/iris/iris diff --git a/modules/irisModule/irisModule.py b/modules/iris/iris.py similarity index 97% rename from modules/irisModule/irisModule.py rename to modules/iris/iris.py index d393df4cb5..40bce92680 100644 --- a/modules/irisModule/irisModule.py +++ b/modules/iris/iris.py @@ -11,9 +11,9 @@ import yaml -class IrisModule(IModule): +class Iris(IModule): # Name: short name of the module. Do not use spaces - name = "Iris" + name = "iris" description = "Global P2P module cooperating with Fides" authors = ["David Otta"] process = None @@ -127,14 +127,14 @@ def pre_main(self): self.print(f"Running Iris using command: {command_str}") - log_dir = os.path.join(self.output_dir, "iris") - os.makedirs(log_dir, exist_ok=True) # Open the log file - self.log_file_path = os.path.join(log_dir, "iris_logs.txt") + self.log_file_path = self.get_module_specific_output_path( + "iris_logs.txt" + ) self.log_file = open(self.log_file_path, "w") self.log_file.write(f"Running Iris using command: {command_str}") - full_cwd = Path.cwd() / "modules" / "irisModule" + full_cwd = Path.cwd() / "modules" / "iris" try: # Start the subprocess, redirecting stdout and stderr to the same file diff --git a/modules/leak_detector/leak_detector.py b/modules/leak_detector/leak_detector.py index 1a99c84a89..c610387bb9 100644 --- a/modules/leak_detector/leak_detector.py +++ b/modules/leak_detector/leak_detector.py @@ -31,7 +31,7 @@ class LeakDetector(IModule): # Name: short name of the module. Do not use spaces - name = "Leak Detector" + name = "leak_detector" description = "Detect leaks of data in the traffic" authors = ["Alya Gomaa"] diff --git a/modules/network_discovery/network_discovery.py b/modules/network_discovery/network_discovery.py index 7931b44a51..556f40929f 100644 --- a/modules/network_discovery/network_discovery.py +++ b/modules/network_discovery/network_discovery.py @@ -26,7 +26,7 @@ class NetworkDiscovery(IModule): This should be converted into a module that wakesup alone when a new alert arrives """ - name = "Network Discovery" + name = "network_discovery" description = "Detect Horizonal, Vertical, and DHCP Scans." authors = ["Sebastian Garcia", "Alya Gomaa"] diff --git a/modules/p2ptrust/__init__.py b/modules/p2p_trust/__init__.py similarity index 100% rename from modules/p2ptrust/__init__.py rename to modules/p2p_trust/__init__.py diff --git a/modules/p2ptrust/data_format.md b/modules/p2p_trust/data_format.md similarity index 96% rename from modules/p2ptrust/data_format.md rename to modules/p2p_trust/data_format.md index 66cddab51d..ba0ae9eff6 100644 --- a/modules/p2ptrust/data_format.md +++ b/modules/p2p_trust/data_format.md @@ -1,6 +1,6 @@ # Data saved to Slips -Slips database expects a dictionary of data. The data from this module (for a given IP) has the following format: +Slips database expects a dictionary of data. The data from this module (for a given IP) has the following format: ```json { @@ -14,18 +14,18 @@ Slips database expects a dictionary of data. The data from this module (for a gi ``` The `p2p4slips` field in the database will the report from the network. The report will have the IP address that is -reported, the computed score and confidence, and an additional value with score of all the peers that gave the opinion. +reported, the computed score and confidence, and an additional value with score of all the peers that gave the opinion. # Communication between python and go parts of the implementation The core of each peer is implemented in python. The python code collects data, shares this data with other peers (report), asks other peers for data (request) etc. Python code doesn't communicate with other peers directly - it relies on the go part of the node to do that work. The two parts of the node exchange information and instructions using redis -channels. +channels. ## The channel from Slips to Go `p2p_pygo` -Slips sends data to go in json. The Json object has two fields: `message` and `recipient`. +Slips sends data to go in json. The Json object has two fields: `message` and `recipient`. ```json {"message": "ewogICAgImtleV90eXBlIjogImlwIiwKICAgICJrZXkiOiAiMS4yLjMuNDAiLAogICAgImV........jYKfQ==", "recipient": "QmYyQSo1c1Ym7orWxLYvCrM2EmxFTANf8wXmmE7DWjhx5N"} @@ -53,7 +53,7 @@ more reports at the same time. ```json { "message_type": "go_data", - "message_contents": + "message_contents": { "reporter": "abcsakughroiauqrghaui", // the peer that sent the data "report_time": 154900000, // time of receiving the data diff --git a/modules/p2ptrust/p2ptrust.py b/modules/p2p_trust/p2p_trust.py similarity index 95% rename from modules/p2ptrust/p2ptrust.py rename to modules/p2p_trust/p2p_trust.py index 0f2af29075..47396c4373 100644 --- a/modules/p2ptrust/p2ptrust.py +++ b/modules/p2p_trust/p2p_trust.py @@ -9,12 +9,13 @@ import json import socket +from slips_files.common.style import green from slips_files.common.parsers.config_parser import ConfigParser from slips_files.common.slips_utils import utils from slips_files.common.abstracts.imodule import IModule -import modules.p2ptrust.trust.base_model as reputation_model -import modules.p2ptrust.utils.utils as p2p_utils -from modules.p2ptrust.utils.go_director import GoDirector +import modules.p2p_trust.trust.base_model as reputation_model +import modules.p2p_trust.utils.utils as p2p_utils +from modules.p2p_trust.utils.go_director import GoDirector from slips_files.core.structures.evidence import ( dict_to_evidence, Evidence, @@ -68,7 +69,7 @@ def validate_slips_data(message_data: str) -> (str, int): class Trust(IModule): - name = "P2P Trust" + name = "p2p_trust" description = "Enables sharing detection data with other Slips instances" authors = ["Dita", "Alya Gomaa"] pigeon_port = 6668 @@ -92,8 +93,8 @@ def init(self, *args, **kwargs): # output dir! so it wont find them and will # generate new keys, and therefore new peerid! # store the keys in slips main dir so they don't change every run - self.p2ptrust_runtime_dir = self.db.get_p2ptrust_dir() - self.sql_db_name = self.db.get_p2ptrust_db_path() + self.p2p_trust_runtime_dir = self.db.get_p2p_trust_dir() + self.sql_db_name = self.db.get_p2p_trust_db_path() self.port = self.get_available_port() self.host = self.get_local_IP() @@ -137,9 +138,11 @@ def subscribe_to_channels(self): def _init_log_files(self): # should be called after reading configs - self.pigeon_logfile_raw = os.path.join(self.output_dir, "p2p.log") - self.p2p_reports_logfile = os.path.join( - self.output_dir, "p2p_reports.log" + self.pigeon_logfile_raw = self.get_module_specific_output_path( + "p2p.log" + ) + self.p2p_reports_logfile = self.get_module_specific_output_path( + "p2p_reports.log" ) if self.create_p2p_logfile: self.setup_pigeon_logfile_rotation() @@ -178,8 +181,7 @@ def _configure(self): self.reputation_model = reputation_model.BaseModel( self.logger, self.trust_db, self.db ) - # print(f"[DEBUGGING] Starting godirector with - # pygo_channel: {self.pygo_channel}") + self.go_director = GoDirector( self.logger, self.trust_db, @@ -197,7 +199,8 @@ def _configure(self): if self.start_pigeon: if not shutil.which(self.pigeon_binary): self.print( - f'P2p4slips binary not found in "{self.pigeon_binary}". ' + f"Warning: P2p4slips binary not found in " + f'"{self.pigeon_binary}". ' f"Did you include it in PATH?. Exiting process." ) return @@ -210,10 +213,7 @@ def _configure(self): "-redis-channel-pygo": self.pygo_channel_raw, "-redis-channel-gopy": self.gopy_channel_raw, } - self.print( - f"P2P is listening on {self.host} port {self.port} " - f"(determined by p2p module)" - ) + self.print(f"P2P is listening on {self.host} port {self.port}.") executable = [self.pigeon_binary] + [ item for pair in params.items() for item in pair ] @@ -224,7 +224,7 @@ def _configure(self): outfile = open(os.devnull, "+w") self.pigeon = subprocess.Popen( - executable, cwd=self.p2ptrust_runtime_dir, stdout=outfile + executable, cwd=self.p2p_trust_runtime_dir, stdout=outfile ) def extract_confidence(self, evidence: Evidence) -> Optional[float]: @@ -594,7 +594,7 @@ def process_message_report( self.db.publish("new_blame", data) def shutdown_gracefully(self): - if hasattr(self, "pigeon"): + if hasattr(self, "pigeon") and self.pigeon is not None: self.pigeon.send_signal(signal.SIGINT) if hasattr(self, "trust_db"): self.trust_db.__del__() @@ -646,7 +646,7 @@ def main(self): # give the pigeon time to put the multiaddr in the db time.sleep(2) multiaddr = self.db.get_multiaddr() - self.print(f"You Multiaddress is: {multiaddr}\n") + self.print(f"You Multiaddress is: {green(multiaddr)}\n") self.mutliaddress_printed = True except Exception: diff --git a/modules/p2ptrust/testing/__init__.py b/modules/p2p_trust/testing/__init__.py similarity index 100% rename from modules/p2ptrust/testing/__init__.py rename to modules/p2p_trust/testing/__init__.py diff --git a/modules/p2ptrust/testing/json_data.py b/modules/p2p_trust/testing/json_data.py similarity index 100% rename from modules/p2ptrust/testing/json_data.py rename to modules/p2p_trust/testing/json_data.py diff --git a/modules/p2ptrust/trust/__init__.py b/modules/p2p_trust/trust/__init__.py similarity index 100% rename from modules/p2ptrust/trust/__init__.py rename to modules/p2p_trust/trust/__init__.py diff --git a/modules/p2ptrust/trust/base_model.py b/modules/p2p_trust/trust/base_model.py similarity index 99% rename from modules/p2ptrust/trust/base_model.py rename to modules/p2p_trust/trust/base_model.py index e7c62595d5..090dbc4d9f 100644 --- a/modules/p2ptrust/trust/base_model.py +++ b/modules/p2p_trust/trust/base_model.py @@ -18,7 +18,7 @@ class BaseModel: doesn't issue any requests to other peers. """ - name = "P2P Base Model" + name = "p2p_base_model" def __init__(self, logger: Output, trustdb, main_slips_db: DBManager): self.trustdb = trustdb diff --git a/modules/p2ptrust/trust/model.py b/modules/p2p_trust/trust/model.py similarity index 90% rename from modules/p2ptrust/trust/model.py rename to modules/p2p_trust/trust/model.py index 7b6ee3b762..905a8451a1 100644 --- a/modules/p2ptrust/trust/model.py +++ b/modules/p2p_trust/trust/model.py @@ -1,6 +1,6 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only -from modules.p2ptrust.trust.trustdb import TrustDB +from modules.p2p_trust.trust.trustdb import TrustDB class Model: @@ -10,7 +10,7 @@ class Model: This class defines a method that trust model is expected to have. """ - name = "P2P Model" + name = "p2p_model" def __init__( self, diff --git a/modules/p2ptrust/trust/trustdb.py b/modules/p2p_trust/trust/trustdb.py similarity index 99% rename from modules/p2ptrust/trust/trustdb.py rename to modules/p2p_trust/trust/trustdb.py index 906d6c7820..b2f9042143 100644 --- a/modules/p2ptrust/trust/trustdb.py +++ b/modules/p2p_trust/trust/trustdb.py @@ -9,7 +9,7 @@ class TrustDB(ISQLite): - name = "P2P Trust DB" + name = "p2p_trust_db" def __init__( self, diff --git a/modules/p2ptrust/utils/__init__.py b/modules/p2p_trust/utils/__init__.py similarity index 100% rename from modules/p2ptrust/utils/__init__.py rename to modules/p2p_trust/utils/__init__.py diff --git a/modules/p2ptrust/utils/go_director.py b/modules/p2p_trust/utils/go_director.py similarity index 98% rename from modules/p2ptrust/utils/go_director.py rename to modules/p2p_trust/utils/go_director.py index 374f5535ed..1ee7b0755d 100644 --- a/modules/p2ptrust/utils/go_director.py +++ b/modules/p2p_trust/utils/go_director.py @@ -9,13 +9,13 @@ from slips_files.common.printer import Printer from slips_files.core.output import Output -from modules.p2ptrust.utils.utils import ( +from modules.p2p_trust.utils.utils import ( validate_ip_address, validate_timestamp, get_ip_info_from_slips, send_evaluation_to_go, ) -from modules.p2ptrust.trust.trustdb import TrustDB +from modules.p2p_trust.trust.trustdb import TrustDB from slips_files.common.parsers.config_parser import ConfigParser from slips_files.common.slips_utils import utils from slips_files.core.structures.evidence import ( @@ -30,7 +30,7 @@ class GoDirector: - """Class that deals with requests and reports from the go part of p2ptrust + """Class that deals with requests and reports from the go part of p2p_trust The reports from other peers are processed and inserted into the database directly. Requests from other peers are validated, data is read from the database @@ -38,7 +38,7 @@ class GoDirector: If peer sends invalid data, his reputation is lowered. """ - name = "P2P Go Director" + name = "p2p_go_director" def __init__( self, @@ -181,7 +181,7 @@ def process_go_data(self, report: dict) -> None: elif message_type == "blame": # TODO SLIPS doesn't getthis kind of msgs at all. all reports are treated as one # self.print("blame is not implemented yet", 0, 2) - # calls process_message_report in p2ptrust.py + # calls process_message_report in p2p_trust.py # which gives the report to evidenceProcess to decide whether to block or not self.report_func(reporter, report_time, data) @@ -288,7 +288,7 @@ def process_message_request( # override_p2p is false by default if self.override_p2p: # print("Overriding p2p") - # calls respond_to_message_request in p2ptrust.py + # calls respond_to_message_request in p2p_trust.py self.request_func(key, reporter) else: # self.print("Not overriding p2p") @@ -361,10 +361,10 @@ def process_message_report( return # after making sure that the data received from peers is valid, - # pass the report to p2ptrust module + # pass the report to p2p_trust module # to decide what to do with it if self.override_p2p: - # calls process_message_report in p2ptrust.py + # calls process_message_report in p2p_trust.py self.report_func(reporter, report_time, data) return diff --git a/modules/p2ptrust/utils/utils.py b/modules/p2p_trust/utils/utils.py similarity index 100% rename from modules/p2ptrust/utils/utils.py rename to modules/p2p_trust/utils/utils.py diff --git a/modules/p2ptrust/testing/test_p2p.py b/modules/p2ptrust/testing/test_p2p.py deleted file mode 100644 index d8d6a686f8..0000000000 --- a/modules/p2ptrust/testing/test_p2p.py +++ /dev/null @@ -1,317 +0,0 @@ -import configparser -import time -import modules.p2ptrust.testing.json_data as json_data -from modules.p2ptrust.utils.utils import save_ip_report_to_db -from modules.p2ptrust.p2ptrust import Trust -from modules.p2ptrust.trust.trustdb import TrustDB -from multiprocessing import Queue -from outputProcess import OutputProcess -import json - -# TODO -# base_dir = "/home/dita/ownCloud/stratosphere/SLIPS/modules/p2ptrust/testing/" -# data_dir = base_dir + "data/experiments-" + str(time.time()) + "/" -# os.mkdir(data_dir) - - -def init_tests(pigeon_port=6669): - config = get_default_config() - output_process_queue = Queue() - output_process_thread = OutputProcess(output_process_queue, 1, 1, config) - output_process_thread.start() - - # Start the DB - __database__.start() - __database__.set_output_queue(output_process_queue) - module_process = Trust( - output_process_queue, - config, - data_dir, - rename_with_port=False, - pigeon_port=pigeon_port, - rename_sql_db_file=False, - ) - - module_process.start() - - time.sleep(1) - print("Initialization complete") - - return module_process - - -def set_ip_data(ip: str, data: dict): - # TODO: remove the first call after database is fixed - __database__.set_new_ip(ip) - __database__.setInfoForIPs(ip, data) - - -def test_slips_integration(): - print("Add new peer on IP 192.168.0.4") - # add a new peer abcsakughroiauqrghaui on IP 192.168.0.4 - __database__.publish( - "p2p_gopy", - '{"message_type":"peer_update","message_contents":{"peerid":"abcsakughroiauqrghaui","ip":"192.168.0.4","reliability":1,"timestamp":0}}', - ) - time.sleep(0.5) - print() - - print("Set evaluation for IP 192.168.0.4") - # module_process.sqlite_db.insert_go_score("abcsakughroiauqrghaui", 1, 0) - # module_process.sqlite_db.insert_go_ip_pairing("abcsakughroiauqrghaui", "192.168.0.4", 1) #B - set_ip_data("192.168.0.4", {"score": -0.1, "confidence": 1}) - time.sleep(0.5) - print() - - print("Add a new peer on IP 192.168.0.5") - # add a new peer anotherreporterspeerid on IP 192.168.0.5 - __database__.publish( - "p2p_gopy", - '{"message_type":"peer_update","message_contents":{"peerid":"anotherreporterspeerid","ip":"192.168.0.5","timestamp":0}}', - ) - time.sleep(0.5) - print() - __database__.publish( - "p2p_gopy", - '{"message_type":"peer_update","message_contents":{"peerid":"anotherreporterspeerid","reliability": 0.8,"timestamp":0}}', - ) - time.sleep(0.5) - print() - - print("Set evaluation for IP 192.168.0.5") - # module_process.sqlite_db.insert_go_score("anotherreporterspeerid", 0.8, 0) - # module_process.sqlite_db.insert_go_ip_pairing("anotherreporterspeerid", "192.168.0.5", 1) #C - set_ip_data("192.168.0.5", {"score": 0.1, "confidence": 1}) - time.sleep(0.5) - print() - - # network asks for data about 1.2.3.4 - print("Network asks about IP 1.2.3.4 (we know nothing about it)") - data = json_data.ok_request - __database__.publish( - "p2p_gopy", '{"message_type":"go_data","message_contents":%s}' % data - ) - time.sleep(0.5) - print() - - # slips makes some detections - print("Slips makes a detection of IP 1.2.3.4") - set_ip_data("1.2.3.4", {"score": 0.3, "confidence": 1}) - time.sleep(0.5) - print() - - print("Slips makes a detection of IP 1.2.3.6") - set_ip_data("1.2.3.6", {"score": -1, "confidence": 0.7}) - time.sleep(0.5) - time.sleep(1) - print() - - print("Network shares detections about IP 1.2.3.40 and 1.2.3.5") - # network shares some detections - # {"key_type": "ip", "key": "1.2.3.40", "evaluation_type": "score_confidence", "evaluation": { "score": 0.9, "confidence": 0.6 }} - # {"key_type": "ip", "key": "1.2.3.5", "evaluation_type": "score_confidence", "evaluation": { "score": 0.9, "confidence": 0.7 }} - data = json_data.two_correctA - published_data = '{"message_type":"go_data","message_contents":%s}' % data - __database__.publish("p2p_gopy", published_data) - data = json_data.two_correctB - published_data = '{"message_type":"go_data","message_contents":%s}' % data - __database__.publish("p2p_gopy", published_data) - time.sleep(1) - print() - - print("Network shares empty detection about IP 1.2.3.7") - data = json_data.ok_empty_report - __database__.publish( - "p2p_gopy", '{"message_type":"go_data","message_contents":%s}' % data - ) - time.sleep(1) - print() - - print("Slips asks about data for 1.2.3.5") - # slips asks for data about 1.2.3.5 - data_to_send = { - "ip": "tst", - "profileid": "profileid_192.168.1.1", - "twid": "timewindow1", - "proto": "TCP", - "ip_state": "dstip", - "stime": time.time(), - "uid": "123", - "cache_age": 1000, - } - data_to_send = json.dumps(data_to_send) - __database__.publish("p2p_data_request", data_to_send) - time.sleep(1) - print() - - # network asks for data about 1.2.3.4 - print("Network asks about IP 1.2.3.4 (we know something now)") - data = json_data.ok_request - __database__.publish( - "p2p_gopy", '{"message_type":"go_data","message_contents":%s}' % data - ) - time.sleep(1) - print() - - # shutdown - __database__.publish("p2p_data_request", "stop_process") - print() - - -def test_ip_info_changed(): - # TODO: wait until __database__.setInfoForIPs is fixed and then test if my module reacts correctly - print( - "Slips makes 5 repeating detections, but module is stupid and shares them all" - ) - set_ip_data("1.2.3.6", {"score": 0.71, "confidence": 0.7}) - set_ip_data("1.2.3.6", {"score": 0.7, "confidence": 0.7}) - set_ip_data("1.2.3.6", {"score": 0.71, "confidence": 0.7}) - set_ip_data("1.2.3.6", {"score": 0.7, "confidence": 0.7}) - set_ip_data("1.2.3.6", {"score": 0.71, "confidence": 0.7}) - time.sleep(1) - - -def test_ip_data_save_to_redis(): - print("Data in slips for ip 1.2.3.4") - print(__database__.get_ip_info("1.2.3.4")) - - print("Update data") - save_ip_report_to_db("1.2.3.4", 1, 0.4, 0.4) - - print("Data in slips for ip 1.2.3.4") - print(__database__.get_ip_info("1.2.3.4")) - - -def test_inputs(): - for test_case_name, test_case in json_data.__dict__.items(): - if test_case_name.startswith("_"): - continue - print() - print("#########################") - print("Running test case:", test_case_name) - print("-------------------------") - __database__.publish("p2p_gopy", f"go_data {test_case}") - # the sleep is not needed, but it makes the log more readable - time.sleep(1) - - print("Tests done.") - - -def get_default_config(): - cfg = configparser.ConfigParser() - cfg.read_file(open("slips.yaml")) - return cfg - - -def make_data(): - # the data is a list of reports from multiple peers. Each report contains information about the remote peer (his IP - # and his credibility), and the data the peer sent. From slips, we know that the data sent contains the IP address - # the peer is reporting (attacker), the score the peer assigned to that ip (how malicious does he find him) and the - # confidence he has in his score evaluation. - pass - - -def slips_listener_test(): - """ - A function to test if the retry queue is working as intended. Needs human interaction (disable network when asked) - Test overview: - - check ip A (will be cached successfully) - - disable network - - check ips B and C (they should be queued) - - check ip from the same network as A (this should load from cache without errors, but not trigger retrying) - - enable network - - check ip from the same network as B (this will run and be cached, and trigger retrying. While retrying is in - progress, it should check ip B and return cached result and then run a new query for C) - :return: None - """ - print("Running slips listener test") - - # invalid command - __database__.publish("p2p_gopy", "foooooooooo") - __database__.publish("p2p_gopy", "") - - # invalid command with parameters - __database__.publish("p2p_gopy", "foooooooooo bar 3") - - # valid command, no parameters - __database__.publish("p2p_gopy", "UPDATE") - - # valid update - __database__.publish("p2p_gopy", "UPDATE ipaddress 1 1") - __database__.publish("p2p_gopy", "UPDATE ipaddress 1.999999999999999 3") - - # update with unparsable parameters - __database__.publish("p2p_gopy", "UPDATE ipaddress 1 five") - __database__.publish("p2p_gopy", "UPDATE ipaddress 3") - - data = make_data() - __database__.publish("p2p_gopy", f"GO_DATA {data}") - - # stop instruction - __database__.publish("p2p_gopy", "stop_process") - - -def test_handle_slips_update(): - print("Slips asks about data for 1.2.3.5") - # slips asks for data about 1.2.3.5 and cache age 1000 - data_to_send = { - "ip": "tst", - "profileid": "profileid_192.168.1.1", - "twid": "timewindow1", - "proto": "TCP", - "ip_state": "dstip", - "stime": time.time(), - "uid": "123", - "cache_age": 1000, - } - data_to_send = json.dumps(data_to_send) - __database__.publish("p2p_data_request", data_to_send) - - time.sleep(1) - - -def test_evaluation_error(): - __database__.publish( - "p2p_gopy", f"go_data {json_data.wrong_message_eval_structure}" - ) - # __database__.publish("p2p_gopy", "go_data " + json_data.wrong_message_type) - - -def test_pigeon(): - # one pigeon is already running at port 6669, we start a second one on 6670 - init_tests(6670) - - # one of the peers makes a detection about IP 1.2.3.4 - # (both peers can read the same data from db, but only one is notified about it, so the other doesn't check it) - __database__.r.hset( - "IPsInfo", "1.2.3.4", '{"score":0.5, "confidence":0.8}' - ) - __database__.publish("ip_info_change6669", "1.2.3.4") - - # peer 6669 should read the database, then notify the other peer. - # The other peer should save the data in the reports table - - -def test_trustdb(): - trustdb = TrustDB(f"{data_dir}trustdb.db6660", None) - print(trustdb.get_opinion_on_ip("1.1.1.3")) - - -if __name__ == "__main__": - t = time.time() - # test_trustdb() - test_pigeon() - - # init_tests() - - # test_evaluation_error() - - # test_ip_info_changed() - # test_inputs() - # test_slips_integration() - # test_ip_data_save_to_redis() - # test_handle_slips_update() - # test_pigeon() - - print(time.time() - t) - time.sleep(10000000) diff --git a/modules/riskiq/__init__.py b/modules/risk_iq/__init__.py similarity index 100% rename from modules/riskiq/__init__.py rename to modules/risk_iq/__init__.py diff --git a/modules/riskiq/riskiq.py b/modules/risk_iq/risk_iq.py similarity index 99% rename from modules/riskiq/riskiq.py rename to modules/risk_iq/risk_iq.py index 4e4cf1532c..58656cd917 100644 --- a/modules/riskiq/riskiq.py +++ b/modules/risk_iq/risk_iq.py @@ -12,7 +12,7 @@ class RiskIQ(IModule): # Name: short name of the module. Do not use spaces - name = "Risk IQ" + name = "risk_iq" description = "Module to get passive DNS info about IPs from RiskIQ" authors = ["Alya Gomaa"] diff --git a/modules/rnn_cc_detection/rnn_cc_detection.py b/modules/rnn_cc_detection/rnn_cc_detection.py index d4d16929a0..873cc8dbff 100644 --- a/modules/rnn_cc_detection/rnn_cc_detection.py +++ b/modules/rnn_cc_detection/rnn_cc_detection.py @@ -33,7 +33,7 @@ class CCDetection(IModule): # Name: short name of the module. Do not use spaces - name = "RNN C&C Detection" + name = "rnn_cc_detection" description = "Detect C&C channels based on behavioral letters" authors = ["Sebastian Garcia", "Kamila Babayeva", "Ondrej Lukas"] diff --git a/modules/template/template.py b/modules/template/template.py index 43115c7f78..97c26f3a30 100644 --- a/modules/template/template.py +++ b/modules/template/template.py @@ -20,7 +20,7 @@ class Template(IModule): # Name: short name of the module. Do not use spaces - name = "Template" + name = "template" description = "Template module" authors = ["Template Author"] diff --git a/modules/threat_intelligence/circl_lu.py b/modules/threat_intelligence/circl_lu.py index 4b80d38912..0635f36834 100644 --- a/modules/threat_intelligence/circl_lu.py +++ b/modules/threat_intelligence/circl_lu.py @@ -6,7 +6,7 @@ class Circllu: - name = "Circl.lu" + name = "circl_lu" description = "Circl.lu lookups of IPs" authors = ["Alya Gomaa"] diff --git a/modules/threat_intelligence/spamhaus.py b/modules/threat_intelligence/spamhaus.py index 11e6e35702..296665d9b1 100644 --- a/modules/threat_intelligence/spamhaus.py +++ b/modules/threat_intelligence/spamhaus.py @@ -12,7 +12,7 @@ class Spamhaus: - name = "Spamhaus" + name = "spamhaus" description = "Spamhaus lookups of IPs" authors = ["Alya Gomaa"] diff --git a/modules/threat_intelligence/threat_intelligence.py b/modules/threat_intelligence/threat_intelligence.py index 007442f2f5..507ef315e3 100644 --- a/modules/threat_intelligence/threat_intelligence.py +++ b/modules/threat_intelligence/threat_intelligence.py @@ -37,7 +37,7 @@ class ThreatIntel(IModule, URLhaus, Spamhaus): - name = "Threat Intelligence" + name = "threat_intelligence" description = ( "Check if the source IP or destination IP" " are in a malicious list of IPs" diff --git a/modules/threat_intelligence/urlhaus.py b/modules/threat_intelligence/urlhaus.py index 31a12b2b69..893b331a1e 100644 --- a/modules/threat_intelligence/urlhaus.py +++ b/modules/threat_intelligence/urlhaus.py @@ -20,7 +20,7 @@ class URLhaus: - name = "URLhaus" + name = "urlhaus" description = "URLhaus lookups of URLs and hashes" authors = ["Alya Gomaa"] diff --git a/modules/timeline/timeline.py b/modules/timeline/timeline.py index 210e44c1df..feae830db3 100644 --- a/modules/timeline/timeline.py +++ b/modules/timeline/timeline.py @@ -17,7 +17,7 @@ class Timeline(IModule): # Name: short name of the module. Do not use spaces - name = "Timeline" + name = "timeline" description = ( "Creates kalipso timeline of what happened in the" " network based on flows and available data" diff --git a/modules/update_manager/update_manager.py b/modules/update_manager/update_manager.py index 1ebe3b2704..0c768dc09b 100644 --- a/modules/update_manager/update_manager.py +++ b/modules/update_manager/update_manager.py @@ -30,7 +30,7 @@ class UpdateManager(IModule): - name = "Update Manager" + name = "update_manager" description = "Update Threat Intelligence files" authors = ["Kamila Babayeva", "Alya Gomaa"] @@ -426,7 +426,7 @@ def should_update(self, file_to_download: str, update_period) -> bool: # update period passed if "risk" in file_to_download: - # updating riskiq TI data does not depend on an e-tag + # updating risk_iq TI data does not depend on an e-tag return True # Update only if the e-tag is different @@ -1729,7 +1729,7 @@ async def update(self) -> bool: ) task.add_done_callback(self.handle_task_exception) ####################################################### - # in case of riskiq files, we don't have a link for them in ti_files, We update these files using their API + # in case of risk_iq files, we don't have a link for them in ti_files, We update these files using their API # check if we have a username and api key and a week has passed since we last updated if self.should_update("riskiq_domains", self.riskiq_update_period): self.update_riskiq_feed() diff --git a/modules/virustotal/virustotal.py b/modules/virustotal/virustotal.py index 794874863a..8b7a539c52 100644 --- a/modules/virustotal/virustotal.py +++ b/modules/virustotal/virustotal.py @@ -17,7 +17,7 @@ class VT(IModule): - name = "Virustotal" + name = "virustotal" description = "IP, domain and file hash lookup on Virustotal" authors = [ "Dita Hollmannova", diff --git a/slips/daemon.py b/slips/daemon.py index 659d983ff5..e037d93b16 100644 --- a/slips/daemon.py +++ b/slips/daemon.py @@ -16,7 +16,7 @@ class Daemon: description = "This module runs when slips is in daemonized mode" - name = "Daemon" + name = "daemon" def __init__(self, slips): # to use read_configurations defined in Main @@ -73,7 +73,7 @@ def prepare_std_streams(self, output_dir): def read_configuration(self): conf = ConfigParser() - self.logsfile = conf.logsfile() + self.logsfile = conf.logs_file() self.stdout = conf.stdout() self.stderr = conf.stderr() # we don't use it anyway diff --git a/slips/main.py b/slips/main.py index 4bd2376958..072b7794cc 100644 --- a/slips/main.py +++ b/slips/main.py @@ -39,7 +39,7 @@ class Main: def __init__(self, testing=False): - self.name = "Main" + self.name = "main" self.alerts_default_path = "output/" self.mode = "interactive" self.sigterm_received = False diff --git a/slips_files/common/abstracts/iasync_module.py b/slips_files/common/abstracts/iasync_module.py index e41f3f6b44..46df07c587 100644 --- a/slips_files/common/abstracts/iasync_module.py +++ b/slips_files/common/abstracts/iasync_module.py @@ -15,11 +15,11 @@ class AsyncModule(IModule): An abstract class for asynchronous slips modules """ - name = "AsyncModule" + name = "iasync_module" def __init__(self, *args, **kwargs): IModule.__init__(self, *args, **kwargs) - # list of async functions to await before flowalerts shuts down + # list of async functions to await before flow_alerts shuts down self.tasks: List[Task] = [] def init(self, **kwargs): ... diff --git a/slips_files/common/abstracts/iexporter.py b/slips_files/common/abstracts/iexporter.py index 673c918d0e..dcb2457ac3 100644 --- a/slips_files/common/abstracts/iexporter.py +++ b/slips_files/common/abstracts/iexporter.py @@ -5,8 +5,13 @@ warden etc. """ +import os + from abc import ABC, abstractmethod +from slips_files.common.output_paths import ( + get_databases_dir_path_inside_output_dir, +) from slips_files.common.printer import Printer from slips_files.core.database.database_manager import DBManager from slips_files.core.output import Output @@ -21,6 +26,30 @@ def __init__(self, logger: Output, db: DBManager, **kwargs): def print(self, *args, **kwargs): return self.printer.print(*args, **kwargs) + def get_output_path( + self, *relative_path_parts: str, module_name: str | None = None + ) -> str: + output_dir = ( + getattr(self.db, "output_dir", None) or self.db.get_output_dir() + ) + if isinstance(output_dir, bytes): + output_dir = output_dir.decode("utf-8") + if not output_dir: + output_dir = "." + module_output_dir = os.path.join(output_dir, module_name or self.name) + os.makedirs(module_output_dir, exist_ok=True) + return os.path.join(module_output_dir, *relative_path_parts) + + def get_database_path(self, filename: str) -> str: + output_dir = ( + getattr(self.db, "output_dir", None) or self.db.get_output_dir() + ) + if isinstance(output_dir, bytes): + output_dir = output_dir.decode("utf-8") + if not output_dir: + output_dir = "." + return get_databases_dir_path_inside_output_dir(output_dir, filename) + @property @abstractmethod def name(self) -> str: diff --git a/slips_files/common/abstracts/iflowalerts_analyzer.py b/slips_files/common/abstracts/iflowalerts_analyzer.py index f644357def..d48f2a9603 100644 --- a/slips_files/common/abstracts/iflowalerts_analyzer.py +++ b/slips_files/common/abstracts/iflowalerts_analyzer.py @@ -2,16 +2,16 @@ # SPDX-License-Identifier: GPL-2.0-only from abc import ABC, abstractmethod -from modules.flowalerts.set_evidence import SetEvidenceHelper +from modules.flow_alerts.set_evidence import SetEvidenceHelper from slips_files.core.database.database_manager import DBManager class IFlowalertsAnalyzer(ABC): """ keep in mind that every class that implements this interface MUST be - registered in flowalerts.py + registered in flow_alerts.py must by started, controlled, and terminated by it. msgs from the - appropriate channels should be passed to that class using flowalerts too. + appropriate channels should be passed to that class using flow_alerts too. """ def __init__(self, db: DBManager, flowalerts=None, **kwargs): @@ -37,7 +37,7 @@ def read_configuration(self): def init(self): """ the goal of this is to have one common __init__() above for all - flowalerts helpers, which is the one in this file, and a different + flow_alerts helpers, which is the one in this file, and a different init() per helper this init will have access to all keyword args passes when initializing the module diff --git a/slips_files/common/abstracts/imodule.py b/slips_files/common/abstracts/imodule.py index 63dc1d322a..2d436c221f 100644 --- a/slips_files/common/abstracts/imodule.py +++ b/slips_files/common/abstracts/imodule.py @@ -1,5 +1,6 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only +import os import sys import traceback import warnings @@ -11,6 +12,7 @@ Optional, ) from slips_files.common.printer import Printer +from slips_files.common.output_paths import get_this_db_path_inside_output_dir from slips_files.core.helpers.bloom_filters_manager import BFManager from slips_files.core.output import Output from slips_files.common.slips_utils import utils @@ -24,7 +26,7 @@ class IModule(ABC, Process): An interface for all slips modules """ - name = "IModule" + name = "imodule" description = "Template module" authors = ["Template Author"] # should be filled with the channels each module subscribes to @@ -44,7 +46,8 @@ def __init__( ): Process.__init__(self) self.redis_port = redis_port - self.output_dir = output_dir + self.parent_output_dir = output_dir + self.output_dir = os.path.join(output_dir, self.name) self.msg_received = False # as parsed by arg_parser, these are the cli args self.args: Namespace = slips_args @@ -58,7 +61,11 @@ def __init__( self.bloom_filters: BFManager = bloom_filters_manager self.printer = Printer(self.logger, self.name) self.db = DBManager( - self.logger, self.output_dir, self.redis_port, self.conf, self.ppid + self.logger, + self.parent_output_dir, + self.redis_port, + self.conf, + self.ppid, ) self.db.client_setname(self.name) self.keyboard_int_ctr = 0 @@ -73,6 +80,25 @@ def __init__( def print(self, *args, **kwargs): return self.printer.print(*args, **kwargs) + def get_module_specific_output_path(self, logfile_name: str) -> str: + """returns the full path of the given file inside this + module-specific output dir inside the parent_output_dir + so if the output dir is slips1, and this module is flow_alerts, + and the filename is detection_log.csv + this functions returns slips1/flow_alerts/detection_log.csv + """ + os.makedirs(self.output_dir, exist_ok=True) + return os.path.join(self.output_dir, logfile_name) + + def get_database_path(self, filename: str) -> str: + """ + this is where any .sqlite database generated by this module + should be stored + """ + return get_this_db_path_inside_output_dir( + self.parent_output_dir, filename + ) + def init_channel_tracker(self) -> Dict[str, Dict[str, bool]]: """ tracks if in the last loop, a msg was received in any of the @@ -198,7 +224,7 @@ def print_traceback(self): def run(self): """ - some modules use async functions like flowalerts, + some modules use async functions like flow_alerts, the goals of this function is to make sure that async and normal shutdown_gracefully() functions run until completion """ diff --git a/slips_files/common/idmefv2.py b/slips_files/common/idmefv2.py index aa74aa3e30..c946a69363 100644 --- a/slips_files/common/idmefv2.py +++ b/slips_files/common/idmefv2.py @@ -45,7 +45,7 @@ class IDMEFv2: https://www.ietf.org/id/draft-lehmann-idmefv2-03.html#name-the-alert-class """ - name = "IDMEFv2" + name = "idmefv2" def __init__(self, logger: Output, db): self.printer = Printer(logger, self.name) diff --git a/slips_files/common/output_paths.py b/slips_files/common/output_paths.py new file mode 100644 index 0000000000..1e00597d0a --- /dev/null +++ b/slips_files/common/output_paths.py @@ -0,0 +1,73 @@ +import os +from pathlib import Path + +from slips_files.common.parsers.config_parser import ConfigParser + +DATABASES_DIRNAME = "databases" +ALERTS_DIRNAME = "alerts" +REDIS_DIRNAME = "redis" + + +def get_databases_dir_path_inside_output_dir(parent_dir: str) -> str: + """ + any db generated by slips and should be overwritten per run, + goes in this dir + return the databses/ path inside the parent output dir + e.g output_123/databses/ + :param parent_dir: parent output dir. the main one given to slips with -o + """ + databases_dir = os.path.join(parent_dir or ".", DATABASES_DIRNAME) + os.makedirs(databases_dir, exist_ok=True) + return databases_dir + + +def get_redis_dir_path_inside_output_dir(parent_dir: str) -> str: + """ + any redis logs generated by slips and should be overwritten per run, + goes in this dir + :param parent_dir: parent output dir. the main one given to slips with -o + """ + redis_dir = os.path.join(parent_dir or ".", REDIS_DIRNAME) + os.makedirs(redis_dir, exist_ok=True) + return redis_dir + + +def get_this_db_path_inside_output_dir(parent_dir: str, filename: str) -> str: + return os.path.join( + get_databases_dir_path_inside_output_dir(parent_dir), filename + ) + + +def get_permanent_dir() -> str: + """ + Return the root directory for persistent runtime data. + + Returns: + Relative path where runtime-generated data that must persist + across different Slips runs is stored. + """ + conf = ConfigParser() + permanent_dir = conf.permanent_dir() + + # make sure it's inside slips root dir + permanent_dir = os.path.join(os.getcwd(), permanent_dir) + Path(permanent_dir).mkdir(parents=True, exist_ok=True) + + return permanent_dir + + +def get_this_filepath_inside_permanent_dir(filename: str) -> str: + """use this func if you plan to persist your filename across runs.""" + return os.path.join(get_permanent_dir(), filename) + + +def get_alerts_path_inside_output_dir(parent_dir: str) -> str: + return os.path.join(parent_dir, ALERTS_DIRNAME) + + +def get_redis_logs_path_inside_output_dir( + parent_dir: str, logfile: str +) -> str: + return os.path.join( + get_redis_dir_path_inside_output_dir(parent_dir), logfile + ) diff --git a/slips_files/common/parsers/config_parser.py b/slips_files/common/parsers/config_parser.py index 60cdf63d09..8ee2768059 100644 --- a/slips_files/common/parsers/config_parser.py +++ b/slips_files/common/parsers/config_parser.py @@ -1,6 +1,7 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only from datetime import timedelta +import os import sys from slips_files.common.input_type import InputType import ipaddress @@ -74,6 +75,26 @@ def read_configuration(self, section, name, default_value): # or no section or no configuration file specified return default_value + def read_module_configuration( + self, section: str, legacy_section: str, name: str, default_value + ): + """ + Read a module configuration value with support for a legacy section. + + Parameters: + section: Preferred module section name. + legacy_section: Backward-compatible section name. + name: Configuration key to read. + default_value: Value returned when neither section defines the key. + + Returns: + The configured value from the preferred or legacy section. + """ + value = self.read_configuration(section, name, default_value) + if value != default_value: + return value + return self.read_configuration(legacy_section, name, default_value) + @property def web_interface_port(self) -> int: port = self.read_configuration("web_interface", "port", 55000) @@ -86,8 +107,8 @@ def get_entropy_threshold(self): """ gets the shannon entropy used in detecting C&C over DNS TXT records from slips.conf/slips.yaml """ - threshold = self.read_configuration( - "flowalerts", "entropy_threshold", 5 + threshold = self.read_module_configuration( + "flow_alerts", "flowalerts", "entropy_threshold", 5 ) try: @@ -96,8 +117,8 @@ def get_entropy_threshold(self): return 5 def get_pastebin_download_threshold(self): - threshold = self.read_configuration( - "flowalerts", "pastebin_download_threshold", 700 + threshold = self.read_module_configuration( + "flow_alerts", "flowalerts", "pastebin_download_threshold", 700 ) try: @@ -186,14 +207,26 @@ def enable_local_whitelist(self): "whitelists", "enable_local_whitelist", True ) - def logsfile(self): - return self.read_configuration("modes", "logsfile", "slips.log") + def logs_file(self): + return self.read_configuration( + "output", + "logs", + self.read_configuration("modes", "logs", "slips.log"), + ) def stdout(self): - return self.read_configuration("modes", "stdout", "slips.log") + return self.read_configuration( + "output", + "stdout", + self.read_configuration("modes", "stdout", "slips.log"), + ) def stderr(self): - return self.read_configuration("modes", "stderr", "errors.log") + return self.read_configuration( + "output", + "stderr", + self.read_configuration("modes", "stderr", "errors.log"), + ) def create_p2p_logfile(self): return self.read_configuration( @@ -260,6 +293,21 @@ def get_tw_width(self) -> str: def enable_metadata(self): return self.read_configuration("parameters", "metadata_dir", False) + def permanent_dir(self) -> str: + """ + Return the directory used for persistent runtime data. + + Returns: + Relative path where databases and runtime-generated files that + must persist across different Slips runs are stored. + """ + permanent_dir = self.read_configuration( + "parameters", "permanent_dir", "permanent" + ) + if not permanent_dir: + permanent_dir = "permanent" + return os.path.normpath(permanent_dir) + def use_local_p2p(self): return self.read_configuration("local_p2p", "use_p2p", False) @@ -267,10 +315,10 @@ def use_global_p2p(self): return self.read_configuration("global_p2p", "use_global_p2p", False) def cesnet_conf_file(self): - return self.read_configuration("CESNET", "configuration_file", False) + return self.read_configuration("cesnet", "configuration_file", False) def poll_delay(self): - poll_delay = self.read_configuration("CESNET", "receive_delay", 86400) + poll_delay = self.read_configuration("cesnet", "receive_delay", 86400) try: poll_delay = int(poll_delay) except ValueError: @@ -280,10 +328,10 @@ def poll_delay(self): return poll_delay def send_to_warden(self): - return self.read_configuration("CESNET", "send_alerts", False) + return self.read_configuration("cesnet", "send_alerts", False) def receive_from_warden(self): - return self.read_configuration("CESNET", "receive_alerts", False) + return self.read_configuration("cesnet", "receive_alerts", False) def verbose(self): verbose = self.read_configuration("parameters", "verbose", 1) @@ -444,8 +492,8 @@ def long_connection_threshold(self): returns threshold in seconds """ # 1500 is in seconds, =25 mins - threshold = self.read_configuration( - "flowalerts", "long_connection_threshold", 1500 + threshold = self.read_module_configuration( + "flow_alerts", "flowalerts", "long_connection_threshold", 1500 ) try: threshold = int(threshold) @@ -457,8 +505,11 @@ def ssh_succesful_detection_threshold(self): """ returns threshold in seconds """ - threshold = self.read_configuration( - "flowalerts", "ssh_succesful_detection_threshold", 4290 + threshold = self.read_module_configuration( + "flow_alerts", + "flowalerts", + "ssh_succesful_detection_threshold", + 4290, ) try: threshold = int(threshold) @@ -467,9 +518,9 @@ def ssh_succesful_detection_threshold(self): return threshold - def ssh_bruteforcing_threshold(self): + def ssh_brute_force_detector_threshold(self): threshold = self.read_configuration( - "bruteforcing", "ssh_attempt_threshold", 9 + "brute_force_detector", "ssh_attempt_threshold", 9 ) try: threshold = int(threshold) @@ -482,8 +533,8 @@ def data_exfiltration_threshold(self): returns threshold in MBs """ # threshold in MBs - threshold = self.read_configuration( - "flowalerts", "data_exfiltration_threshold", 500 + threshold = self.read_module_configuration( + "flow_alerts", "flowalerts", "data_exfiltration_threshold", 500 ) try: threshold = int(threshold) @@ -492,7 +543,9 @@ def data_exfiltration_threshold(self): return threshold def get_ml_mode(self): - return self.read_configuration("flowmldetection", "mode", "test") + return self.read_module_configuration( + "flow_ml_detection", "flowmldetection", "mode", "test" + ) def https_anomaly_training_hours(self) -> int: training_hours = self.read_configuration( @@ -801,11 +854,11 @@ def mac_db_update_period(self): def delete_prev_db(self): return self.read_configuration("parameters", "deletePrevdb", True) - def rotation_period(self): - rotation_period = self.read_configuration( - "parameters", "rotation_period", "1 day" + def default_rotation_interval(self): + default_rotation_interval = self.read_configuration( + "parameters", "default_rotation_interval", "1 day" ) - return utils.sanitize(rotation_period) + return utils.sanitize(default_rotation_interval) def parse_ip(self, ip: str): """converts the given IP address or CIDR to an obj""" @@ -920,12 +973,12 @@ def get_disabled_modules(self, input_type: str) -> list: use_p2p = self.use_local_p2p() if not (use_p2p and "-i" in sys.argv): - to_ignore.append("p2ptrust") + to_ignore.append("p2p_trust") use_global_p2p = self.use_global_p2p() if not (use_global_p2p and ("-i" in sys.argv)): - to_ignore.append("fidesModule") - to_ignore.append("irisModule") + to_ignore.append("fides") + to_ignore.append("iris") # ignore CESNET sharing module if send and receive are # disabled in slips.yaml @@ -1008,7 +1061,7 @@ def get_bootstrapping_setting(self) -> (bool, list): self.read_configuration("global_p2p", "bootstrapping_node", False) and self.read_configuration("global_p2p", "use_global_p2p", False) and ("-i" in sys.argv or "-g" in sys.argv), - ["fidesModule", "irisModule"], + ["fides", "iris"], ) def is_bootstrapping_node(self) -> bool: @@ -1022,5 +1075,5 @@ def get_bootstrapping_modules(self) -> list: return self.read_configuration( "global_p2p", "bootstrapping_modules", - ["fidesModule", "irisModule"], + ["fides", "iris"], ) diff --git a/slips_files/core/aid_manager.py b/slips_files/core/aid_manager.py index 5842c17d46..b06ae5d883 100644 --- a/slips_files/core/aid_manager.py +++ b/slips_files/core/aid_manager.py @@ -24,7 +24,7 @@ def __init__( self._process = Process( target=self._worker_loop, args=(self._aid_queue, self.db), - name="AIDManager", + name="aid_manager", daemon=True, ) utils.start_process(self._process, self.db) diff --git a/slips_files/core/database/database_manager.py b/slips_files/core/database/database_manager.py index bbf6602db3..aa1cff8179 100644 --- a/slips_files/core/database/database_manager.py +++ b/slips_files/core/database/database_manager.py @@ -20,6 +20,9 @@ from slips_files.core.database.redis_db.database import RedisDB from slips_files.core.database.sqlite_db.database import SQLiteDB from slips_files.common.parsers.config_parser import ConfigParser +from slips_files.common.output_paths import ( + get_this_filepath_inside_permanent_dir, +) from slips_files.common.performance_paths import get_performance_csv_path from slips_files.core.structures.evidence import Evidence from slips_files.core.structures.alerts import Alert @@ -33,7 +36,7 @@ class DBManager: handler in here. """ - name = "DBManager" + name = "db_manager" def __init__( self, @@ -66,9 +69,9 @@ def __init__( if self.conf.use_local_p2p(): # import this on demand because slips light version doesn't # include the P2P dir - from modules.p2ptrust.trust.trustdb import TrustDB + from modules.p2p_trust.trust.trustdb import TrustDB - self.trust_db_path: str = self.init_p2ptrust_db() + self.trust_db_path: str = self.init_p2p_trust_db() self.trust_db = TrustDB( self.logger, self.trust_db_path, @@ -123,12 +126,21 @@ def backup_db(self, db_path: str): f"restart Slips." ) - def init_p2ptrust_db(self) -> str: - """Initializes and returns the path to a valid trustdb inside p2ptrust_runtime_dir.""" - p2ptrust_runtime_dir = os.path.join(os.getcwd(), "p2ptrust_runtime/") - Path(p2ptrust_runtime_dir).mkdir(parents=True, exist_ok=True) - db_path = os.path.join(p2ptrust_runtime_dir, "trustdb.db") - self.p2ptrust_runtime_dir = p2ptrust_runtime_dir + def init_p2p_trust_db(self) -> str: + """ + Initialize and return the path to the persistent local P2P trust DB. + + Returns: + Path to the local P2P trust SQLite database. + """ + p2p_trust_runtime_dir = get_this_filepath_inside_permanent_dir( + "p2p_trust_runtime" + ) + + Path(p2p_trust_runtime_dir).mkdir(parents=True, exist_ok=True) + + db_path = os.path.join(p2p_trust_runtime_dir, "trustdb.db") + self.p2p_trust_runtime_dir = p2p_trust_runtime_dir if os.path.exists(db_path): if self.is_db_malformed(db_path): @@ -181,10 +193,10 @@ def has_write_access_to_sqlite(self, db_path: str) -> bool: ): return False - def get_p2ptrust_dir(self) -> str: - return self.p2ptrust_runtime_dir + def get_p2p_trust_dir(self) -> str: + return self.p2p_trust_runtime_dir - def get_p2ptrust_db_path(self) -> str: + def get_p2p_trust_db_path(self) -> str: return self.trust_db_path def print(self, *args, **kwargs): diff --git a/slips_files/core/database/redis_db/alert_handler.py b/slips_files/core/database/redis_db/alert_handler.py index d399867493..bca7e43b11 100644 --- a/slips_files/core/database/redis_db/alert_handler.py +++ b/slips_files/core/database/redis_db/alert_handler.py @@ -50,7 +50,7 @@ class AlertHandler: set_profileid_field: Callable[..., Any] add_profile: Callable[..., Any] - name = "DB" + name = "alert_handler_db" def set_evidence_causing_alert(self, alert: Alert): """ diff --git a/slips_files/core/database/redis_db/cleanup_mixin.py b/slips_files/core/database/redis_db/cleanup_mixin.py index 31285c7764..55a2da2c51 100644 --- a/slips_files/core/database/redis_db/cleanup_mixin.py +++ b/slips_files/core/database/redis_db/cleanup_mixin.py @@ -16,7 +16,7 @@ class CleanupMixin: get_blocked_timewindows_of_profile: Callable[..., Any] print: Callable[..., Any] - name = "CleanupMixin" + name = "cleanup_mixin" def _del_all_profile_tw_keys(self, profileid: str, twid: str, pipe): """ diff --git a/slips_files/core/database/redis_db/database.py b/slips_files/core/database/redis_db/database.py index f31fff37c5..f8d68dd1a3 100644 --- a/slips_files/core/database/redis_db/database.py +++ b/slips_files/core/database/redis_db/database.py @@ -3,6 +3,9 @@ import shutil import socket +from slips_files.common.output_paths import ( + get_redis_logs_path_inside_output_dir, +) from slips_files.common.printer import Printer from slips_files.common.slips_utils import utils from slips_files.common.parsers.config_parser import ConfigParser @@ -194,12 +197,14 @@ def __init__(self, *args, **kwargs): @classmethod def _get_conf_file_path(cls, redis_port: Optional[int] = None) -> str: - """Return a per-run redis config path to avoid parallel overwrite.""" + """ + Return a per-run redis config path to avoid parallel overwrite. + """ redis_port = redis_port or cls.redis_port output_dir = os.fspath(cls.output_dir or "output") os.makedirs(output_dir, exist_ok=True) - return os.path.join( - output_dir, f"redis-server-port-{redis_port}-{os.getpid()}.conf" + return get_redis_logs_path_inside_output_dir( + cls.output_dir, f"redis-server-port-{redis_port}.conf" ) def _init_ttls(self): @@ -256,7 +261,7 @@ def _setup_config_file(cls): # because slips may use different redis ports at the same time, # logs should be port specific - logfile = os.path.join( + logfile = get_redis_logs_path_inside_output_dir( cls.output_dir, f"redis-server-port-{cls.redis_port}.log" ) cls._options.update({"logfile": logfile}) diff --git a/slips_files/core/database/redis_db/flow_tracker_db.py b/slips_files/core/database/redis_db/flow_tracker_db.py index d2fa4372b2..6fd3ede145 100644 --- a/slips_files/core/database/redis_db/flow_tracker_db.py +++ b/slips_files/core/database/redis_db/flow_tracker_db.py @@ -13,7 +13,7 @@ class FlowTracker: r: Any constants: Any - name = "FlowTrackerDB" + name = "flow_tracker_db" # channels that recv actual flows, not msgs that we need to pass between # modules. subscribers_of_channels_that_recv_flows = { diff --git a/slips_files/core/database/redis_db/ioc_handler.py b/slips_files/core/database/redis_db/ioc_handler.py index 343bc90dd7..bc1cadecca 100644 --- a/slips_files/core/database/redis_db/ioc_handler.py +++ b/slips_files/core/database/redis_db/ioc_handler.py @@ -39,7 +39,7 @@ class IoCHandler: is_trie_cached: bool twid_width: int - name = "DB" + name = "ioc_handler_db" def setup(self, *args, **kwargs): # used for faster domain lookups diff --git a/slips_files/core/database/redis_db/p2p_handler.py b/slips_files/core/database/redis_db/p2p_handler.py index 77029f9e72..28097b49af 100644 --- a/slips_files/core/database/redis_db/p2p_handler.py +++ b/slips_files/core/database/redis_db/p2p_handler.py @@ -20,7 +20,7 @@ class P2PHandler: extended_ttl: int zadd_but_keep_n_entries: Callable[..., Any] - name = "P2PHandlerDB" + name = "p2p_handler_db" def get_fides_ti(self, target: str): """ diff --git a/slips_files/core/database/redis_db/profile_handler.py b/slips_files/core/database/redis_db/profile_handler.py index e6ac375b8a..4ccdd95de3 100644 --- a/slips_files/core/database/redis_db/profile_handler.py +++ b/slips_files/core/database/redis_db/profile_handler.py @@ -53,7 +53,7 @@ class ProfileHandler: _is_gw_mac: Callable[..., Any] _last_sit: Callable[..., Any] - name = "DB" + name = "profile_handler_db" def set_profileid_field(self, profileid, field, value, pipe=None): """Set a single field in the profileid hash.""" diff --git a/slips_files/core/database/redis_db/scan_detections_db.py b/slips_files/core/database/redis_db/scan_detections_db.py index a1a403bae2..6d9d4c0ac6 100644 --- a/slips_files/core/database/redis_db/scan_detections_db.py +++ b/slips_files/core/database/redis_db/scan_detections_db.py @@ -69,7 +69,7 @@ class ScanDetectionsHandler: get_port_info: Callable[..., Any] mark_profile_tw_as_modified: Callable[..., Any] - name = "DB" + name = "scan_detections_handler_db" def setup(self, *args, **kwargs): self.use_local_p2p: bool = self.conf.use_local_p2p() diff --git a/slips_files/core/database/sqlite_db/database.py b/slips_files/core/database/sqlite_db/database.py index 3638acd8cd..db502b2967 100644 --- a/slips_files/core/database/sqlite_db/database.py +++ b/slips_files/core/database/sqlite_db/database.py @@ -9,6 +9,7 @@ from dataclasses import asdict from slips_files.common.abstracts.isqlite import ISQLite +from slips_files.common.output_paths import get_this_db_path_inside_output_dir from slips_files.common.printer import Printer from slips_files.common.slips_utils import utils from slips_files.core.structures.alerts import Alert @@ -21,11 +22,13 @@ class SQLiteDB(ISQLite): Creates a new db and connects to it if there's none in the given output_dir """ - name = "SQLiteDB" + name = "sqlite_db" def __init__(self, logger: Output, output_dir: str, main_pid: int): self.printer = Printer(logger, self.name) - self._flows_db = os.path.join(output_dir, "flows.sqlite") + self._flows_db = get_this_db_path_inside_output_dir( + output_dir, "flows.sqlite" + ) db_newly_created = False if not os.path.exists(self._flows_db): diff --git a/slips_files/core/evidence_handler.py b/slips_files/core/evidence_handler.py index 44003610ab..40e8d9cf0c 100644 --- a/slips_files/core/evidence_handler.py +++ b/slips_files/core/evidence_handler.py @@ -26,6 +26,8 @@ import time from multiprocessing import Process + +from slips_files.common.output_paths import get_alerts_path_inside_output_dir from slips_files.common.style import ( green, ) @@ -41,7 +43,7 @@ # Evidence Process class EvidenceHandler(ICore): - name = "EvidenceHandler" + name = "evidence_handler" def init(self): self.read_configuration() @@ -63,7 +65,9 @@ def init(self): self.evidence_logger = EvidenceLogger( logger_stop_signal=self.logger_stop_signal, evidence_logger_q=self.evidence_logger_q, - output_dir=self.output_dir, + output_dir=get_alerts_path_inside_output_dir( + self.parent_output_dir + ), ) self.logger_thread = threading.Thread( target=self.evidence_logger.run_logger_thread, @@ -121,10 +125,10 @@ def stop_evidence_workers(self): pass def start_evidence_worker(self, worker_id: int = None): - worker_name = f"EvidenceHandlerWorker_Process_{worker_id}" + worker_name = f"evidence_handler_worker_process_{worker_id}" worker = EvidenceHandlerWorker( logger=self.logger, - output_dir=self.output_dir, + output_dir=self.parent_output_dir, redis_port=self.redis_port, termination_event=self.termination_event, conf=self.conf, diff --git a/slips_files/core/evidence_handler_worker.py b/slips_files/core/evidence_handler_worker.py index 5ffa395b40..83b6daf52e 100644 --- a/slips_files/core/evidence_handler_worker.py +++ b/slips_files/core/evidence_handler_worker.py @@ -32,7 +32,7 @@ class EvidenceHandlerWorker(IModule): - name = "EvidenceHandlerWorker" + name = "evidence_handler_worker" def init( self, @@ -477,7 +477,7 @@ def handle_new_blame_message(self, msg: dict): try: data = json.loads(data) except json.decoder.JSONDecodeError: - self.print("Error in the report received from p2ptrust module") + self.print("Error in the report received from p2p_trust module") return key = data["key"] diff --git a/slips_files/core/helpers/filemonitor.py b/slips_files/core/helpers/filemonitor.py index 3b4130cb6d..2a54b59a3d 100644 --- a/slips_files/core/helpers/filemonitor.py +++ b/slips_files/core/helpers/filemonitor.py @@ -44,7 +44,7 @@ def on_moved(self, event): """ this will be triggered everytime zeek renames all log files """ - # tell inputProcess to change open handles + # tell input.py to change open handles if event.dest_path != "True": to_send = {"old_file": event.dest_path, "new_file": event.src_path} to_send = json.dumps(to_send) diff --git a/slips_files/core/helpers/symbols_handler.py b/slips_files/core/helpers/symbols_handler.py index 81ccca208f..9235343c6f 100644 --- a/slips_files/core/helpers/symbols_handler.py +++ b/slips_files/core/helpers/symbols_handler.py @@ -8,7 +8,7 @@ class SymbolHandler: - name = "SymbolHandler" + name = "symbol_handler" def __init__(self, logger: Output, db): self.printer = Printer(logger, self.name) diff --git a/slips_files/core/helpers/whitelist/whitelist.py b/slips_files/core/helpers/whitelist/whitelist.py index 7d80c8b6cf..b5579990da 100644 --- a/slips_files/core/helpers/whitelist/whitelist.py +++ b/slips_files/core/helpers/whitelist/whitelist.py @@ -29,7 +29,7 @@ class Whitelist: - name = "Whitelist" + name = "whitelist" def __init__(self, logger: Output, db, bloom_filter_manager: BFManager): self.printer = Printer(logger, self.name) diff --git a/slips_files/core/input/input.py b/slips_files/core/input/input.py index 7d2589326e..de594dc0ec 100644 --- a/slips_files/core/input/input.py +++ b/slips_files/core/input/input.py @@ -46,7 +46,7 @@ class Input(ICore): """A class process to run the process of the flows""" - name = "Input" + name = "input" def init( self, @@ -177,7 +177,7 @@ def read_configuration(self): self.packet_filter = self.packet_filter or conf.packet_filter() self.tcp_inactivity_timeout = conf.tcp_inactivity_timeout() self.enable_rotation = conf.rotation() - self.rotation_period = conf.rotation_period() + self.default_rotation_interval = conf.default_rotation_interval() self.keep_rotated_files_for = conf.keep_rotated_files_for() def stop_queues(self): diff --git a/slips_files/core/input/zeek/utils/zeek_file_remover.py b/slips_files/core/input/zeek/utils/zeek_file_remover.py index 3a65cddaf0..41f7a31dac 100644 --- a/slips_files/core/input/zeek/utils/zeek_file_remover.py +++ b/slips_files/core/input/zeek/utils/zeek_file_remover.py @@ -1,17 +1,20 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only -import datetime import json import os import threading +import time -from slips_files.common.slips_utils import utils from slips_files.core.supported_logfiles import SUPPORTED_LOGFILES class ZeekFileRemover: def __init__(self, input_process, zeek_utils): + """ + Handles deleting rotated zeek files. + :param input_process: Input.py instance that owns the channels. + """ self.input = input_process self.zeek_utils = zeek_utils self.thread = threading.Thread( @@ -22,6 +25,11 @@ def __init__(self, input_process, zeek_utils): self._started = False def start(self): + """ + Start the remover thread once and ensure the rotation channel exists. + + :return: None + """ if self._started: return self._started = True @@ -30,12 +38,46 @@ def start(self): self.thread.start() def shutdown_gracefully(self): + """ + Wait briefly for the remover thread to exit. + + :return: True + """ try: self.thread.join(3) except Exception: pass return True + def process_rotation_message(self, changed_files: dict): + """ + Close any stale handle for a rotated Zeek file and schedule cleanup. + + :param changed_files: a dict with old_file and new_file paths. + """ + # for example the old log file should be ./zeek_files/dns.2022-05-11-14-43-20.log + # new log file should be dns.log without the ts + old_log_file = changed_files["old_file"] + new_log_file = changed_files["new_file"] + new_logfile_without_path = new_log_file.split("/")[-1].split(".")[0] + + # ignored files have no open handle, so we should only delete them from disk + if new_logfile_without_path not in SUPPORTED_LOGFILES: + try: + # just delete the old file + os.remove(old_log_file) + except FileNotFoundError: + pass + return + + self.zeek_utils.close_rotated_file_handle(new_log_file) + # this file was just rotated. + rotated_at = time.time() + # zeek utils decides when to delete it. + self.zeek_utils.schedule_rotated_file_deletion( + old_log_file, rotated_at + ) + def remove_old_zeek_files(self): """ This thread waits for filemonitor.py to tell it that zeek changed the log files, @@ -47,39 +89,4 @@ def remove_old_zeek_files(self): # this channel receives renamed zeek log files, # we can safely delete them and close their handle changed_files = json.loads(msg["data"]) - - # for example the old log file should be ./zeek_files/dns.2022-05-11-14-43-20.log - # new log file should be dns.log without the ts - old_log_file = changed_files["old_file"] - new_log_file = changed_files["new_file"] - new_logfile_without_path = new_log_file.split("/")[-1].split( - "." - )[0] - # ignored files have no open handle, so we should only delete them from disk - if new_logfile_without_path not in SUPPORTED_LOGFILES: - # just delete the old file - os.remove(old_log_file) - continue - - # don't allow inputprocess to access the - # open_file_handlers dict until this thread sleeps again - lock = threading.Lock() - lock.acquire() - try: - # close slips' open handles - self.zeek_utils.open_file_handlers[new_log_file].close() - # delete cached filename - del self.zeek_utils.open_file_handlers[new_log_file] - except KeyError: - # we don't have a handle for that file, - # we probably don't need it in slips - # ex: loaded_scripts.log, stats.log etc.. - pass - # delete the old log file (the one with the ts) - self.zeek_utils.to_be_deleted.append(old_log_file) - self.zeek_utils.time_rotated = float( - utils.convert_ts_format( - datetime.datetime.now(), "unixtimestamp" - ) - ) - lock.release() + self.process_rotation_message(changed_files) diff --git a/slips_files/core/input/zeek/utils/zeek_input_utils.py b/slips_files/core/input/zeek/utils/zeek_input_utils.py index 63ed406537..0eda2d528c 100644 --- a/slips_files/core/input/zeek/utils/zeek_input_utils.py +++ b/slips_files/core/input/zeek/utils/zeek_input_utils.py @@ -8,7 +8,7 @@ import subprocess import threading import time - +from typing import List, Tuple from re import split from slips_files.common.slips_utils import utils @@ -18,12 +18,12 @@ class ZeekInputUtils: def __init__(self, input_process): self.input = input_process - self.open_file_handlers = {} + self.open_file_handles = {} + self.open_file_handlers_lock = threading.RLock() self.cache_lines = {} self.file_time = {} self.last_updated_file_time = None - self.to_be_deleted = [] - self.time_rotated = None + self.rotated_files_to_delete: List[Tuple[str, float]] = [] self.zeek_files = {} self.zeek_threads = [] self.zeek_pids = [] @@ -33,45 +33,71 @@ def check_if_time_to_del_rotated_files(self): After a specific period (keep_rotated_files_for), slips deletes all rotated files Check if it's time to do so """ - if not self.time_rotated: + if not self.rotated_files_to_delete: return False - now = float( - utils.convert_ts_format(datetime.datetime.now(), "unixtimestamp") - ) - time_to_delete = ( - now >= self.time_rotated + self.input.keep_rotated_files_for - ) - if time_to_delete: - # getting here means that the rotated - # files are kept enough ( keep_rotated_files_for seconds) - # and it's time to delete them - for file in self.to_be_deleted: - try: - os.remove(file) - except FileNotFoundError: - pass - self.to_be_deleted = [] - - def get_file_handle(self, filename): - # Update which files we know about - try: - # We already opened this file - file_handler = self.open_file_handlers[filename] - except KeyError: - # First time opening this file. + now = time.time() + while self.rotated_files_to_delete: + file, delete_after = self.rotated_files_to_delete.pop() + if now < delete_after: + # not time to del it yet + break + + try: + os.remove(file) + self.input.print( + f"Done deleting rotated zeek file:" f" {file}.", + log_to_logfiles_only=True, + ) + except FileNotFoundError: + pass + + def schedule_rotated_file_deletion( + self, file_path: str, rotated_at: float = None + ): + """ + Schedule a rotated Zeek logfile for deletion after the configured delay. + + :param file_path: Full path to the rotated logfile. + :param rotated_at: Rotation timestamp as a unix timestamp. + :return: None + """ + if rotated_at is None: + rotated_at = time.time() + + delete_after = rotated_at + self.input.keep_rotated_files_for + self.rotated_files_to_delete.append((file_path, delete_after)) + + def close_rotated_file_handle(self, filename: str): + """ + closes the given file's handle and removes it from + self.open_file_handlers + + :param filename: Full path to the active logfile name. + """ + with self.open_file_handlers_lock: + file_handler = self.open_file_handles.pop(filename, None) + + if file_handler is not None: + file_handler.close() + + def get_file_handle(self, filename: str): + with self.open_file_handlers_lock: + file_handle = self.open_file_handles.get(filename) + + if file_handle: + return file_handle + try: - file_handler = open(filename, "r") - lock = threading.Lock() - lock.acquire() - self.open_file_handlers[filename] = file_handler - lock.release() + # First time opening this file. + file_handle = open(filename, "r") + self.open_file_handles[filename] = file_handle # now that we replaced the old handle with the newly created file handle # delete the old .log file, that has a timestamp in its name. except FileNotFoundError: # for example dns.log # zeek changes the dns.log file name every 1d, it adds a - # timestamp to it it doesn't create the new dns.log until a + # timestamp to it, it doesn't create the new dns.log until a # new dns request # occurs # if slips tries to read from the old dns.log now it won't @@ -79,7 +105,7 @@ def get_file_handle(self, filename): # created yet simply continue until the new log file is # created and added to the zeek_files list return False - return file_handler + return file_handle def get_ts_from_line(self, zeek_line: str): """ @@ -179,12 +205,16 @@ def reached_timeout(self) -> bool: return False def close_all_handles(self): - # We reach here after the break produced + # We reach here after the break that happens # if no zeek files are being updated. # No more files to read. Close the files - for file, handle in self.open_file_handlers.items(): - self.input.print(f"Closing file {file}", 2, 0) - handle.close() + with self.open_file_handlers_lock: + handles = list(self.open_file_handles.items()) + self.open_file_handles = {} + + for file, handle in handles: + self.input.print(f"Closing file {file}", 2, 0) + handle.close() def get_earliest_line(self): """ @@ -217,7 +247,7 @@ def read_zeek_files(self) -> int: """ try: self.zeek_files = self.input.db.get_all_zeek_files() - self.open_file_handlers = {} + self.open_file_handles = {} # stores zeek_log_file_name: timestamp of the last flow read from # that file self.file_time = {} @@ -323,7 +353,7 @@ def _construct_zeek_cmd(self, pcap_or_interface: str, tcpdump_filter=None): builder = ZeekCommandBuilder( zeek_or_bro=self.input.zeek_or_bro, input_type=self.input.input_type, - rotation_period=self.input.rotation_period, + default_rotation_interval=self.input.default_rotation_interval, enable_rotation=self.input.enable_rotation, tcp_inactivity_timeout=self.input.tcp_inactivity_timeout, packet_filter=self.input.packet_filter, diff --git a/slips_files/core/output.py b/slips_files/core/output.py index c3e448d7fc..423da0c69f 100644 --- a/slips_files/core/output.py +++ b/slips_files/core/output.py @@ -34,7 +34,7 @@ class Output(IObserver): or logs, it should use The printer that uses this process. """ - name = "Output" + name = "output" slips_logfile_lock = Lock() errors_logfile_lock = Lock() cli_lock = Lock() @@ -43,8 +43,8 @@ def __init__( self, verbose=1, debug=0, - stderr="output/errors.log", - slips_logfile="output/slips.log", + stderr="errors.log", + slips_logfile="slips.log", input_type=False, create_logfiles: bool = True, stdout="", diff --git a/slips_files/core/profiler.py b/slips_files/core/profiler.py index a87b7f23db..e2d6046d8a 100644 --- a/slips_files/core/profiler.py +++ b/slips_files/core/profiler.py @@ -63,7 +63,7 @@ class Profiler(ICore, IObservable): """A class to create the profiles for IPs""" - name = "Profiler" + name = "profiler" def init( self, @@ -213,10 +213,10 @@ def get_msg_from_queue(self, q: multiprocessing.Queue): def start_profiler_worker(self, worker_id: int = None): """starts A profiler worker for faster processing of the flows""" - worker_name = f"ProfilerWorker_Process_{worker_id}" + worker_name = f"profiler_worker_process_{worker_id}" worker = ProfilerWorker( logger=self.logger, - output_dir=self.output_dir, + output_dir=self.parent_output_dir, redis_port=self.redis_port, termination_event=self.termination_event, conf=self.conf, @@ -341,7 +341,7 @@ def _check_if_high_throughput_and_add_workers(self): self.print( f"Warning: High throughput detected. Started " f"additional worker: " - f"ProfilerWorker_{worker_id} to handle the flows." + f"profiler_worker_{worker_id} to handle the flows." ) if self.last_worker_id == self.max_workers - 1: diff --git a/slips_files/core/profiler_worker.py b/slips_files/core/profiler_worker.py index c43cf3f1c5..23d4165428 100644 --- a/slips_files/core/profiler_worker.py +++ b/slips_files/core/profiler_worker.py @@ -33,7 +33,7 @@ class ProfilerWorker(IModule): - name = "ProfilerWorker" + name = "profiler_worker" def init( self, @@ -147,7 +147,7 @@ def _get_slips_start_time(self) -> float: return time.time() def _get_latency_filename_prefix(self) -> str: - if self.name.startswith("ProfilerWorker_Process_"): + if self.name.startswith("profiler_worker_process_"): worker_id = self.name.split("_")[-1] return f"profiler_worker_{worker_id}" return self.name.lower() @@ -555,7 +555,7 @@ def pre_main(self): """ worker_number = self.name.split("_")[-1] self.print( - f"Started Profiler Worker {green(worker_number)} [PID" + f"Started {green('Profiler Worker')} {green(worker_number)} [PID" f" {green(os.getpid())}]" ) diff --git a/slips_files/core/zeek_cmd_builder.py b/slips_files/core/zeek_cmd_builder.py index 39d52c2171..4ffd6cd4d0 100644 --- a/slips_files/core/zeek_cmd_builder.py +++ b/slips_files/core/zeek_cmd_builder.py @@ -13,14 +13,15 @@ def __init__( self, zeek_or_bro: str, input_type: InputType, - rotation_period: str, + default_rotation_interval: str, enable_rotation: bool, tcp_inactivity_timeout: int, packet_filter: Optional[str] = None, ): self.zeek_or_bro = zeek_or_bro self.input_type = input_type - self.rotation_period = rotation_period + # this represents the zeek default_rotation_interval parameter + self.default_rotation_interval = default_rotation_interval self.enable_rotation = enable_rotation self.tcp_inactivity_timeout = tcp_inactivity_timeout self.packet_filter = packet_filter @@ -40,11 +41,11 @@ def _get_input_parameter(self, pcap_or_interface: str) -> List[str]: def _get_rotation_args(self) -> List[str]: # rotation is disabled unless it's an interface if self.input_type == InputType.INTERFACE and self.enable_rotation: - # how often to rotate zeek files? taken from slips.yaml + # default_rotation_interval is how often to rotate zeek files? + # taken from slips.yaml return [ "-e", - f'"redef Log::default_rotation_interval =' - f' {self.rotation_period} ;"', + f"redef Log::default_rotation_interval={self.default_rotation_interval};", ] return [] @@ -106,8 +107,6 @@ def build( f"redef tcp_inactivity_timeout={self.tcp_inactivity_timeout}mins;", *rotation, zeek_scripts_dir, - # putting -f last is best practice *packet_filter, ] - return command diff --git a/tests/common_test_utils.py b/tests/common_test_utils.py index dcc2271e7c..2fc9176734 100644 --- a/tests/common_test_utils.py +++ b/tests/common_test_utils.py @@ -1,11 +1,14 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only +import sqlite3 +from importlib.util import find_spec from pathlib import Path import os import shutil import binascii import subprocess import base64 +import sys from typing import ( Dict, Optional, @@ -88,10 +91,21 @@ def do_nothing(*args): def run_slips(cmd): """runs slips and waits for it to end""" slips = subprocess.Popen(cmd, stdin=subprocess.PIPE, shell=True) - return_code = slips.wait() + _, _ = slips.communicate(input=b"y\n") + return_code = slips.returncode return return_code +def get_slips_test_command(arguments): + """ + Build a Slips CLI command that uses the current Python interpreter. + + :param arguments: CLI arguments to pass to slips.py + :return: shell command string + """ + return f"{sys.executable} ./slips.py {arguments}" + + def get_random_uid(): return base64.b64encode(binascii.b2a_hex(os.urandom(9))).decode("utf-8") @@ -126,6 +140,41 @@ def create_output_dir(dirname) -> PosixPath: return path +def skip_if_missing_runtime_dependencies( + python_modules=None, binaries=None, require_zeek_or_bro=False +): + """ + Skip an integration test when required runtime dependencies are missing. + + :param python_modules: iterable of importable module names + :param binaries: iterable of executable names expected on PATH + :param require_zeek_or_bro: require either zeek or bro to exist + :return: None + """ + python_modules = python_modules or () + binaries = binaries or () + missing_dependencies = [] + + for module_name in python_modules: + if find_spec(module_name) is None: + missing_dependencies.append(module_name) + + for binary_name in binaries: + if shutil.which(binary_name) is None: + missing_dependencies.append(binary_name) + + if require_zeek_or_bro and not ( + shutil.which("zeek") or shutil.which("bro") + ): + missing_dependencies.append("zeek|bro") + + if missing_dependencies: + missing = ", ".join(missing_dependencies) + import pytest + + pytest.skip(f"Missing integration runtime dependencies: {missing}") + + def msgs_published_are_eq_msgs_received_by_each_module(db) -> bool: """ This functions checks that all modules received all msgs that were @@ -153,6 +202,22 @@ def check_for_text(txt, output_dir): return False +def get_label_count_from_output_db(output_dir, label): + """ + Return the number of flows stored with a given label in SQLite. + + :param output_dir: integration test output directory + :param label: flow label to count + :return: number of matching flows + """ + db_path = os.path.join(output_dir, "databases", "flows.sqlite") + with sqlite3.connect(db_path) as conn: + row = conn.execute( + "SELECT COUNT(*) FROM flows WHERE label = ?", (label,) + ).fetchone() + return int(row[0] or 0) + + def has_error_keywords(line): """ these keywords indicate that an error needs to diff --git a/tests/integration/config/fides_config.yaml b/tests/integration/config/fides_config.yaml index 52d748df29..734d227b3f 100644 --- a/tests/integration/config/fides_config.yaml +++ b/tests/integration/config/fides_config.yaml @@ -170,7 +170,7 @@ modules: # List of modules to ignore. By default we always ignore the template! do not remove it from the list # Names of other modules that you can disable (they all should be lowercase with no special characters): # threatintelligence, blocking, networkdiscovery, timeline, virustotal, - # rnnccdetection, flowmldetection, updatemanager + # rnnccdetection, flow_ml_detection, updatemanager disable: [template, updatemanager] # For each line in timeline file there is a timestamp. @@ -180,8 +180,8 @@ modules: ############################# -flowmldetection: - # The mode 'train' should be used to tell the flowmldetection module +flow_ml_detection: + # The mode 'train' should be used to tell the flow_ml_detection module # that the flows received are all for training. # A label should be provided in the [Parameters] section # mode : train @@ -206,7 +206,7 @@ virustotal: ############################# threatintelligence: - # by default, slips starts without the TI files, and runs the Update Manager in the background + # by default, slips starts without the TI files, and runs the update_manager in the background # if thi option is set to yes, slips will not start untill the update manager is done # and all TI files are loaded successfully # this is usefull if you want to ensure that slips doesn't miss the detection of any blacklisted IPs @@ -264,7 +264,7 @@ threatintelligence: update_period : 604800 ############################# -flowalerts: +flow_alerts: # we need a thrshold to determine a long connection. in slips by default is. long_connection_threshold : 1500 @@ -323,7 +323,7 @@ exporting_alerts: taxii_password : admin ############################# -CESNET: +cesnet: # Slips supports exporting and importing evidence in the IDEA format to/from warden servers. send_alerts : False diff --git a/tests/integration/config/slips_iris_main.yaml b/tests/integration/config/slips_iris_main.yaml index b68632ca9b..d095cc84b4 100644 --- a/tests/integration/config/slips_iris_main.yaml +++ b/tests/integration/config/slips_iris_main.yaml @@ -1,8 +1,5 @@ -CESNET: - configuration_file: config/warden.conf - receive_alerts: false - receive_delay: 86400 - send_alerts: false +Debug: + generate_performance_plots: false DisabledAlerts: disabled_detections: [] Docker: @@ -12,61 +9,95 @@ Profiling: cpu_profiler_dev_mode_entries: 500000 cpu_profiler_enable: false cpu_profiler_mode: dev - cpu_profiler_multiprocess: false + cpu_profiler_multiprocess: true cpu_profiler_output_limit: 20 cpu_profiler_sampling_interval: 20 memory_profiler_enable: false memory_profiler_mode: live memory_profiler_multiprocess: true +anomaly_detection_https: + adaptation_score_threshold: 2.0 + adwin_clock: 1 + adwin_delta: 0.01 + adwin_grace_period: 5 + adwin_min_window_length: 5 + baseline_alpha: 0.5 + drift_alpha: 0.05 + empirical_threshold_quantile: 0.995 + flow_zscore_threshold: 3.5 + hourly_zscore_threshold: 3.0 + ja3_min_variants_per_server: 3 + log_verbosity: 3 + max_small_flow_anomalies: 1 + min_baseline_points: 6 + suspicious_alpha: 0.005 + training_alpha: 1 + training_fit_method: welford + training_hours: 2 + use_adwin_drift: true +brute_force_detector: + ssh_attempt_threshold: 9 +cesnet: + configuration_file: config/warden.conf + receive_alerts: false + receive_delay: 86400 + send_alerts: false detection: evidence_detection_threshold: 0.25 popup_alerts: false exporting_alerts: TAXII_server: localhost - collection_name: collection-a - discovery_path: /services/discovery-a + collection_name: Alerts + direct_export: true + direct_export_max_workers: 12 + direct_export_retry_backoff: 0.5 + direct_export_retry_max: 0 + direct_export_retry_max_delay: 5.0 + direct_export_workers: 4 + discovery_path: /taxii2/ export_to: [] - inbox_path: /services/inbox-a - jwt_auth_path: /management/auth port: 1234 push_delay: 3600 sensor_name: sensor1 slack_api_path: config/slack_bot_token_secret slack_channel_name: proj_slips_alerting_module - taxii_password: admin + taxii_password: changeme_before_installing_a_medallion_server + taxii_timeout: 10 taxii_username: admin + taxii_version: 2 use_https: false -flowalerts: +flow_alerts: data_exfiltration_threshold: 500 entropy_threshold: 5 long_connection_threshold: 1500 pastebin_download_threshold: 700 ssh_succesful_detection_threshold: 4290 -flowmldetection: +flow_ml_detection: mode: test global_p2p: bootstrapping_modules: - - fidesModule - - irisModule + - fides + - iris bootstrapping_node: false iris_conf: config/iris_config.yaml use_global_p2p: true local_p2p: create_p2p_logfile: false use_p2p: false -modes: - logsfile: slips.log - stderr: errors.log - stdout: slips.log modules: disable: - template - updatemanager timeline_human_timestamp: true +output: + logs: slips.log + stderr: errors.log + stdout: slips.log parameters: analysis_direction: out client_ips: [] debug: 0 + default_rotation_interval: 30sec deletePrevdb: true delete_zeek_files: false export_format: json @@ -76,8 +107,8 @@ parameters: label: normal metadata_dir: true pcapfilter: false + permanent_dir: permanent rotation: true - rotation_period: 1day store_a_copy_of_zeek_files: false store_zeek_files_in_the_output_dir: true tcp_inactivity_timeout: 60 diff --git a/tests/integration/config/test.yaml b/tests/integration/config/test.yaml index bfa905496d..aca544389f 100644 --- a/tests/integration/config/test.yaml +++ b/tests/integration/config/test.yaml @@ -1,4 +1,4 @@ -CESNET: +cesnet: configuration_file: config/warden.conf receive_alerts: false receive_delay: 86400 @@ -37,18 +37,18 @@ exporting_alerts: taxii_password: admin taxii_username: admin use_https: false -flowalerts: +flow_alerts: data_exfiltration_threshold: 500 entropy_threshold: 5 long_connection_threshold: 1500 pastebin_download_threshold: 700 ssh_succesful_detection_threshold: 4290 -flowmldetection: +flow_ml_detection: mode: test global_p2p: bootstrapping_modules: - - fidesModule - - irisModule + - fides + - iris bootstrapping_node: false iris_conf: config/iris_config.yaml use_global_p2p: false @@ -63,8 +63,8 @@ modules: disable: - template - ensembling - - Flow ML Detection - - Update Manager + - flow_ml_detection + - update_manager timeline_human_timestamp: true parameters: analysis_direction: all diff --git a/tests/integration/test.yaml b/tests/integration/test.yaml new file mode 100644 index 0000000000..fb64279b53 --- /dev/null +++ b/tests/integration/test.yaml @@ -0,0 +1,143 @@ +Debug: + generate_performance_plots: false +DisabledAlerts: + disabled_detections: + - ConnectionWithoutDNS +Docker: + GID: 0 + UID: 0 +Profiling: + cpu_profiler_dev_mode_entries: 500000 + cpu_profiler_enable: false + cpu_profiler_mode: dev + cpu_profiler_multiprocess: true + cpu_profiler_output_limit: 20 + cpu_profiler_sampling_interval: 20 + memory_profiler_enable: false + memory_profiler_mode: live + memory_profiler_multiprocess: true +anomaly_detection_https: + adaptation_score_threshold: 2.0 + adwin_clock: 1 + adwin_delta: 0.01 + adwin_grace_period: 5 + adwin_min_window_length: 5 + baseline_alpha: 0.5 + drift_alpha: 0.05 + empirical_threshold_quantile: 0.995 + flow_zscore_threshold: 3.5 + hourly_zscore_threshold: 3.0 + ja3_min_variants_per_server: 3 + log_verbosity: 3 + max_small_flow_anomalies: 1 + min_baseline_points: 6 + suspicious_alpha: 0.005 + training_alpha: 1 + training_fit_method: welford + training_hours: 2 + use_adwin_drift: true +brute_force_detector: + ssh_attempt_threshold: 9 +cesnet: + configuration_file: config/warden.conf + receive_alerts: false + receive_delay: 86400 + send_alerts: false +detection: + evidence_detection_threshold: 0.1 + popup_alerts: false +exporting_alerts: + TAXII_server: localhost + collection_name: Alerts + direct_export: true + direct_export_max_workers: 12 + direct_export_retry_backoff: 0.5 + direct_export_retry_max: 0 + direct_export_retry_max_delay: 5.0 + direct_export_workers: 4 + discovery_path: /taxii2/ + export_to: [] + port: 1234 + push_delay: 3600 + sensor_name: sensor1 + slack_api_path: config/slack_bot_token_secret + slack_channel_name: proj_slips_alerting_module + taxii_password: changeme_before_installing_a_medallion_server + taxii_timeout: 10 + taxii_username: admin + taxii_version: 2 + use_https: false +flow_alerts: + data_exfiltration_threshold: 500 + entropy_threshold: 5 + long_connection_threshold: 1500 + pastebin_download_threshold: 700 + ssh_succesful_detection_threshold: 4290 +flow_ml_detection: + mode: test +global_p2p: + bootstrapping_modules: + - fides + - iris + bootstrapping_node: false + iris_conf: config/iris_config.yaml + use_global_p2p: false +local_p2p: + create_p2p_logfile: false + use_p2p: false +modules: + disable: + - template + - ensembling + - flow_ml_detection + - update_manager + timeline_human_timestamp: true +output: + logs: slips.log + stderr: errors.log + stdout: slips.log +parameters: + analysis_direction: all + client_ips: [] + debug: 0 + default_rotation_interval: 30sec + deletePrevdb: true + delete_zeek_files: true + export_format: json + export_labeled_flows: false + export_strato_letters: false + keep_rotated_files_for: 1 day + label: malicious + metadata_dir: true + pcapfilter: false + permanent_dir: permanent + rotation: true + store_a_copy_of_zeek_files: true + store_zeek_files_in_the_output_dir: false + tcp_inactivity_timeout: 60 + time_window_width: only_one_tw + verbose: 1 + wait_for_modules_to_finish: 10080 mins +threatintelligence: + RiskIQ_credentials_path: config/RiskIQ_credentials + TI_files_update_period: 86400 + download_path_for_remote_threat_intelligence: modules/threat_intelligence/remote_data_files/ + ja3_feeds: config/JA3_feeds.csv + local_threat_intelligence_files: config/local_ti_files/ + mac_db: https://maclookup.app/downloads/json-database/get-db + mac_db_update: 1209600 + riskiq_update_period: 604800 + ssl_feeds: config/SSL_feeds.csv + ti_files: config/TI_feeds.csv + wait_for_TI_to_finish: false +virustotal: + api_key_file: config/vt_api_key + virustotal_update_period: 259200 +web_interface: + port: 55000 +whitelists: + enable_local_whitelist: true + enable_online_whitelist: true + local_whitelist_path: config/whitelist.conf + online_whitelist: https://tranco-list.eu/download/X5QNN/10000 + online_whitelist_update_period: 86400 diff --git a/tests/integration/test2.yaml b/tests/integration/test2.yaml new file mode 100644 index 0000000000..009bb60b1d --- /dev/null +++ b/tests/integration/test2.yaml @@ -0,0 +1,142 @@ +Debug: + generate_performance_plots: false +DisabledAlerts: + disabled_detections: [] +Docker: + GID: 0 + UID: 0 +Profiling: + cpu_profiler_dev_mode_entries: 500000 + cpu_profiler_enable: false + cpu_profiler_mode: dev + cpu_profiler_multiprocess: true + cpu_profiler_output_limit: 20 + cpu_profiler_sampling_interval: 20 + memory_profiler_enable: false + memory_profiler_mode: live + memory_profiler_multiprocess: true +anomaly_detection_https: + adaptation_score_threshold: 2.0 + adwin_clock: 1 + adwin_delta: 0.01 + adwin_grace_period: 5 + adwin_min_window_length: 5 + baseline_alpha: 0.5 + drift_alpha: 0.05 + empirical_threshold_quantile: 0.995 + flow_zscore_threshold: 3.5 + hourly_zscore_threshold: 3.0 + ja3_min_variants_per_server: 3 + log_verbosity: 3 + max_small_flow_anomalies: 1 + min_baseline_points: 6 + suspicious_alpha: 0.005 + training_alpha: 1 + training_fit_method: welford + training_hours: 2 + use_adwin_drift: true +brute_force_detector: + ssh_attempt_threshold: 9 +cesnet: + configuration_file: config/warden.conf + receive_alerts: false + receive_delay: 86400 + send_alerts: false +detection: + evidence_detection_threshold: 0.1 + popup_alerts: false +exporting_alerts: + TAXII_server: localhost + collection_name: Alerts + direct_export: true + direct_export_max_workers: 12 + direct_export_retry_backoff: 0.5 + direct_export_retry_max: 0 + direct_export_retry_max_delay: 5.0 + direct_export_workers: 4 + discovery_path: /taxii2/ + export_to: [] + port: 1234 + push_delay: 3600 + sensor_name: sensor1 + slack_api_path: config/slack_bot_token_secret + slack_channel_name: proj_slips_alerting_module + taxii_password: changeme_before_installing_a_medallion_server + taxii_timeout: 10 + taxii_username: admin + taxii_version: 2 + use_https: false +flow_alerts: + data_exfiltration_threshold: 500 + entropy_threshold: 5 + long_connection_threshold: 1500 + pastebin_download_threshold: 700 + ssh_succesful_detection_threshold: 4290 +flow_ml_detection: + mode: test +global_p2p: + bootstrapping_modules: + - fides + - iris + bootstrapping_node: false + iris_conf: config/iris_config.yaml + use_global_p2p: false +local_p2p: + create_p2p_logfile: false + use_p2p: false +modules: + disable: + - template + - ensembling + - flow_ml_detection + - update_manager + timeline_human_timestamp: true +output: + logs: slips.log + stderr: errors.log + stdout: slips.log +parameters: + analysis_direction: out + client_ips: [] + debug: 0 + default_rotation_interval: 30sec + deletePrevdb: true + delete_zeek_files: false + export_format: json + export_labeled_flows: false + export_strato_letters: false + keep_rotated_files_for: 1 day + label: normal + metadata_dir: false + pcapfilter: false + permanent_dir: permanent + rotation: true + store_a_copy_of_zeek_files: false + store_zeek_files_in_the_output_dir: false + tcp_inactivity_timeout: 60 + time_window_width: 3600 + verbose: 1 + wait_for_modules_to_finish: 10080 mins +threatintelligence: + RiskIQ_credentials_path: config/RiskIQ_credentials + TI_files_update_period: 86400 + download_path_for_remote_threat_intelligence: modules/threat_intelligence/remote_data_files/ + ja3_feeds: config/JA3_feeds.csv + local_threat_intelligence_files: config/local_ti_files/ + mac_db: https://maclookup.app/downloads/json-database/get-db + mac_db_update: 1209600 + riskiq_update_period: 604800 + ssl_feeds: config/SSL_feeds.csv + ti_files: config/TI_feeds.csv + wait_for_TI_to_finish: false +virustotal: + api_key_file: config/vt_api_key + virustotal_update_period: 259200 +web_interface: + port: 55000 +whitelists: + enable_local_whitelist: true + enable_online_whitelist: true + local_whitelist_path: config/whitelist.conf + online_whitelist: https://tranco-list.eu/download/X5QNN/10000 + online_whitelist_update_period: 86400 diff --git a/tests/integration/test_config_files.py b/tests/integration/test_config_files.py index a4f2d08925..6c6b5c055c 100644 --- a/tests/integration/test_config_files.py +++ b/tests/integration/test_config_files.py @@ -12,7 +12,11 @@ create_output_dir, assert_no_errors, check_for_text, + get_label_count_from_output_db, + run_slips, + get_slips_test_command, modify_yaml_config, + skip_if_missing_runtime_dependencies, ) from tests.module_factory import ModuleFactory import pytest @@ -46,6 +50,11 @@ def test_conf_file(pcap_path, expected_profiles, output_dir, redis_port): """ In this test we're using tests/test.conf """ + skip_if_missing_runtime_dependencies( + python_modules=("termcolor",), + binaries=("redis-server",), + require_zeek_or_bro=True, + ) config_file = "tests/integration/test.yaml" modify_yaml_config( output_filename=config_file, @@ -67,26 +76,22 @@ def test_conf_file(pcap_path, expected_profiles, output_dir, redis_port): "disable": [ "template", "ensembling", - "Flow ML Detection", - "Update Manager", + "flow_ml_detection", + "update_manager", ] }, }, ) output_dir = create_output_dir(output_dir) output_file = os.path.join(output_dir, "slips_output.txt") - command = ( - f"./slips.py " - f"-t -e 1 " - f"-f {pcap_path} " - f"-o {output_dir} " - f"-c {config_file} " - f"-P {redis_port} " - f"> {output_file} 2>&1" + command = get_slips_test_command( + f"-t -e 1 -f {pcap_path} -o {output_dir} -c {config_file} " + f"-P {redis_port}" ) + command = f"{command} > {output_file} 2>&1" print("running slips ...") # this function returns when slips is done - os.system(command) + run_slips(command) print("Slip is done, checking for errors in the output dir.") assert_no_errors(output_dir) print("Comparing profiles with expected profiles") @@ -97,7 +102,7 @@ def test_conf_file(pcap_path, expected_profiles, output_dir, redis_port): # expected_profiles is more than 50 because we're using direction = all assert profiles > expected_profiles print("Checking for a random evidence") - log_file = os.path.join(output_dir, alerts_file) + log_file = output_dir / "alerts" / alerts_file # testing disabled_detections param in the configuration file disabled_evidence = "a connection without DNS resolution" @@ -108,8 +113,7 @@ def test_conf_file(pcap_path, expected_profiles, output_dir, redis_port): print("Make sure slips didn't delete zeek files.") # test delete_zeek_files param - zeek_output_dir = database.get_zeek_output_dir()[2:] - assert zeek_output_dir not in os.listdir() + assert "zeek_files_test7-malicious" not in os.listdir() print("Test storing a copy of zeek files.") # test store_a_copy_of_zeek_files assert "zeek_files" in os.listdir(output_dir) @@ -123,11 +127,11 @@ def test_conf_file(pcap_path, expected_profiles, output_dir, redis_port): print("Checking malicious label count") # test label=malicious - assert int(database.get_label_count("malicious")) > 370 + assert get_label_count_from_output_db(output_dir, "malicious") > 370 # test disable - for module in ["template", "Flow ML Detection"]: + for module in ["template", "flow_ml_detection"]: print(f"Checking if {module} is disabled") - assert module in database.get_disabled_modules() + assert check_for_text(module, output_dir) print("Deleting the output directory") shutil.rmtree(output_dir) os.remove(config_file) @@ -148,6 +152,11 @@ def test_conf_file2(pcap_path, expected_profiles, output_dir, redis_port): """ In this test we're using tests/test2.conf """ + skip_if_missing_runtime_dependencies( + python_modules=("termcolor",), + binaries=("redis-server",), + require_zeek_or_bro=True, + ) config_file = "tests/integration/test2.yaml" modify_yaml_config( output_filename=config_file, @@ -162,8 +171,8 @@ def test_conf_file2(pcap_path, expected_profiles, output_dir, redis_port): "disable": [ "template", "ensembling", - "Flow ML Detection", - "Update Manager", + "flow_ml_detection", + "update_manager", ] }, }, @@ -171,17 +180,13 @@ def test_conf_file2(pcap_path, expected_profiles, output_dir, redis_port): output_dir = create_output_dir(output_dir) output_file = os.path.join(output_dir, "slips_output.txt") - command = ( - f"./slips.py " - f"-t -e 1 " - f"-f {pcap_path} " - f"-o {output_dir} " - f"-c {config_file} " - f"-P {redis_port} " - f"> {output_file} 2>&1" + command = get_slips_test_command( + f"-t -e 1 -f {pcap_path} -o {output_dir} -c {config_file} " + f"-P {redis_port}" ) + command = f"{command} > {output_file} 2>&1" print("running slips ...") - os.system(command) + run_slips(command) print("Slip is done, checking for errors in the output dir.") assert_no_errors(output_dir) print("Deleting the output directory") diff --git a/tests/integration/test_dataset.py b/tests/integration/test_dataset.py index 8034d1f5bd..13f46f835f 100644 --- a/tests/integration/test_dataset.py +++ b/tests/integration/test_dataset.py @@ -10,7 +10,8 @@ is_evidence_present, create_output_dir, assert_no_errors, - msgs_published_are_eq_msgs_received_by_each_module, + get_slips_test_command, + skip_if_missing_runtime_dependencies, ) from tests.module_factory import ModuleFactory import pytest @@ -67,29 +68,28 @@ def test_binetflow( output_dir, redis_port, ): + skip_if_missing_runtime_dependencies( + python_modules=("termcolor",), binaries=("redis-server",) + ) output_dir = create_output_dir(output_dir) output_file = os.path.join(output_dir, "slips_output.txt") - command = ( - f"./slips.py -e 1 -t " - f"-o {output_dir} " - f"-P {redis_port} " - f"-f {binetflow_path} " - f"> {output_file} 2>&1" + command = get_slips_test_command( + f"-e 1 -t -o {output_dir} -P {redis_port} -f {binetflow_path}" ) + command = f"{command} > {output_file} 2>&1" # this function returns when slips is done run_slips(command) assert_no_errors(output_dir) - db = ModuleFactory().create_db_manager_obj( + database = ModuleFactory().create_db_manager_obj( redis_port, output_dir=output_dir, start_redis_server=False ) - profiles = db.get_profiles_len() + profiles = database.get_profiles_len() assert profiles > expected_profiles - assert msgs_published_are_eq_msgs_received_by_each_module(db) - log_file = os.path.join(output_dir, alerts_file) + log_file = output_dir / "alerts" / alerts_file assert is_evidence_present(log_file, expected_evidence) is True shutil.rmtree(output_dir) @@ -110,30 +110,29 @@ def test_binetflow( ], ) def test_suricata(suricata_path, output_dir, redis_port, expected_evidence): + skip_if_missing_runtime_dependencies( + python_modules=("termcolor",), binaries=("redis-server",) + ) output_dir = create_output_dir(output_dir) output_file = os.path.join(output_dir, "slips_output.txt") - command = ( - f"./slips.py -e 1 -t " - f"-f {suricata_path} " - f"-o {output_dir} " - f"-P {redis_port} " - f"> {output_file} 2>&1" + command = get_slips_test_command( + f"-e 1 -t -f {suricata_path} -o {output_dir} -P {redis_port}" ) + command = f"{command} > {output_file} 2>&1" # this function returns when slips is done run_slips(command) assert_no_errors(output_dir) - db = ModuleFactory().create_db_manager_obj( + database = ModuleFactory().create_db_manager_obj( redis_port, output_dir=output_dir, start_redis_server=False ) - profiles = db.get_profiles_len() + profiles = database.get_profiles_len() # todo the profiles should be way more than 10, maybe 76, but it varies # each run, we need to sy why assert profiles > 10 - assert msgs_published_are_eq_msgs_received_by_each_module(db) - log_file = os.path.join(output_dir, alerts_file) + log_file = output_dir / "alerts" / alerts_file assert any(is_evidence_present(log_file, ev) for ev in expected_evidence) shutil.rmtree(output_dir) @@ -150,31 +149,27 @@ def test_nfdump(nfdump_path, output_dir, redis_port): checks that slips is reading nfdump no issue, the file is not malicious so there's no evidence that should be present """ + skip_if_missing_runtime_dependencies( + python_modules=("termcolor",), binaries=("redis-server",) + ) output_dir = create_output_dir(output_dir) # expected_evidence = 'Connection to unknown destination port 902/TCP' output_file = os.path.join(output_dir, "slips_output.txt") - command = ( - f"./slips.py -e 1 -t " - f"-f {nfdump_path} " - f"-o {output_dir} " - f"-P {redis_port} " - f"> {output_file} 2>&1" + command = get_slips_test_command( + f"-e 1 -t -f {nfdump_path} -o {output_dir} -P {redis_port}" ) + command = f"{command} > {output_file} 2>&1" # this function returns when slips is done run_slips(command) - db = ModuleFactory().create_db_manager_obj( + database = ModuleFactory().create_db_manager_obj( redis_port, output_dir=output_dir, start_redis_server=False ) - profiles = db.get_profiles_len() + profiles = database.get_profiles_len() assert_no_errors(output_dir) - # make sure slips generated profiles for this file (can't - # put the number of profiles exactly because slips - # doesn't generate a const number of profiles per file) assert profiles > 0 - assert msgs_published_are_eq_msgs_received_by_each_module(db) # log_file = os.path.join(output_dir, alerts_file) # assert is_evidence_present(log_file, expected_evidence) == True diff --git a/tests/integration/test_fides.py b/tests/integration/test_fides.py index 78ca1cc93b..242db07289 100644 --- a/tests/integration/test_fides.py +++ b/tests/integration/test_fides.py @@ -4,12 +4,12 @@ """ import shutil -from pathlib import PosixPath +from pathlib import PosixPath, Path import redis -from modules.fidesModule.model.peer import PeerInfo -from modules.fidesModule.persistence.fides_sqlite_db import FidesSQLiteDB +from modules.fides.model.peer import PeerInfo +from modules.fides.persistence.fides_sqlite_db import FidesSQLiteDB from tests.common_test_utils import ( create_output_dir, assert_no_errors, @@ -21,7 +21,7 @@ import time import sys from unittest.mock import Mock -import modules.fidesModule.model.peer_trust_data as ptd +import modules.fides.model.peer_trust_data as ptd # TODO # from tests.common_test_utils import ( @@ -221,9 +221,9 @@ def test_conf_file2(path, output_dir, redis_port): ) # iris is supposed to be receiving this msg, that last thing fides does # is send a msg to this channel for iris to receive it - assert db.get_msgs_received_at_runtime("Fides")["fides2network"] == "1" - assert db.get_msgs_received_at_runtime("Fides")["new_alert"] == "1" - print(db.get_msgs_received_at_runtime("Fides")) + assert db.get_msgs_received_at_runtime("fides")["fides2network"] == "1" + assert db.get_msgs_received_at_runtime("fides")["new_alert"] == "1" + print(db.get_msgs_received_at_runtime("fides")) print("Deleting the output directory") shutil.rmtree(output_dir, ignore_errors=True) @@ -281,11 +281,12 @@ def test_trust_recommendation_response(path, output_dir, redis_port): "-P", str(redis_port), ] - config_file_path = "modules/fidesModule/config/fides.conf.yml" - config_temp_path = "modules/fidesModule/config/fides.conf.yml.bak" + config_file_path = "modules/fides/config/fides.conf.yml" + config_temp_path = "modules/fides/config/fides.conf.yml.bak" config_line = "database: 'fides_test_database.sqlite'\n" shutil.copy(config_file_path, config_temp_path) - test_db = "fides_test_database.sqlite" + test_db = Path("permanent") / "fides_test_database.sqlite" + test_db.parent.mkdir(parents=True, exist_ok=True) try: # Append the new line to the config @@ -299,7 +300,7 @@ def test_trust_recommendation_response(path, output_dir, redis_port): mock_logger.print_line = Mock() mock_logger.error = Mock() print("Manipulating database") - fdb = FidesSQLiteDB(mock_logger, test_db) + fdb = FidesSQLiteDB(mock_logger, str(test_db)) fdb.store_peer_trust_data( ptd.trust_data_prototype( peer=PeerInfo( @@ -354,7 +355,7 @@ def test_trust_recommendation_response(path, output_dir, redis_port): redis_port, output_dir=output_dir, start_redis_server=False ) - # assert db.get_msgs_received_at_runtime("Fides")["fides2network"] == "1" + # assert db.get_msgs_received_at_runtime("fides")["fides2network"] == "1" print("Checking Fides' data outlets") assert fdb.get_peer_trust_data("peer1").service_history != [] diff --git a/tests/integration/test_iris.py b/tests/integration/test_iris.py index 5ff2c4d542..332cfc2f0e 100644 --- a/tests/integration/test_iris.py +++ b/tests/integration/test_iris.py @@ -87,6 +87,33 @@ def check_strings_in_file(string_list, file_path): return False +def wait_for_file(file_path, timeout_seconds): + """ + Wait until a file exists or the timeout elapses. + + Parameters: + file_path: Path to the expected file. + timeout_seconds: Maximum number of seconds to wait. + + Returns: + True if the file exists before the timeout, otherwise False. + """ + deadline = time.time() + timeout_seconds + while time.time() < deadline: + if os.path.exists(file_path): + return True + time.sleep(1) + return os.path.exists(file_path) + + +def get_default_interface(): + with open("/proc/net/route") as f: + for line in f.readlines()[1:]: + fields = line.strip().split() + if fields[1] == "00000000": # default route + return fields[0] + + @pytest.mark.parametrize( "zeek_dir_path, output_dir, peer_output_dir, redis_port, peer_redis_port", [ @@ -115,6 +142,8 @@ def test_messaging( which extends the standard use case of connecting to such P2P network. """ # Two Slips instances are necessary to be run in this test. + default_interface = get_default_interface() + # Prepare output dir for the main Slips instance. # The logs of both beers will be clearly separated and kept intact. output_dir: PosixPath = create_output_dir(output_dir) @@ -170,13 +199,14 @@ def test_messaging( # to the same file # command for the main Slips instance command = [ + sys.executable, "./slips.py", "-t", "-g", str(zeek_dir_path), # dummy interface required by -g "-i", - "eth0", + default_interface, "-e", "1", "-o", @@ -198,6 +228,9 @@ def test_messaging( countdown(20, "second peer") # get the connection string from the first peer and give it # to the second one so it is reachable + assert wait_for_file( + log_file_first_iris, 30 + ), f"Expected Iris log file was not created: {log_file_first_iris}" with open(log_file_first_iris, "r") as log: for line in log: match = re.search(r"connection string:\s+'(.+)'", line) @@ -230,13 +263,14 @@ def test_messaging( ) # generate a second command for the second peer peer_command = [ + sys.executable, "./slips.py", "-t", "-g", str(zeek_dir_path), # dummy interface required by -g "-i", - "eth0", + default_interface, "-e", "1", "-o", @@ -297,7 +331,7 @@ def test_messaging( print("Deleting the output directories") shutil.rmtree(output_dir) shutil.rmtree(output_dir_peer) - os.remove("modules/irisModule/second.priv") + os.remove("modules/iris/second.priv") modify_yaml_config( input_path="config/iris_config.yaml", output_dir=os.path.dirname(iris_peer_config_file), diff --git a/tests/integration/test_pcap_dataset.py b/tests/integration/test_pcap_dataset.py index c514a0079c..60f380989d 100644 --- a/tests/integration/test_pcap_dataset.py +++ b/tests/integration/test_pcap_dataset.py @@ -45,6 +45,6 @@ def test_pcap( profiles = db.get_profiles_len() assert profiles > expected_profiles - log_file = os.path.join(output_dir, alerts_file) + log_file = output_dir / "alerts" / alerts_file assert is_evidence_present(log_file, expected_evidence) is True shutil.rmtree(output_dir) diff --git a/tests/integration/test_portscans.py b/tests/integration/test_portscans.py index be0da30066..7359d3ba55 100644 --- a/tests/integration/test_portscans.py +++ b/tests/integration/test_portscans.py @@ -9,6 +9,8 @@ is_evidence_present, create_output_dir, assert_no_errors, + get_slips_test_command, + skip_if_missing_runtime_dependencies, ) from tests.module_factory import ModuleFactory @@ -29,6 +31,9 @@ def test_horizontal(path, output_dir, redis_port): """ checks that slips is detecting horizontal ps no issue, """ + skip_if_missing_runtime_dependencies( + python_modules=("termcolor",), binaries=("redis-server",) + ) output_dir = create_output_dir(output_dir) expected_evidence = ( @@ -36,26 +41,21 @@ def test_horizontal(path, output_dir, redis_port): ) output_file = os.path.join(output_dir, "slips_output.txt") - command = ( - f"./slips.py -e 1 -t -f {path} " - f" -o {output_dir} " - f"-P {redis_port} > {output_file} 2>&1" + command = get_slips_test_command( + f"-e 1 -t -f {path} -o {output_dir} -P {redis_port}" ) + command = f"{command} > {output_file} 2>&1" # this function returns when slips is done run_slips(command) + assert_no_errors(output_dir) database = ModuleFactory().create_db_manager_obj( redis_port, output_dir=output_dir, start_redis_server=False ) - - assert_no_errors(output_dir) - # make sure slips generated profiles for this file (can't - # put the number of profiles exactly because slips - # doesn't generate a const number of profiles per file) - profiles: int = database.get_profiles_len() + profiles = database.get_profiles_len() assert profiles > 0 - log_file = os.path.join(output_dir, alerts_file) + log_file = output_dir / "alerts" / alerts_file assert is_evidence_present(log_file, expected_evidence) shutil.rmtree(output_dir) @@ -69,6 +69,9 @@ def test_vertical(path, output_dir, redis_port): """ checks that slips is detecting horizontal ps no issue, """ + skip_if_missing_runtime_dependencies( + python_modules=("termcolor",), binaries=("redis-server",) + ) output_dir = create_output_dir(output_dir) expected_evidence = ( @@ -76,26 +79,22 @@ def test_vertical(path, output_dir, redis_port): ) output_file = os.path.join(output_dir, "slips_output.txt") - command = ( - f"./slips.py -e 1 -t -f {path} " - f" -o {output_dir}" - f" -P {redis_port} > {output_file} 2>&1" + command = get_slips_test_command( + f"-e 1 -t -f {path} -o {output_dir} -P {redis_port}" ) + command = f"{command} > {output_file} 2>&1" # this function returns when slips is done run_slips(command) + assert_no_errors(output_dir) + database = ModuleFactory().create_db_manager_obj( redis_port, output_dir=output_dir, start_redis_server=False ) - assert_no_errors(output_dir) - - # make sure slips generated profiles for this file (can't - # put the number of profiles exactly because slips - # doesn't generate a const number of profiles per file) - profiles: int = database.get_profiles_len() + profiles = database.get_profiles_len() assert profiles > 0 - log_file = os.path.join(output_dir, alerts_file) + log_file = output_dir / "alerts" / alerts_file assert is_evidence_present(log_file, expected_evidence) shutil.rmtree(output_dir) diff --git a/tests/integration/test_zeek_dataset.py b/tests/integration/test_zeek_dataset.py index 3606155c94..ea33a457ef 100644 --- a/tests/integration/test_zeek_dataset.py +++ b/tests/integration/test_zeek_dataset.py @@ -124,7 +124,7 @@ "dataset/test9-mixed-zeek-dir/conn.log", 4, "non-HTTP established connection to port 80. " - "destination IP: 194.132.197.198", # the flows with uid + "destination IP: 194.132.197.198", # the flow with uid # CAwUdr34dVnyOwbUuj should trigger this "test9-conn_log_only/", 6659, @@ -162,6 +162,6 @@ def test_zeek_conn_log( profiles = database.get_profiles_len() assert profiles > expected_profiles - log_file = os.path.join(output_dir, alerts_file) + log_file = output_dir / "alerts" / alerts_file assert is_evidence_present(log_file, expected_evidence) is True shutil.rmtree(output_dir) diff --git a/tests/module_factory.py b/tests/module_factory.py index d006a1a220..30972a7af7 100644 --- a/tests/module_factory.py +++ b/tests/module_factory.py @@ -70,8 +70,10 @@ def create_db_manager_obj( conf.disabled_detections = Mock(return_value=[]) conf.get_tw_width_as_float = Mock(return_value=3600.0) conf.get_tw_width_in_seconds = Mock(return_value=3600) + conf.get_args = Mock(return_value=Mock(killall=False)) conf.client_ips = Mock(return_value=[]) conf.use_local_p2p = Mock(return_value=False) + conf.permanent_dir = Mock(return_value="permanent") conf.width = Mock(return_value=3600) with ( @@ -99,7 +101,7 @@ def create_db_manager_obj( ) db.print = Mock() - db.init_p2ptrust_db = Mock() + db.init_p2p_trust_db = Mock() # for easier access to redis db db.r = db.rdb.r assert db.get_used_redis_port() == port @@ -134,8 +136,18 @@ def create_http_analyzer_obj(self, mock_db): return http_analyzer @patch(MODULE_DB_MANAGER, name="mock_db") - def create_fides_module_obj(self, mock_db): - from modules.fidesModule.fidesModule import FidesModule + def create_fides_obj(self, mock_db): + from modules.fides.fides import FidesModule + + db_path = os.path.join("permanent", "databases", "fides_p2p_db.sqlite") + + def get_permanent_database_path(_filename): + os.makedirs(os.path.dirname(db_path), exist_ok=True) + return db_path + + mock_db.return_value.get_permanent_database_path.side_effect = ( + get_permanent_database_path + ) fm = FidesModule( logger=self.logger, @@ -209,10 +221,10 @@ def create_checker_obj(self): @patch(MODULE_DB_MANAGER, name="mock_db") def create_go_director_obj(self, mock_db): - from modules.p2ptrust.trust.trustdb import TrustDB - from modules.p2ptrust.utils.go_director import GoDirector + from modules.p2p_trust.trust.trustdb import TrustDB + from modules.p2p_trust.utils.go_director import GoDirector - with patch("modules.p2ptrust.utils.utils.send_evaluation_to_go"): + with patch("modules.p2p_trust.utils.utils.send_evaluation_to_go"): go_director = GoDirector( logger=self.logger, trustdb=Mock(spec=TrustDB), @@ -252,7 +264,7 @@ def dummy_acquire_flock(self): @patch("sqlite3.connect") def create_trust_db_obj(self, sqlite_mock): - from modules.p2ptrust.trust.trustdb import TrustDB + from modules.p2p_trust.trust.trustdb import TrustDB with ( patch("slips_files.common.abstracts.isqlite.ISQLite._init_flock"), @@ -274,7 +286,7 @@ def create_trust_db_obj(self, sqlite_mock): @patch(MODULE_DB_MANAGER, name="mock_db") def create_base_model_obj(self, mock_db): - from modules.p2ptrust.trust.base_model import BaseModel + from modules.p2p_trust.trust.base_model import BaseModel from slips_files.core.output import Output logger = Mock(spec=Output) @@ -362,7 +374,7 @@ def create_unblocker_obj(self, mock_db): @patch(MODULE_DB_MANAGER, name="mock_db") def create_flowalerts_obj(self, mock_db): - from modules.flowalerts.flowalerts import FlowAlerts + from modules.flow_alerts.flow_alerts import FlowAlerts flowalerts = FlowAlerts( logger=self.logger, @@ -378,10 +390,12 @@ def create_flowalerts_obj(self, mock_db): return flowalerts @patch(MODULE_DB_MANAGER, name="mock_db") - def create_bruteforcing_obj(self, mock_db): - from modules.bruteforcing.bruteforcing import Bruteforcing + def create_brute_force_detector_obj(self, mock_db): + from modules.brute_force_detector.brute_force_detector import ( + BruteforceDetector, + ) - bruteforcing = Bruteforcing( + brute_force_detector = BruteforceDetector( logger=self.logger, output_dir="dummy_output_dir", redis_port=6379, @@ -391,68 +405,68 @@ def create_bruteforcing_obj(self, mock_db): ppid=Mock(), bloom_filters_manager=Mock(), ) - bruteforcing.print = Mock() - return bruteforcing + brute_force_detector.print = Mock() + return brute_force_detector @patch(DB_MANAGER, name="mock_db") def create_dns_analyzer_obj(self, mock_db): - from modules.flowalerts.dns import DNS + from modules.flow_alerts.dns import DNS flowalerts = self.create_flowalerts_obj() return DNS(flowalerts.db, flowalerts=flowalerts) @patch(DB_MANAGER, name="mock_db") def create_notice_analyzer_obj(self, mock_db): - from modules.flowalerts.notice import Notice + from modules.flow_alerts.notice import Notice flowalerts = self.create_flowalerts_obj() return Notice(flowalerts.db, flowalerts=flowalerts) @patch(DB_MANAGER, name="mock_db") def create_smtp_analyzer_obj(self, mock_db): - from modules.flowalerts.smtp import SMTP + from modules.flow_alerts.smtp import SMTP flowalerts = self.create_flowalerts_obj() return SMTP(flowalerts.db, flowalerts=flowalerts) @patch(DB_MANAGER, name="mock_db") def create_ssl_analyzer_obj(self, mock_db): - from modules.flowalerts.ssl import SSL + from modules.flow_alerts.ssl import SSL flowalerts = self.create_flowalerts_obj() return SSL(flowalerts.db, flowalerts=flowalerts) @patch(DB_MANAGER, name="mock_db") def create_ssh_analyzer_obj(self, mock_db): - from modules.flowalerts.ssh import SSH + from modules.flow_alerts.ssh import SSH flowalerts = self.create_flowalerts_obj() return SSH(flowalerts.db, flowalerts=flowalerts) @patch(DB_MANAGER, name="mock_db") def create_downloaded_file_analyzer_obj(self, mock_db): - from modules.flowalerts.downloaded_file import DownloadedFile + from modules.flow_alerts.downloaded_file import DownloadedFile flowalerts = self.create_flowalerts_obj() return DownloadedFile(flowalerts.db, flowalerts=flowalerts) @patch(DB_MANAGER, name="mock_db") def create_tunnel_analyzer_obj(self, mock_db): - from modules.flowalerts.tunnel import Tunnel + from modules.flow_alerts.tunnel import Tunnel flowalerts = self.create_flowalerts_obj() return Tunnel(flowalerts.db, flowalerts=flowalerts) @patch(DB_MANAGER, name="mock_db") def create_conn_analyzer_obj(self, mock_db): - from modules.flowalerts.conn import Conn + from modules.flow_alerts.conn import Conn flowalerts = self.create_flowalerts_obj() return Conn(flowalerts.db, flowalerts=flowalerts) @patch(DB_MANAGER, name="mock_db") def create_software_analyzer_obj(self, mock_db): - from modules.flowalerts.software import Software + from modules.flow_alerts.software import Software flowalerts = self.create_flowalerts_obj() return Software(flowalerts.db, flowalerts=flowalerts) @@ -723,7 +737,7 @@ def create_circllu_obj(self, mock_db): @patch(MODULE_DB_MANAGER, name="mock_db") def create_set_evidence_helper(self, mock_db): - from modules.flowalerts.set_evidence import SetEvidenceHelper + from modules.flow_alerts.set_evidence import SetEvidenceHelper """Create an instance of SetEvidenceHelper.""" set_evidence_helper = SetEvidenceHelper(mock_db) @@ -921,7 +935,7 @@ def fake_read_configuration(worker): conf=Mock(), ppid=Mock(), bloom_filters_manager=Mock(), - name="EvidenceHandlerWorker_Process_0", + name="evidence_handler_worker_process_0", evidence_queue=Mock(), evidence_logger_q=Mock(), ) @@ -984,9 +998,9 @@ def create_symbol_handler_obj(self, mock_db): @patch(MODULE_DB_MANAGER, name="mock_db") def create_riskiq_obj(self, mock_db): - from modules.riskiq.riskiq import RiskIQ + from modules.risk_iq.risk_iq import RiskIQ - riskiq = RiskIQ( + risk_iq = RiskIQ( logger=self.logger, output_dir="dummy_output_dir", redis_port=6379, @@ -996,8 +1010,8 @@ def create_riskiq_obj(self, mock_db): ppid=Mock(), bloom_filters_manager=Mock(), ) - riskiq.db = mock_db - return riskiq + risk_iq.db = mock_db + return risk_iq @patch(MODULE_DB_MANAGER, name="mock_db") def create_timeline_object(self, mock_db): @@ -1075,8 +1089,8 @@ def create_process_manager_obj(self): # main_mock.conf.get_bootstrapping_setting.return_value = (False, []) main_mock.conf.is_bootstrapping_node.return_value = False main_mock.conf.get_bootstrapping_modules.return_value = [ - "fidesModule", - "irisModule", + "fides", + "iris", ] main_mock.conf.generate_performance_plots.return_value = False main_mock.input_type = InputType.PCAP diff --git a/tests/unit/managers/test_process_manager.py b/tests/unit/managers/test_process_manager.py index 7f53753721..386dd9ae35 100644 --- a/tests/unit/managers/test_process_manager.py +++ b/tests/unit/managers/test_process_manager.py @@ -115,7 +115,7 @@ def test_print_disabled_modules(): [ # Test case 1: No pending modules, no additional print calls ([], 1), - # Test case 2: Pending modules without Update Manager, one additional print call + # Test case 2: Pending modules without update_manager, one additional print call ([Mock(name="Module1"), Mock(name="Module2")], 1), ], ) @@ -139,13 +139,13 @@ def test_warn_about_pending_modules(pending_modules, expected_print_calls): @pytest.mark.parametrize( "blocking_enabled, exporting_alerts_disabled, " "expected_kill_first, expected_kill_last", - [ # Testcase1: blocking enabled, Exporting Alerts enabled + [ # Testcase1: blocking enabled, exporting_alerts enabled (True, False, [1, 2], [3, 4, 5]), - # Testcase2: Blocking disabled, Exporting Alerts enabled + # Testcase2: blocking disabled, exporting_alerts enabled (False, False, [1, 2, 4], [3, 5]), - # Testcase3: Blocking enabled, Exporting Alerts disabled + # Testcase3: blocking enabled, exporting_alerts disabled (True, True, [1, 2, 5], [3, 4]), - # Testcase4: Blocking disabled, Exporting Alerts disabled + # Testcase4: blocking disabled, exporting_alerts disabled (False, True, [1, 2, 4, 5], [3]), # Testcase5: All enabled, some PIDs are None (True, False, [1, 2], [3, 4, 5]), @@ -161,15 +161,15 @@ def test_get_hitlist_in_order( process_manager.children = [ Mock(pid=1, name="Process1"), Mock(pid=2, name="Process2"), - Mock(pid=3, name="EvidenceHandler"), - Mock(pid=4, name="Blocking"), - Mock(pid=5, name="Exporting Alerts"), + Mock(pid=3, name="evidence_handler"), + Mock(pid=4, name="blocking"), + Mock(pid=5, name="exporting_alerts"), ] process_manager.main.db.get_pid_of = lambda x: { - "EvidenceHandler": 3, - "Blocking": 4, - "Exporting Alerts": 5, + "evidence_handler": 3, + "blocking": 4, + "exporting_alerts": 5, }.get(x) process_manager.main.args.blocking = blocking_enabled process_manager.main.db.get_disabled_modules = lambda: ( @@ -471,7 +471,7 @@ def test_start_evidence_process(output_dir, redis_port): mock_evidence_process.start.assert_called_once() process_manager.main.print.assert_called_once() process_manager.main.db.store_pid.assert_called_once_with( - "EvidenceHandler", 13579 + "evidence_handler", 13579 ) @@ -485,7 +485,7 @@ def test_print_started_module(): ) mock_print.assert_called_once_with( - "\t\tStarting the module green_module_name " + "\t\tStarting green_module_name module " "(Test description) [PID green_module_name]", 1, 0, diff --git a/tests/unit/managers/test_slips.py b/tests/unit/managers/test_slips.py index 6bc46bfaf3..2ba204b07a 100644 --- a/tests/unit/managers/test_slips.py +++ b/tests/unit/managers/test_slips.py @@ -10,6 +10,7 @@ def test_load_modules(): proc_manager.modules_to_ignore = [ "template", "mldetection-1", + "fides", ] failed_to_load_modules = proc_manager.get_modules()[1] assert failed_to_load_modules == 0 diff --git a/tests/unit/modules/anomaly_detection_https/test_anomaly_detection_https.py b/tests/unit/modules/anomaly_detection_https/test_anomaly_detection_https.py index 940ada8a6e..ccbf1f16bf 100644 --- a/tests/unit/modules/anomaly_detection_https/test_anomaly_detection_https.py +++ b/tests/unit/modules/anomaly_detection_https/test_anomaly_detection_https.py @@ -60,6 +60,11 @@ def test_https_anomaly_module_is_instantiable_and_subscribes_to_new_ssl( ) assert isinstance(module, AnomalyDetectionHTTPS) + assert module.output_dir == str(tmp_path / "anomaly_detection_https") + assert module.parent_output_dir == str(tmp_path) + assert module.operational_log_path == str( + tmp_path / "anomaly_detection_https" / "anomaly_detection_https.log" + ) module.subscribe_to_channels() diff --git a/tests/unit/modules/arp/test_arp_filter.py b/tests/unit/modules/arp/test_arp_filter.py index 221fe7b28b..2a72978ced 100644 --- a/tests/unit/modules/arp/test_arp_filter.py +++ b/tests/unit/modules/arp/test_arp_filter.py @@ -40,7 +40,7 @@ def test_is_slips_peer(p2p_enabled, is_private, peer_trust, expected): ) def test_is_self_defense(ip, our_ips, blocking, has_poisoner, expected): db = Mock() - db.get_pids.return_value = {"ARP Poisoner": 123} if has_poisoner else {} + db.get_pids.return_value = {"arp_poisoner": 123} if has_poisoner else {} args = Mock() args.blocking = blocking diff --git a/tests/unit/modules/bruteforcing/test_bruteforcing.py b/tests/unit/modules/brute_force_detector/test_brute_force_detector.py similarity index 68% rename from tests/unit/modules/bruteforcing/test_bruteforcing.py rename to tests/unit/modules/brute_force_detector/test_brute_force_detector.py index 9075fb7cee..a2ae2defc2 100644 --- a/tests/unit/modules/bruteforcing/test_bruteforcing.py +++ b/tests/unit/modules/brute_force_detector/test_brute_force_detector.py @@ -84,11 +84,11 @@ def drive_threshold(module, client_banner="SSH-2.0-OpenSSH_9.6p1"): return module.db.set_evidence.call_args[0][0] -def test_software_banner_increases_bruteforcing_confidence(): - plain_module = ModuleFactory().create_bruteforcing_obj() +def test_software_banner_increases_brute_force_detector_confidence(): + plain_module = ModuleFactory().create_brute_force_detector_obj() plain_evidence = drive_threshold(plain_module) - banner_module = ModuleFactory().create_bruteforcing_obj() + banner_module = ModuleFactory().create_brute_force_detector_obj() banner_module._handle_software(make_software_flow()) banner_evidence = drive_threshold( banner_module, client_banner="SSH-2.0-libssh2_1.11.0" @@ -100,34 +100,34 @@ def test_software_banner_increases_bruteforcing_confidence(): assert banner_evidence.dst_port == 902 -def test_bruteforcing_uses_sparse_bucketed_reporting(): - bruteforcing = ModuleFactory().create_bruteforcing_obj() - bruteforcing.db.get_port_info.return_value = "SSH" +def test_brute_force_detector_uses_sparse_bucketed_reporting(): + brute_force_detector = ModuleFactory().create_brute_force_detector_obj() + brute_force_detector.db.get_port_info.return_value = "SSH" for attempt in range(1, 25): - bruteforcing._handle_ssh( + brute_force_detector._handle_ssh( PROFILEID, TWID, make_ssh_flow(uid=f"uid-{attempt}"), ) - assert bruteforcing.db.set_evidence.call_count == 5 + assert brute_force_detector.db.set_evidence.call_count == 5 observed_attempt_counts = [ len(call_args[0][0].uid) - for call_args in bruteforcing.db.set_evidence.call_args_list + for call_args in brute_force_detector.db.set_evidence.call_args_list ] assert observed_attempt_counts == [9, 10, 12, 16, 24] def test_confidence_reaches_full_at_30_attempts(): - bruteforcing = ModuleFactory().create_bruteforcing_obj() - threshold_confidence = bruteforcing._calculate_confidence( - bruteforcing.ssh_attempt_threshold, + brute_force_detector = ModuleFactory().create_brute_force_detector_obj() + threshold_confidence = brute_force_detector._calculate_confidence( + brute_force_detector.ssh_attempt_threshold, "SSH-2.0-OpenSSH_9.6p1", "ssh.log", ) - full_confidence = bruteforcing._calculate_confidence( - bruteforcing.ssh_full_confidence_attempts, + full_confidence = brute_force_detector._calculate_confidence( + brute_force_detector.ssh_full_confidence_attempts, "SSH-2.0-OpenSSH_9.6p1", "ssh.log", ) @@ -135,34 +135,36 @@ def test_confidence_reaches_full_at_30_attempts(): assert threshold_confidence < 1.0 assert full_confidence == 1.0 - evidence = drive_threshold(bruteforcing) + evidence = drive_threshold(brute_force_detector) assert evidence.threat_level == ThreatLevel.MEDIUM def test_notice_confirmation_emits_zeek_evidence_and_confirms_future_alerts(): - bruteforcing = ModuleFactory().create_bruteforcing_obj() - drive_threshold(bruteforcing) - bruteforcing.db.set_evidence.reset_mock() + brute_force_detector = ModuleFactory().create_brute_force_detector_obj() + drive_threshold(brute_force_detector) + brute_force_detector.db.set_evidence.reset_mock() - bruteforcing._handle_notice(PROFILEID, TWID, make_notice_flow()) - zeek_evidence = bruteforcing.db.set_evidence.call_args[0][0] + brute_force_detector._handle_notice(PROFILEID, TWID, make_notice_flow()) + zeek_evidence = brute_force_detector.db.set_evidence.call_args[0][0] assert zeek_evidence.confidence == 1.0 assert zeek_evidence.threat_level == ThreatLevel.MEDIUM assert "Confirmed by Zeek notice.log." in zeek_evidence.description - bruteforcing.db.set_evidence.reset_mock() - bruteforcing._handle_ssh(PROFILEID, TWID, make_ssh_flow(uid="uid-10")) - confirmed_evidence = bruteforcing.db.set_evidence.call_args[0][0] + brute_force_detector.db.set_evidence.reset_mock() + brute_force_detector._handle_ssh( + PROFILEID, TWID, make_ssh_flow(uid="uid-10") + ) + confirmed_evidence = brute_force_detector.db.set_evidence.call_args[0][0] assert confirmed_evidence.confidence == 1.0 assert "Confirmed by Zeek notice.log." in confirmed_evidence.description def test_repeated_ssh_sessions_without_auth_attempts_still_trigger_detection(): - bruteforcing = ModuleFactory().create_bruteforcing_obj() - bruteforcing.db.get_port_info.return_value = "SSH" + brute_force_detector = ModuleFactory().create_brute_force_detector_obj() + brute_force_detector.db.get_port_info.return_value = "SSH" for attempt in range(20): - bruteforcing._handle_ssh( + brute_force_detector._handle_ssh( PROFILEID, TWID, make_ssh_flow( @@ -172,4 +174,4 @@ def test_repeated_ssh_sessions_without_auth_attempts_still_trigger_detection(): ), ) - assert bruteforcing.db.set_evidence.call_count == 4 + assert brute_force_detector.db.set_evidence.call_count == 4 diff --git a/tests/unit/modules/fidesModule/test_fides_bridge.py b/tests/unit/modules/fides/test_fides_bridge.py similarity index 85% rename from tests/unit/modules/fidesModule/test_fides_bridge.py rename to tests/unit/modules/fides/test_fides_bridge.py index 7661d72027..600ae348f6 100644 --- a/tests/unit/modules/fidesModule/test_fides_bridge.py +++ b/tests/unit/modules/fides/test_fides_bridge.py @@ -1,11 +1,11 @@ import pytest from unittest.mock import MagicMock -from modules.fidesModule.messaging.network_bridge import NetworkBridge -from modules.fidesModule.messaging.queue import Queue -from modules.fidesModule.messaging.message_handler import MessageHandler -from modules.fidesModule.messaging.network_bridge import NetworkMessage -from modules.fidesModule.model.aliases import PeerId, Target -from modules.fidesModule.model.threat_intelligence import ThreatIntelligence +from modules.fides.messaging.network_bridge import NetworkBridge +from modules.fides.messaging.queue import Queue +from modules.fides.messaging.message_handler import MessageHandler +from modules.fides.messaging.network_bridge import NetworkMessage +from modules.fides.model.aliases import PeerId, Target +from modules.fides.model.threat_intelligence import ThreatIntelligence @pytest.fixture diff --git a/tests/unit/modules/fidesModule/test_fides_module.py b/tests/unit/modules/fides/test_fides_module.py similarity index 75% rename from tests/unit/modules/fidesModule/test_fides_module.py rename to tests/unit/modules/fides/test_fides_module.py index b91999e492..297c8a9093 100644 --- a/tests/unit/modules/fidesModule/test_fides_module.py +++ b/tests/unit/modules/fides/test_fides_module.py @@ -1,5 +1,5 @@ """ -Unit tests for modules/fidesModule/fidesModule.py +Unit tests for modules/fides/fides.py The sqlite database used by and implemented in FidesModule has its own unit tests. You may find them here: .test_fides_sqlite_db.py @@ -15,7 +15,7 @@ @pytest.fixture def cleanup_database(): # name of the database created by Fides - db_name = "fides_p2p_db.sqlite" + db_name = os.path.join("permanent", "databases", "fides_p2p_db.sqlite") yield # Let the test run @@ -25,10 +25,10 @@ def cleanup_database(): def test_pre_main(mocker, cleanup_database): - fides_module = ModuleFactory().create_fides_module_obj() + fides = ModuleFactory().create_fides_obj() mocker.patch( "slips_files.common.slips_utils.Utils.drop_root_privs_permanently" ) - fides_module.subscribe_to_channels() - fides_module.pre_main() + fides.subscribe_to_channels() + fides.pre_main() utils.drop_root_privs_permanently.assert_called_once() diff --git a/tests/unit/modules/fidesModule/test_fides_queues.py b/tests/unit/modules/fides/test_fides_queues.py similarity index 98% rename from tests/unit/modules/fidesModule/test_fides_queues.py rename to tests/unit/modules/fides/test_fides_queues.py index 105510246c..f2fa327e68 100644 --- a/tests/unit/modules/fidesModule/test_fides_queues.py +++ b/tests/unit/modules/fides/test_fides_queues.py @@ -1,7 +1,7 @@ import pytest from unittest.mock import MagicMock from threading import Thread -from modules.fidesModule.messaging.redis_simplex_queue import ( +from modules.fides.messaging.redis_simplex_queue import ( RedisSimplexQueue, RedisDuplexQueue, ) diff --git a/tests/unit/modules/fidesModule/test_fides_sqlite_db.py b/tests/unit/modules/fides/test_fides_sqlite_db.py similarity index 82% rename from tests/unit/modules/fidesModule/test_fides_sqlite_db.py rename to tests/unit/modules/fides/test_fides_sqlite_db.py index c1a7e33c6e..e76de05000 100644 --- a/tests/unit/modules/fidesModule/test_fides_sqlite_db.py +++ b/tests/unit/modules/fides/test_fides_sqlite_db.py @@ -1,17 +1,17 @@ import pytest from unittest.mock import MagicMock -from modules.fidesModule.model.peer import PeerInfo -from modules.fidesModule.model.peer_trust_data import PeerTrustData -from modules.fidesModule.model.threat_intelligence import ( +from modules.fides.model.peer import PeerInfo +from modules.fides.model.peer_trust_data import PeerTrustData +from modules.fides.model.threat_intelligence import ( SlipsThreatIntelligence, ) -from modules.fidesModule.persistence.fides_sqlite_db import FidesSQLiteDB +from modules.fides.persistence.fides_sqlite_db import FidesSQLiteDB -from modules.fidesModule.model.recommendation_history import ( +from modules.fides.model.recommendation_history import ( RecommendationHistoryRecord, ) -from modules.fidesModule.model.service_history import ServiceHistoryRecord +from modules.fides.model.service_history import ServiceHistoryRecord @pytest.fixture @@ -76,6 +76,25 @@ def test_get_slips_threat_intelligence_by_target(db): def test_get_peer_trust_data(db): + db.store_peer_trust_data( + PeerTrustData( + info=PeerInfo( + id="peer-before", + organisations=["org0"], + ip="192.168.0.1", + ), + has_fixed_trust=False, + service_trust=0.1, + reputation=0.2, + recommendation_trust=0.3, + competence_belief=0.4, + integrity_belief=0.5, + initial_reputation_provided_by_count=1, + service_history=[], + recommendation_history=[], + ) + ) + # Create peer info and peer trust data peer_info = PeerInfo( id="peer123", organisations=["org1", "org2"], ip="192.168.0.10" @@ -121,6 +140,54 @@ def test_get_peer_trust_data(db): assert result.recommendation_history[0].satisfaction == 0.8 +def test_store_peer_trust_data_overwrites_existing_history(db): + """Ensure peer trust updates replace older rows for the same peer.""" + peer_info = PeerInfo( + id="peer123", organisations=["org1", "org2"], ip="192.168.0.10" + ) + + db.store_peer_trust_data( + PeerTrustData( + info=peer_info, + has_fixed_trust=False, + service_trust=0.1, + reputation=0.2, + recommendation_trust=0.3, + competence_belief=0.4, + integrity_belief=0.5, + initial_reputation_provided_by_count=1, + service_history=[], + recommendation_history=[], + ) + ) + + db.store_peer_trust_data( + PeerTrustData( + info=peer_info, + has_fixed_trust=False, + service_trust=0.7, + reputation=0.8, + recommendation_trust=0.9, + competence_belief=0.6, + integrity_belief=0.5, + initial_reputation_provided_by_count=2, + service_history=[ + ServiceHistoryRecord( + satisfaction=0.5, weight=0.9, timestamp=20.15 + ) + ], + recommendation_history=[], + ) + ) + + result = db.get_peer_trust_data("peer123") + + assert result is not None + assert result.service_trust == 0.7 + assert result.reputation == 0.8 + assert len(result.service_history) == 1 + + def test_get_connected_peers_1(db): # Create PeerInfo data for multiple peers peers = [ diff --git a/tests/unit/modules/flowalerts/test_conn.py b/tests/unit/modules/flow_alerts/test_conn.py similarity index 99% rename from tests/unit/modules/flowalerts/test_conn.py rename to tests/unit/modules/flow_alerts/test_conn.py index 5d6c520d86..c2281603b8 100644 --- a/tests/unit/modules/flowalerts/test_conn.py +++ b/tests/unit/modules/flow_alerts/test_conn.py @@ -1,6 +1,6 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only -"""Unit test for modules/flowalerts/conn.py""" +"""Unit test for modules/flow_alerts/conn.py""" from slips_files.core.flows.zeek import Conn from tests.module_factory import ModuleFactory @@ -163,7 +163,7 @@ def test_check_unknown_port( conn.db.is_ftp_port.return_value = mock_is_ftp_port port_belongs_mock = mocker.patch( - "modules.flowalerts.conn.Conn.port_belongs_to_an_org" + "modules.flow_alerts.conn.Conn.port_belongs_to_an_org" ) port_belongs_mock.return_value = mock_port_belongs_to_an_org @@ -352,7 +352,7 @@ def test_check_multiple_reconnection_attempts( """ conn = ModuleFactory().create_conn_analyzer_obj() mock_set_evidence = mocker.patch( - "modules.flowalerts.set_evidence." + "modules.flow_alerts.set_evidence." "SetEvidenceHelper.multiple_reconnection_attempts" ) conn.db.get_reconnections_for_tw.return_value = {} @@ -466,7 +466,7 @@ def test_check_data_upload( conn = ModuleFactory().create_conn_analyzer_obj() conn.is_ignored_ip_data_upload = Mock(return_value=ignored_ip) mock_set_evidence = mocker.patch( - "modules.flowalerts.set_evidence.SetEvidenceHelper.data_exfiltration" + "modules.flow_alerts.set_evidence.SetEvidenceHelper.data_exfiltration" ) conn.gateway = "192.168.1.1" flow = Conn( diff --git a/tests/unit/modules/flowalerts/test_dns.py b/tests/unit/modules/flow_alerts/test_dns.py similarity index 99% rename from tests/unit/modules/flowalerts/test_dns.py rename to tests/unit/modules/flow_alerts/test_dns.py index 2d9ba93f04..f1bdd00649 100644 --- a/tests/unit/modules/flowalerts/test_dns.py +++ b/tests/unit/modules/flow_alerts/test_dns.py @@ -1,6 +1,6 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only -"""Unit test for modules/flowalerts/dns.py""" +"""Unit test for modules/flow_alerts/dns.py""" from dataclasses import asdict diff --git a/tests/unit/modules/flowalerts/test_downloaded_file.py b/tests/unit/modules/flow_alerts/test_downloaded_file.py similarity index 98% rename from tests/unit/modules/flowalerts/test_downloaded_file.py rename to tests/unit/modules/flow_alerts/test_downloaded_file.py index 3a8c297153..d6e2e9da5c 100644 --- a/tests/unit/modules/flowalerts/test_downloaded_file.py +++ b/tests/unit/modules/flow_alerts/test_downloaded_file.py @@ -1,6 +1,6 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only -"""Unit test for modules/flowalerts/download_file.py""" +"""Unit test for modules/flow_alerts/download_file.py""" from dataclasses import asdict from unittest.mock import Mock diff --git a/tests/unit/modules/flowalerts/test_notice.py b/tests/unit/modules/flow_alerts/test_notice.py similarity index 99% rename from tests/unit/modules/flowalerts/test_notice.py rename to tests/unit/modules/flow_alerts/test_notice.py index c2238b860c..d63770b5df 100644 --- a/tests/unit/modules/flowalerts/test_notice.py +++ b/tests/unit/modules/flow_alerts/test_notice.py @@ -1,6 +1,6 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only -"""Unit test for modules/flowalerts/notice.py""" +"""Unit test for modules/flow_alerts/notice.py""" from dataclasses import asdict from unittest.mock import Mock diff --git a/tests/unit/modules/flowalerts/test_set_evidence.py b/tests/unit/modules/flow_alerts/test_set_evidence.py similarity index 100% rename from tests/unit/modules/flowalerts/test_set_evidence.py rename to tests/unit/modules/flow_alerts/test_set_evidence.py diff --git a/tests/unit/modules/flowalerts/test_smtp.py b/tests/unit/modules/flow_alerts/test_smtp.py similarity index 98% rename from tests/unit/modules/flowalerts/test_smtp.py rename to tests/unit/modules/flow_alerts/test_smtp.py index 5edbdbefe3..f22b9553aa 100644 --- a/tests/unit/modules/flowalerts/test_smtp.py +++ b/tests/unit/modules/flow_alerts/test_smtp.py @@ -1,6 +1,6 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only -"""Unit test for modules/flowalerts/flowalerts.py""" +"""Unit test for modules/flow_alerts/flow_alerts.py""" from dataclasses import asdict diff --git a/tests/unit/modules/flowalerts/test_software.py b/tests/unit/modules/flow_alerts/test_software.py similarity index 99% rename from tests/unit/modules/flowalerts/test_software.py rename to tests/unit/modules/flow_alerts/test_software.py index df5e5aada4..9a1471ad00 100644 --- a/tests/unit/modules/flowalerts/test_software.py +++ b/tests/unit/modules/flow_alerts/test_software.py @@ -1,6 +1,6 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only -"""Unit test for modules/flowalerts/software.py""" +"""Unit test for modules/flow_alerts/software.py""" from slips_files.core.flows.zeek import Software from tests.module_factory import ModuleFactory diff --git a/tests/unit/modules/flowalerts/test_ssh.py b/tests/unit/modules/flow_alerts/test_ssh.py similarity index 95% rename from tests/unit/modules/flowalerts/test_ssh.py rename to tests/unit/modules/flow_alerts/test_ssh.py index ad7fb34ce5..a85a349e55 100644 --- a/tests/unit/modules/flowalerts/test_ssh.py +++ b/tests/unit/modules/flow_alerts/test_ssh.py @@ -1,6 +1,6 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only -"""Unit test for modules/flowalerts/ssh.py""" +"""Unit test for modules/flow_alerts/ssh.py""" from dataclasses import asdict @@ -50,10 +50,10 @@ async def test_check_successful_ssh( mocker.patch("asyncio.sleep", return_value=get_mock_coro(None)) mock_set_evidence_ssh_successful_by_zeek = mocker.patch( - "modules.flowalerts.ssh.SSH.set_evidence_ssh_successful_by_zeek" + "modules.flow_alerts.ssh.SSH.set_evidence_ssh_successful_by_zeek" ) mock_detect_slips = mocker.patch( - "modules.flowalerts.ssh.SSH.detect_successful_ssh_by_slips" + "modules.flow_alerts.ssh.SSH.detect_successful_ssh_by_slips" ) flow = SSH( starttime="1726655400.0", @@ -85,7 +85,7 @@ async def test_check_successful_ssh( assert mock_detect_slips.called == expected_called_slips -@patch("modules.flowalerts.ssh.ConfigParser") +@patch("modules.flow_alerts.ssh.ConfigParser") def test_read_configuration(mock_config_parser): mock_parser = mock_config_parser.return_value mock_parser.ssh_succesful_detection_threshold.return_value = 12345 diff --git a/tests/unit/modules/flowalerts/test_ssl.py b/tests/unit/modules/flow_alerts/test_ssl.py similarity index 97% rename from tests/unit/modules/flowalerts/test_ssl.py rename to tests/unit/modules/flow_alerts/test_ssl.py index a71fe1ac7e..41872e3d6a 100644 --- a/tests/unit/modules/flowalerts/test_ssl.py +++ b/tests/unit/modules/flow_alerts/test_ssl.py @@ -1,6 +1,6 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only -"""Unit test for modules/flowalerts/ssl.py""" +"""Unit test for modules/flow_alerts/ssl.py""" from dataclasses import asdict from unittest.mock import ( @@ -47,7 +47,7 @@ def test_check_self_signed_certs( ): ssl = ModuleFactory().create_ssl_analyzer_obj() mock_set_evidence = mocker.patch( - "modules.flowalerts.set_evidence." + "modules.flow_alerts.set_evidence." "SetEvidenceHelper.self_signed_certificates" ) flow = SSL( @@ -99,10 +99,10 @@ def test_detect_malicious_ja3( ): ssl = ModuleFactory().create_ssl_analyzer_obj() mock_set_evidence_ja3 = mocker.patch( - "modules.flowalerts.set_evidence.SetEvidenceHelper.malicious_ja3" + "modules.flow_alerts.set_evidence.SetEvidenceHelper.malicious_ja3" ) mock_set_evidence_ja3s = mocker.patch( - "modules.flowalerts.set_evidence.SetEvidenceHelper.malicious_ja3s" + "modules.flow_alerts.set_evidence.SetEvidenceHelper.malicious_ja3s" ) ssl.db.get_all_blacklisted_ja3.return_value = { @@ -150,7 +150,7 @@ def test_detect_malicious_ja3( def test_detect_doh(mocker, is_doh, expected_calls): ssl = ModuleFactory().create_ssl_analyzer_obj() mock_set_evidence_doh = mocker.patch( - "modules.flowalerts.set_evidence.SetEvidenceHelper.doh" + "modules.flow_alerts.set_evidence.SetEvidenceHelper.doh" ) ssl.db.set_ip_info = Mock() flow = SSL( @@ -204,7 +204,7 @@ async def test_check_pastebin_download( ssl.pastebin_downloads_threshold = 12000 ssl.wait_for_new_flows_or_timeout = get_mock_coro(True) mock_set_evidence = mocker.patch( - "modules.flowalerts.set_evidence.SetEvidenceHelper.pastebin_download" + "modules.flow_alerts.set_evidence.SetEvidenceHelper.pastebin_download" ) flow = SSL( @@ -260,7 +260,7 @@ async def test_check_pastebin_download( def test_detect_incompatible_cn(mocker, subject, expected_call_count): ssl = ModuleFactory().create_ssl_analyzer_obj() mock_set_evidence = mocker.patch( - "modules.flowalerts.set_evidence.SetEvidenceHelper.incompatible_cn" + "modules.flow_alerts.set_evidence.SetEvidenceHelper.incompatible_cn" ) ssl.db.whitelist.organization_whitelist.is_ip_in_org.return_value = False diff --git a/tests/unit/modules/flowalerts/test_tunnel.py b/tests/unit/modules/flow_alerts/test_tunnel.py similarity index 94% rename from tests/unit/modules/flowalerts/test_tunnel.py rename to tests/unit/modules/flow_alerts/test_tunnel.py index 473ba49fa1..7ca78b722a 100644 --- a/tests/unit/modules/flowalerts/test_tunnel.py +++ b/tests/unit/modules/flow_alerts/test_tunnel.py @@ -1,6 +1,6 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only -"""Unit test for modules/flowalerts/tunnel.py""" +"""Unit test for modules/flow_alerts/tunnel.py""" from dataclasses import asdict from unittest.mock import Mock @@ -30,7 +30,7 @@ def test_check_gre_tunnel(mocker, tunnel_type, expected_call_count): """ tunnel = ModuleFactory().create_tunnel_analyzer_obj() mock_set_evidence = mocker.patch( - "modules.flowalerts.set_evidence.SetEvidenceHelper.gre_tunnel" + "modules.flow_alerts.set_evidence.SetEvidenceHelper.gre_tunnel" ) flow = Tunnel( starttime="1726655400.0", @@ -61,7 +61,7 @@ def test_check_gre_tunnel(mocker, tunnel_type, expected_call_count): def test_check_gre_scan(mocker, tunnel_type, expected_call_count): tunnel = ModuleFactory().create_tunnel_analyzer_obj() mock_set_evidence = mocker.patch( - "modules.flowalerts.set_evidence.SetEvidenceHelper.gre_scan" + "modules.flow_alerts.set_evidence.SetEvidenceHelper.gre_scan" ) flow = Tunnel( starttime="1726655400.0", diff --git a/tests/unit/modules/p2ptrust/trust/test_base_model.py b/tests/unit/modules/p2p_trust/trust/test_base_model.py similarity index 100% rename from tests/unit/modules/p2ptrust/trust/test_base_model.py rename to tests/unit/modules/p2p_trust/trust/test_base_model.py diff --git a/tests/unit/modules/p2ptrust/trust/test_go_director.py b/tests/unit/modules/p2p_trust/trust/test_go_director.py similarity index 98% rename from tests/unit/modules/p2ptrust/trust/test_go_director.py rename to tests/unit/modules/p2p_trust/trust/test_go_director.py index eb8ed9e92c..e9337eaf75 100644 --- a/tests/unit/modules/p2ptrust/trust/test_go_director.py +++ b/tests/unit/modules/p2p_trust/trust/test_go_director.py @@ -556,11 +556,11 @@ def test_respond_to_message_request_with_info(): confidence = 0.8 with patch( - "modules.p2ptrust.utils." "go_director.get_ip_info_from_slips", + "modules.p2p_trust.utils." "go_director.get_ip_info_from_slips", return_value=(score, confidence), ) as mock_get_info: with patch( - "modules.p2ptrust." "utils.go_director." "send_evaluation_to_go" + "modules.p2p_trust." "utils.go_director." "send_evaluation_to_go" ) as mock_send_evaluation: go_director.respond_to_message_request(key, reporter) @@ -591,11 +591,11 @@ def test_respond_to_message_request_without_info(): confidence = None with patch( - "modules.p2ptrust.utils." "go_director.get_ip_info_from_slips", + "modules.p2p_trust.utils." "go_director.get_ip_info_from_slips", return_value=(score, confidence), ) as mock_get_info: with patch( - "modules.p2ptrust.utils." "go_director.send_evaluation_to_go" + "modules.p2p_trust.utils." "go_director.send_evaluation_to_go" ) as mock_send_evaluation: go_director.respond_to_message_request(key, reporter) diff --git a/tests/unit/modules/p2ptrust/trust/test_trustdb.py b/tests/unit/modules/p2p_trust/trust/test_trustdb.py similarity index 100% rename from tests/unit/modules/p2ptrust/trust/test_trustdb.py rename to tests/unit/modules/p2p_trust/trust/test_trustdb.py diff --git a/tests/unit/modules/riskiq/test_riskiq.py b/tests/unit/modules/risk_iq/test_risk_iq.py similarity index 85% rename from tests/unit/modules/riskiq/test_riskiq.py rename to tests/unit/modules/risk_iq/test_risk_iq.py index 1b6f3f6216..ef83583ae6 100644 --- a/tests/unit/modules/riskiq/test_riskiq.py +++ b/tests/unit/modules/risk_iq/test_risk_iq.py @@ -36,11 +36,11 @@ def test_get_passive_dns(mock_get, ip, api_response, expected_result): mock_response.status_code = 200 mock_response.text = json.dumps(api_response) - riskiq = ModuleFactory().create_riskiq_obj() - riskiq.riskiq_email = "test@example.com" - riskiq.riskiq_key = "testkey" + risk_iq = ModuleFactory().create_riskiq_obj() + risk_iq.riskiq_email = "test@example.com" + risk_iq.riskiq_key = "testkey" - result = riskiq.get_passive_dns(ip) + result = risk_iq.get_passive_dns(ip) assert result == expected_result mock_get.assert_called_once_with( @@ -68,7 +68,7 @@ def test_pre_main(email, key, expected_result, mock_db): with patch( "slips_files.common.slips_utils.utils.drop_root_privs_permanently" ): - riskiq = ModuleFactory().create_riskiq_obj() - riskiq.riskiq_email = email - riskiq.riskiq_key = key - assert riskiq.pre_main() == expected_result + risk_iq = ModuleFactory().create_riskiq_obj() + risk_iq.riskiq_email = email + risk_iq.riskiq_key = key + assert risk_iq.pre_main() == expected_result diff --git a/tests/unit/slips/test_main.py b/tests/unit/slips/test_main.py index eef002190e..9efb1b8072 100644 --- a/tests/unit/slips/test_main.py +++ b/tests/unit/slips/test_main.py @@ -203,7 +203,7 @@ def test_print( text, verbose, debug, log_to_logfiles_only, expected_notification ): main = ModuleFactory().create_main_obj() - main.name = "Main" + main.name = "main" main.printer = Mock() main.printer.print = Mock() main.print(text, verbose, debug, log_to_logfiles_only) diff --git a/tests/unit/slips_files/common/test_output_paths.py b/tests/unit/slips_files/common/test_output_paths.py new file mode 100644 index 0000000000..4da0e21718 --- /dev/null +++ b/tests/unit/slips_files/common/test_output_paths.py @@ -0,0 +1,33 @@ +"""Unit tests for output path helpers.""" + +from slips_files.common.output_paths import ( + DATABASES_DIRNAME, + get_databases_dir_path_inside_output_dir, + get_this_db_path_inside_output_dir, +) +from tests.module_factory import ModuleFactory + + +def test_get_databases_dir_path_inside_output_dir_creates_directory(tmp_path): + """The databases helper should create and return the output databases directory.""" + module_factory = ModuleFactory() + assert module_factory is not None + + databases_dir = get_databases_dir_path_inside_output_dir( + str(tmp_path / "output") + ) + + assert databases_dir.endswith(DATABASES_DIRNAME) + assert (tmp_path / "output" / DATABASES_DIRNAME).is_dir() + + +def test_get_output_sqlite_path_joins_filename_under_databases_dir(tmp_path): + """The sqlite path helper should return a path inside the databases directory.""" + module_factory = ModuleFactory() + assert module_factory is not None + + sqlite_path = get_this_db_path_inside_output_dir( + str(tmp_path / "output"), "test.db" + ) + + assert sqlite_path.endswith(f"{DATABASES_DIRNAME}/test.db") diff --git a/tests/unit/slips_files/core/database/test_database.py b/tests/unit/slips_files/core/database/test_database.py index 8fd597d1a4..9c54ff69d5 100644 --- a/tests/unit/slips_files/core/database/test_database.py +++ b/tests/unit/slips_files/core/database/test_database.py @@ -10,6 +10,7 @@ import os from slips_files.core.flows.zeek import Conn +from slips_files.core.database.database_manager import DBManager from slips_files.core.database.redis_db.database import RedisDB from tests.module_factory import ModuleFactory @@ -163,20 +164,37 @@ def test_setup_config_file_uses_isolated_path_and_preserves_save( ) monkeypatch.setattr(RedisDB, "_conf_file_template", str(template)) - monkeypatch.setattr(RedisDB, "output_dir", tmp_path) - monkeypatch.setattr(RedisDB, "redis_port", 6379) - monkeypatch.setattr(RedisDB, "args", Mock(save=False)) + monkeypatch.setattr(RedisDB, "output_dir", tmp_path, raising=False) + monkeypatch.setattr(RedisDB, "redis_port", 6379, raising=False) + monkeypatch.setattr(RedisDB, "args", Mock(save=False), raising=False) RedisDB._setup_config_file() expected_conf = ( - tmp_path / f"redis-server-port-{RedisDB.redis_port}-{os.getpid()}.conf" + tmp_path / "redis" / f"redis-server-port-{RedisDB.redis_port}.conf" ) assert RedisDB._conf_file == str(expected_conf) conf_contents = expected_conf.read_text(encoding="utf-8").splitlines() assert 'save ""' in conf_contents assert ( - f"logfile {tmp_path / f'redis-server-port-{RedisDB.redis_port}.log'}" + f"logfile {tmp_path / 'redis' / f'redis-server-port-{RedisDB.redis_port}.log'}" in conf_contents ) + + +def test_init_p2p_trust_db_uses_permanent_dir(tmp_path, monkeypatch): + db = ModuleFactory().create_db_manager_obj(6379) + monkeypatch.chdir(tmp_path) + db.init_p2p_trust_db = DBManager.init_p2p_trust_db.__get__(db, DBManager) + monkeypatch.setattr( + "slips_files.core.database.database_manager.get_this_filepath_inside_permanent_dir", + lambda filename: os.path.join("persistent_state", filename), + ) + + db_path = db.init_p2p_trust_db() + + assert db_path == os.path.join( + "persistent_state", "p2p_trust_runtime", "trustdb.db" + ) + assert os.path.isdir(os.path.join("persistent_state", "p2p_trust_runtime")) diff --git a/tests/unit/slips_files/core/input/test_input.py b/tests/unit/slips_files/core/input/test_input.py index c89ba7eac1..4979c5957a 100644 --- a/tests/unit/slips_files/core/input/test_input.py +++ b/tests/unit/slips_files/core/input/test_input.py @@ -410,15 +410,13 @@ def test_zeek_log_file_shutdown_closes_handles(): "", InputType.ZEEK_LOG_FILE ) handler = input_process.input_handlers[InputType.ZEEK_LOG_FILE] - input_process.zeek_utils.open_file_handlers = { - "test_file.log": MagicMock() - } + mock_handle = MagicMock() + input_process.zeek_utils.open_file_handles = {"test_file.log": mock_handle} input_process.mark_self_as_done_processing = MagicMock() assert handler.shutdown_gracefully() is True - assert input_process.zeek_utils.open_file_handlers[ - "test_file.log" - ].close.called + mock_handle.close.assert_called_once() + assert input_process.zeek_utils.open_file_handles == {} input_process.mark_self_as_done_processing.assert_called_once() @@ -429,7 +427,7 @@ def test_close_all_handles(): ) mock_handle1 = MagicMock() mock_handle2 = MagicMock() - input_process.zeek_utils.open_file_handlers = { + input_process.zeek_utils.open_file_handles = { "file1": mock_handle1, "file2": mock_handle2, } @@ -490,21 +488,19 @@ def test_check_if_time_to_del_rotated_files_deletes_old_files(): input_process = ModuleFactory().create_input_obj( "", InputType.ZEEK_LOG_FILE ) - input_process.keep_rotated_files_for = 0 - input_process.zeek_utils.time_rotated = 1 - input_process.zeek_utils.to_be_deleted = ["old1.log", "old2.log"] + input_process.zeek_utils.rotated_files_to_delete = [ + ("old1.log", 1.0), + ("old2.log", 1.0), + ] with ( - patch( - "slips_files.core.input.zeek.utils.zeek_input_utils.utils.convert_ts_format", - return_value=2, - ), + patch("time.time", return_value=2.0), patch("os.remove") as mock_remove, ): input_process.zeek_utils.check_if_time_to_del_rotated_files() assert mock_remove.call_count == 2 - assert input_process.zeek_utils.to_be_deleted == [] + assert input_process.zeek_utils.rotated_files_to_delete == [] @pytest.mark.parametrize( diff --git a/tests/unit/slips_files/core/input/test_zeek_file_remover.py b/tests/unit/slips_files/core/input/test_zeek_file_remover.py index f9d7b1b67b..2a3f21ea90 100644 --- a/tests/unit/slips_files/core/input/test_zeek_file_remover.py +++ b/tests/unit/slips_files/core/input/test_zeek_file_remover.py @@ -2,6 +2,9 @@ # SPDX-License-Identifier: GPL-2.0-only import json from unittest.mock import MagicMock, patch + +import pytest + from tests.module_factory import ModuleFactory from slips_files.core.input.zeek.utils.zeek_file_remover import ZeekFileRemover from slips_files.common.input_type import InputType @@ -23,7 +26,23 @@ def test_zeek_file_remover_start_subscribes_once(): assert input_process.channels["remove_old_files"] == "chan" -def test_remove_old_zeek_files_closes_and_marks(): +def test_zeek_file_remover_start_uses_existing_subscription(): + input_process = ModuleFactory().create_input_obj( + "", InputType.ZEEK_LOG_FILE + ) + input_process.channels["remove_old_files"] = "existing-chan" + input_process.db.subscribe = MagicMock(return_value="new-chan") + remover = ZeekFileRemover(input_process, input_process.zeek_utils) + remover.thread.start = MagicMock() + + remover.start() + + input_process.db.subscribe.assert_called_once_with("remove_old_files") + remover.thread.start.assert_called_once() + assert input_process.channels["remove_old_files"] == "new-chan" + + +def test_remove_old_zeek_files_closes_and_schedules_cleanup(): input_process = ModuleFactory().create_input_obj( "", InputType.ZEEK_LOG_FILE ) @@ -33,17 +52,57 @@ def test_remove_old_zeek_files_closes_and_marks(): old_file = "/tmp/conn.2020-01-01.log" new_file = "/tmp/conn.log" - input_process.zeek_utils.open_file_handlers[new_file] = MagicMock() + mock_handle = MagicMock() + input_process.zeek_utils.open_file_handles[new_file] = mock_handle msg = {"data": json.dumps({"old_file": old_file, "new_file": new_file})} input_process.get_msg = MagicMock(return_value=msg) - with patch( - "slips_files.core.input.zeek.utils.zeek_file_remover.utils.convert_ts_format", - return_value=123.0, - ): + with patch("time.time", return_value=123.0): remover.remove_old_zeek_files() - assert input_process.zeek_utils.open_file_handlers.get(new_file) is None - assert old_file in input_process.zeek_utils.to_be_deleted - assert input_process.zeek_utils.time_rotated == 123.0 + mock_handle.close.assert_called_once() + assert input_process.zeek_utils.open_file_handles.get(new_file) is None + assert input_process.zeek_utils.rotated_files_to_delete == [ + (old_file, 123.0 + input_process.keep_rotated_files_for) + ] + + +@pytest.mark.parametrize("keep_rotated_files_for", [0, 5]) +def test_check_if_time_to_del_rotated_files_deletes_ready_files( + keep_rotated_files_for, +): + input_process = ModuleFactory().create_input_obj( + "", InputType.ZEEK_LOG_FILE + ) + input_process.keep_rotated_files_for = keep_rotated_files_for + input_process.zeek_utils.rotated_files_to_delete = [ + ("/tmp/conn.2020-01-01.log", 100.0 + keep_rotated_files_for) + ] + + with patch( + "slips_files.core.input.zeek.utils.zeek_input_utils.utils.convert_ts_format", + return_value=105.0, + ), patch("slips_files.core.input.zeek.utils.zeek_input_utils.os.remove"): + input_process.zeek_utils.check_if_time_to_del_rotated_files() + + assert input_process.zeek_utils.rotated_files_to_delete == [] + + +def test_process_rotation_message_deletes_unsupported_files_immediately(): + input_process = ModuleFactory().create_input_obj( + "", InputType.ZEEK_LOG_FILE + ) + remover = ZeekFileRemover(input_process, input_process.zeek_utils) + old_file = "/tmp/loaded_scripts.2020-01-01.log" + new_file = "/tmp/loaded_scripts.log" + + with patch( + "slips_files.core.input.zeek.utils.zeek_file_remover.os.remove" + ) as mock_remove: + remover.process_rotation_message( + {"old_file": old_file, "new_file": new_file} + ) + + mock_remove.assert_called_once_with(old_file) + assert input_process.zeek_utils.rotated_files_to_delete == [] diff --git a/tests/unit/slips_files/core/test_evidence_handler.py b/tests/unit/slips_files/core/test_evidence_handler.py index e889a16130..070f1636bd 100644 --- a/tests/unit/slips_files/core/test_evidence_handler.py +++ b/tests/unit/slips_files/core/test_evidence_handler.py @@ -55,14 +55,14 @@ def test_start_evidence_worker(mock_worker_cls): mock_worker_cls.assert_called_once_with( logger=handler.logger, - output_dir=handler.output_dir, + output_dir=handler.parent_output_dir, redis_port=handler.redis_port, termination_event=handler.termination_event, conf=handler.conf, ppid=handler.ppid, slips_args=handler.args, bloom_filters_manager=handler.bloom_filters, - name="EvidenceHandlerWorker_Process_7", + name="evidence_handler_worker_process_7", evidence_queue=handler.evidence_worker_queue, evidence_logger_q=handler.evidence_logger_q, ) diff --git a/tests/unit/slips_files/core/test_profiler.py b/tests/unit/slips_files/core/test_profiler.py index 56af98abe1..a101defab1 100644 --- a/tests/unit/slips_files/core/test_profiler.py +++ b/tests/unit/slips_files/core/test_profiler.py @@ -1,11 +1,11 @@ # SPDX-FileCopyrightText: 2021 Sebastian Garcia # SPDX-License-Identifier: GPL-2.0-only -"""Unit test for slips_files/core/iperformance_profiler.py""" +"""Unit tests for the profiler core process.""" from unittest.mock import Mock, patch -from tests.module_factory import ModuleFactory import pytest +from tests.module_factory import ModuleFactory def mock_print(*args, **kwargs): @@ -136,3 +136,39 @@ def test_notify_observers_with_correct_message(): test_msg = {"action": "test_action"} profiler.notify_observers(test_msg) observer_mock.update.assert_called_once_with(test_msg) + + +@patch("slips_files.core.profiler.ProfilerWorker") +def test_start_profiler_worker_uses_parent_output_dir(mock_worker_cls): + profiler = ModuleFactory().create_profiler_obj() + worker = mock_worker_cls.return_value + profiler.profiler_child_processes = [] + profiler.workers = [] + profiler.profiler_queue = Mock() + profiler.input_handler_obj = Mock() + profiler.aid_queue = Mock() + profiler.aid_manager = Mock() + profiler.is_input_done_event = Mock() + + profiler.start_profiler_worker(7) + + mock_worker_cls.assert_called_once_with( + logger=profiler.logger, + output_dir=profiler.parent_output_dir, + redis_port=profiler.redis_port, + termination_event=profiler.termination_event, + conf=profiler.conf, + ppid=profiler.ppid, + slips_args=profiler.args, + bloom_filters_manager=profiler.bloom_filters, + name="profiler_worker_process_7", + profiler_queue=profiler.profiler_queue, + input_handler=profiler.input_handler_obj, + aid_queue=profiler.aid_queue, + aid_manager=profiler.aid_manager, + is_input_done_event=profiler.is_input_done_event, + ) + worker.start.assert_called_once() + assert profiler.profiler_child_processes == [worker] + assert profiler.workers == [] + profiler.db.increment_profiler_workers_started.assert_called_once() diff --git a/tests/unit/slips_files/core/test_profiler_worker.py b/tests/unit/slips_files/core/test_profiler_worker.py index 8477719223..14f96f8fec 100644 --- a/tests/unit/slips_files/core/test_profiler_worker.py +++ b/tests/unit/slips_files/core/test_profiler_worker.py @@ -119,8 +119,8 @@ def test_get_slips_start_time_falls_back_to_now(mock_time): @pytest.mark.parametrize( "name, expected_prefix", [ - ("ProfilerWorker_Process_2", "profiler_worker_2"), - ("ProfilerWorker", "profilerworker"), + ("profiler_worker_process_2", "profiler_worker_2"), + ("profiler_worker", "profiler_worker"), ("mock_name", "mock_name"), ], ) @@ -604,7 +604,7 @@ def test_should_stop_always_returns_false(): def test_pre_main_updates_line_processor_cache(): profiler = ModuleFactory().create_profiler_worker_obj() - profiler.name = "ProfilerWorker_Process_2" + profiler.name = "profiler_worker_process_2" profiler.input_handler.line_processor_cache = {} profiler.db.get_line_processors.return_value = { "conn.log": json.dumps({"ts": 0})