Merge branch 'develop' into fix-check-import

.github/workflows/build-docs.yml

@@ -45,6 +45,10 @@ jobs:
           git config user.name intelmq-bot
           git config user.email intelmq-bot
 
+      - name: "Fix images path"
+        run: |
+          sed -i 's|docs/static/images/|static/images/|' docs/index.md
+
       - name: "Build docs without publishing"
         if: github.event_name == 'pull_request'
         run: |

CHANGELOG.md

@@ -34,6 +34,7 @@
   (PR#2408 and PR#2414 by Jan Kaliszewski).
 - `intelmq.lib.upgrades`: Replace deprecated instances of `url2fqdn` experts by the new `url` expert in runtime configuration (PR#2432 by Sebastian Wagner).
 - `intelmq.lib.bot`: Ensure closing log files on reloading (PR#2435 by Kamil Mankowski).
+- Only load the config once when starting intelmqctl (which makes IntelMQ API calls take less time) (PR#2444 by DigitalTrustCenter).
 
 ### Development
 - Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).
@@ -61,11 +62,18 @@
   - Remove undocumented and unused attributes of `StompCollectorBot` instances:
     `ssl_ca_cert`, `ssl_cl_cert`, `ssl_cl_cert_key`.
   - Minor fixes/improvements and some refactoring (see also above: *Core*...).
+- `intelmq.bots.collectors.shadowserver.collector_reports_api`:
+  - The 'json' option is no longer supported as the 'csv' option provides better performance (PR#2372 by elsif2).
+- `intelmq.bots.collectors.alienvault_otx.collector` (PR#2449 by qux-bbb):
+  - Fix modified_pulses_only is always False.
+
 
 #### Parsers
-  - `intelmq.bots.parsers.netlab_360.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
-  - `intelmq.bots.parsers.webinspektor.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
-  - `intelmq.bots.parsers.sucuri.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
+- `intelmq.bots.parsers.netlab_360.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
+- `intelmq.bots.parsers.webinspektor.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
+- `intelmq.bots.parsers.sucuri.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
+- `intelmq.bots.parsers.shadowserver._config`:
+  - Switch to dynamic configuration to decouple report schema changes from IntelMQ releases by regularly downloading them from the Shadowserver server (PR#2372 by elsif2).
 
 #### Experts
 - `intelmq.bots.experts.jinja` (PR#2417 by Mikk Margus Möll):
@@ -100,6 +108,8 @@
 - Update/fix/improve the stuff related to the STOMP bots and integration with the *n6*'s
   Stream API (PR#2408 and PR#2414 by Jan Kaliszewski).
 - Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorný).
+- Adds warning banner if not browsing the latest version of the docs (PR#2445 by Filip Pokorný).
+- Fix logo path in index.md when building the docs (PR#2445 by Filip Pokorný).
 
 ### Packaging
 - Add `pendulum` to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner).
@@ -159,13 +169,10 @@
   - added support for `Subject NOT LIKE` queries,
   - added support for multiple values in ticket subject queries.
 - `intelmq.bots.collectors.rsync`: Support for optional private key, relative time parsing for the source path, extra rsync parameters and strict host key checking (PR#2241 by Mateo Durante).
-- `intelmq.bots.collectors.shadowserver.collector_reports_api`:
-  - The 'json' option is no longer supported as the 'csv' option provides better performance.
 
 #### Parsers
 - `intelmq.bots.parsers.shadowserver._config`:
   - Reset detected `feedname` at shutdown to re-detect the feedname on reloads (PR#2361 by @elsif2, fixes #2360).
-  - Switch to dynamic configuration to decouple report schema changes from IntelMQ releases. 
   - Added 'IPv6-Vulnerable-Exchange' alias and 'Accessible-WS-Discovery-Service' report. (PR#2338)
   - Removed unused `p0f_genre` and `p0f_detail` from the 'DNS-Open-Resolvers' report. (PR#2338)
   - Added 'Accessible-SIP' report. (PR#2348)

docs/_overrides/main.html

@@ -0,0 +1,11 @@
+{#SPDX-FileCopyrightText: 2023 Filip Pokorný#}
+{#SPDX-License-Identifier: AGPL-3.0-or-later#}
+
+{% extends "base.html" %}
+
+{% block outdated %}
+  You're not viewing the latest version.
+  <a href="{{ '../' ~ base_url }}" style="color: black">
+    <strong>Click here to go to latest.</strong>
+  </a>
+{% endblock %}

docs/index.md

@@ -7,7 +7,7 @@
 [![CII Badge](https://bestpractices.coreinfrastructure.org/projects/4186/badge)](https://bestpractices.coreinfrastructure.org/projects/4186/)
 -->
 
-![IntelMQ](/docs/static/images/Logo_Intel_MQ.svg)
+![IntelMQ](docs/static/images/Logo_Intel_MQ.svg)
 
 # Introduction
 

docs/user/bots.md

@@ -2128,23 +2128,23 @@ correct mapping of the columns:
 
 1. **Automatic report type detection**
 
-   Since IntelMQ version 2.1 the parser can detect the feed based on metadata provided by the collector.
-
-   When processing a report, this bot takes `extra.file_name` from the report and looks in `config.py` how the report
-   should be parsed. If this lookup is not possible, and the `feedname` is not given as parameter, the feed cannot be
-   parsed.
-
-   The field `extra.file_name` has the following structure: `%Y-%m-%d-${report_name}[-suffix].csv` where the optional
-   suffix can be something like `country-geo`. For example, some possible filenames
-   are `2019-01-01-scan_http-country-geo.csv` or `2019-01-01-scan_tftp.csv`. The important part is the `report_name`,
-   between the date and the suffix. Since version 2.1.2 the date in the filename is optional, so filenames
-   like `scan_tftp.csv` are also detected.
+    Since IntelMQ version 2.1 the parser can detect the feed based on metadata provided by the collector.
+    
+    When processing a report, this bot takes `extra.file_name` from the report and looks in `config.py` how the report
+    should be parsed. If this lookup is not possible, and the `feedname` is not given as parameter, the feed cannot be
+    parsed.
+    
+    The field `extra.file_name` has the following structure: `%Y-%m-%d-${report_name}[-suffix].csv` where the optional
+    suffix can be something like `country-geo`. For example, some possible filenames
+    are `2019-01-01-scan_http-country-geo.csv` or `2019-01-01-scan_tftp.csv`. The important part is the `report_name`,
+    between the date and the suffix. Since version 2.1.2 the date in the filename is optional, so filenames
+    like `scan_tftp.csv` are also detected.
 
 2. **Fixed report type**
 
-   If the method above is not possible and for upgraded instances, the report type can be set with the `feedname`
-   parameter. Report type is derived from the subject of Shadowserver e-mails. A list of possible values of
-   the `feedname` parameter can be found in the table below in the column "Report Type".
+    If the method above is not possible and for upgraded instances, the report type can be set with the `feedname`
+    parameter. Report type is derived from the subject of Shadowserver e-mails. A list of possible values of
+    the `feedname` parameter can be found in the table below in the column "Report Type".
 
 **Module:**
 
@@ -2200,6 +2200,7 @@ The schema revision history is maintained at https://github.com/The-Shadowserver
       auto_update: true
     run_mode: continuous
 ```
+
 ---
 
 ### Shodan <div id="intelmq.bots.parsers.shodan.parser" />

intelmq/bin/intelmqctl.py

@@ -87,6 +87,11 @@ def __init__(self, interactive: bool = False, returntype: ReturnType = ReturnTyp
         self._parameters.logging_handler = 'file'
         self._parameters.logging_path = DEFAULT_LOGGING_PATH
 
+        try:
+            self._runtime_configuration = utils.load_configuration(RUNTIME_CONF_FILE)
+        except ValueError as exc:  # pragma: no cover
+            self.abort(f'Error loading {RUNTIME_CONF_FILE!r}: {exc}')
+
         # Try to get logging_level from defaults configuration, else use default (defined above)
         defaults_loading_exc = None
         try:
@@ -203,11 +208,6 @@ def __init__(self, interactive: bool = False, returntype: ReturnType = ReturnTyp
     intelmqctl debug --get-environment-variables
 '''
 
-        try:
-            self._runtime_configuration = utils.load_configuration(RUNTIME_CONF_FILE)
-        except ValueError as exc:  # pragma: no cover
-            self.abort(f'Error loading {RUNTIME_CONF_FILE!r}: {exc}')
-
         self._processmanagertype = getattr(self._parameters, 'process_manager', 'intelmq')
         if self._processmanagertype not in process_managers():
             self.abort('Invalid process manager given: %r, should be one of %r.' '' % (self._processmanagertype, list(process_managers().keys())))
@@ -384,7 +384,8 @@ def __init__(self, interactive: bool = False, returntype: ReturnType = ReturnTyp
             )
 
     def load_defaults_configuration(self, silent=False):
-        for option, value in utils.get_global_settings().items():
+        global_settings = self._runtime_configuration.get('global', {})
+        for option, value in global_settings.items():
             setattr(self._parameters, option, value)
 
         # copied from intelmq.lib.bot, should be refactored to e.g. intelmq.lib.config

intelmq/bots/collectors/alienvault_otx/collector.py

@@ -26,10 +26,6 @@ def init(self):
         if OTXv2 is None:
             raise MissingDependencyError("OTXv2")
 
-        self.modified_pulses_only = False
-        if hasattr(self, 'modified_pulses_only'):
-            self.modified_pulses_only = self.modified_pulses_only
-
     def process(self):
         self.logger.info("Downloading report through API.")
         otx = OTXv2(self.api_key, proxy=self.https_proxy)

intelmq/bots/parsers/shadowserver/_config.py

@@ -126,7 +126,7 @@ def enable_auto_update(enable):
     __config.auto_update = enable
 
 
-def get_feed_by_feedname(given_feedname: str) -> Optional[Dict[str, Any]]:
+def get_feed_by_feedname(given_feedname: str) -> Optional[Tuple[str, Dict[str, Any]]]:
     return __config.feedname_mapping.get(given_feedname, None)
 
 

intelmq/bots/parsers/shadowserver/parser.py

@@ -63,6 +63,7 @@ def init(self):
         if self.feedname is not None:
             self._sparser_config = config.get_feed_by_feedname(self.feedname)
             if self._sparser_config:
+                self._sparser_config = self._sparser_config[1]
                 self.logger.info('Using fixed feed name %r for parsing reports.' % self.feedname)
                 self._mode = 'fixed'
             else:

mkdocs.yml

@@ -19,6 +19,7 @@ site_dir: docs_build
 theme:
   name: material
   logo: static/images/Logo_Intel_MQ.svg
+  custom_dir: docs/_overrides
   palette:
     # Palette toggle for light mode
     - scheme: default