Merge branch 'develop' into shadowserver-dynamic-config

CHANGELOG.md

@@ -33,6 +33,7 @@
   `intelmq.bots.collectors.stomp.collector` and `intelmq.bots.outputs.stomp.output`
   (PR#2408 and PR#2414 by Jan Kaliszewski).
 - `intelmq.lib.upgrades`: Replace deprecated instances of `url2fqdn` experts by the new `url` expert in runtime configuration (PR#2432 by Sebastian Wagner).
+- `intelmq.lib.bot`: Ensure closing log files on reloading (PR#2435 by Kamil Mankowski).
 
 ### Development
 - Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).
@@ -87,6 +88,8 @@
   - Minor fixes/improvements and some refactoring (see also above: *Core*...).
 - `intelmq.bots.outputs.stomp.output` (PR#2423 by Kamil Mankowski):
   - Try to reconnect on `NotConnectedException`.
+- `intelmq.bots.outputs.smtp_batch.output` (PR #2439 by Edvard Rejthar):
+  - Fix ability to send with the default `bcc`
 
 ### Documentation
 - Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
@@ -104,7 +107,9 @@
 ### Tests
 
 ### Tools
- - `intelmq_psql_initdb` got support for providing custom harmonization file, generating view for storing `raw` fields separately, and adding `IF NOT EXISTS`/`OR REPLACE` clauses ([PR#2404](https://github.com/certtools/intelmq/pull/2404) by Kamil Mankowski).
+ - `intelmq_psql_initdb`:
+   -  got support for providing custom harmonization file, generating view for storing `raw` fields separately, and adding `IF NOT EXISTS`/`OR REPLACE` clauses ([PR#2404](https://github.com/certtools/intelmq/pull/2404) by Kamil Mankowski).
+   -  got support for generating JSONB fields for PostgreSQL schema (PR#2436 by Kamil Mankowski).
 
 ### Contrib
 

docs/user/bots.md

@@ -4956,6 +4956,132 @@ rpz.yourdomain.eu. *.secondmaliciousdomain.com CNAME rpz.yourdomain.eu.
 
 ---
 
+### SMTP Batch <div id="intelmq.bots.outputs.smtp_batch.output" />
+
+Aggregate events by e-mail addresses in the `source.abuse_contact` field and batch send them at once as a zipped CSV file attachment in a GPG signed message.
+
+When the bot is run normally by IntelMQ, it just aggregates the events for later use into a custom Redis database.
+If run through CLI (by a cron or manually), it shows e-mail messages that are ready to be sent and let you send them to the tester's e-mail OR to abuse contact e-mails.
+E-mails are sent in a zipped CSV file, delimited by a comma, while keeping strings in double quotes.
+Note: The field "raw" gets base64 decoded if possible. Bytes `\n` and `\r` are replaced with "\n" and "\r" strings in order to guarantee best CSV files readability both in Microsoft Office and LibreOffice. (A multiline string may be stored in "raw" which completely confused Microsoft Excel.)
+
+Launch it like this:
+```
+</usr/local/bin executable> <bot-id> cli [--tester tester's email]
+```
+Example:
+```bash
+intelmq.bots.outputs.smtp_batch.output smtp-batch-output --cli --tester [email protected]
+```
+
+CLI flags:
+```
+-h, --help            show this help message and exit
+--cli                 initiate CLI interface
+--tester TESTING_TO   tester's e-mail
+--ignore-older-than-days IGNORE_OLDER_THAN_DAYS
+                      1..n skip all events with time.observation older than 1..n day; 0 disabled (allow all)
+--gpg-key GPG_KEY     fingerprint of gpg key to be used
+--limit-results LIMIT_RESULTS
+                      Just send first N mails.
+--send                Sends now, without dialog.
+```
+
+You can schedule the batch sending easily with a cron script, I.E. put this into `crontab -e` of the `intelmq` user:
+
+```
+# Send the e-mails every day at 6 AM
+0 6 * * *  /usr/local/bin/intelmq.bots.outputs.smtp_batch.output smtp-batch-output-cz cli --ignore-older-than-days 4 --send > /tmp/intelmq-send.log
+```
+
+**Module:** `intelmq.bots.outputs.smtp_batch.output`
+
+**Parameters:**
+
+**`alternative_mails`**
+
+(optional, string) Path to CSV in the form `[email protected],[email protected]`. Needed when some of the recipients ask you to forward their e-mails to another address.
+
+**`attachment_name`**
+
+(optional, string)  Attachment file name for the outgoing messages. May contain date formatting like this `%Y-%m-%d`. Example: "events_%Y-%m-%d" will appear as "events_2022-12-01.zip". Defaults to "intelmq_%Y-%m-%d".
+
+**`bcc`**
+
+(optional, array of strings) An array of e-mails to be put in the `Bcc` field for every mail.
+
+**`email_from`**
+
+(required, string) Sender's e-mail of the outgoing messages.
+
+
+**`gpg_key`**
+
+(optional, string) The Key or the fingerprint of a GPG key stored in ~/.gnupg keyring folder.
+
+
+**`gpg_pass`**
+
+(optional, string) Password for the GPG key if needed.
+
+
+**`mail_template`**
+
+(required, string) Path to the file containing the body of the mail for the outgoing messages.
+
+
+**`ignore_older_than_days`**
+
+(optional, integer) Skips events with time.observation older than now-N. (If your queue gets stuck for a reason, you do not want to send old and probably already solved events.) Defaults to 0 (allow all).
+
+
+**`limit_results`**
+
+(optional, integer) Intended as a debugging option, allows loading just first N e-mails from the queue.
+
+
+**`redis_cache_db`**
+
+(required, integer)  Redis database used for event aggregation. As the databases < 10 are reserved for the IntelMQ core, recommended is a bigger number.
+
+
+**`redis_cache_host`**
+
+(required, string) Hostname of the Redis database.
+
+
+**`redis_cache_port`**
+
+(required, string)  Port of the Redis database.
+
+
+**`redis_cache_ttl`**
+
+(required, integer) TTL in seconds used for caching. Recommended 1728000 for 20 days.
+
+
+**`smtp_server`**
+
+(required, string/array/object) SMTP server information and credentials. See [SMTP parameter](https://github.com/CZ-NIC/envelope#sending) of the envelope module.
+
+Examples: 
+```yaml
+smtp_server: "mailer"
+smtp_server: {"host": "mailer", "port": 587, "user": "john", "password": "123"}
+smtp_server: ["mailer", 587, "john", "password"]
+```
+
+**`subject`**
+
+(required, string) Subject for the outgoing messages. May contain date formatting like this `%Y-%m-%d`. Example: "IntelMQ weekly warning (%d.%m.%Y)".
+
+
+**`testing_to`**
+
+(optional, string) Tester's e-mail.
+
+---
+
 ### SMTP <div id="intelmq.bots.outputs.smtp.output" />
 
 Sends a MIME Multipart message containing the text and the event as CSV for every single event.

intelmq/bin/intelmq_psql_initdb.py

@@ -132,7 +132,8 @@ def _generate_separated_raws_schema(fields: dict, partition_key: str) -> list:
 
 
 def generate(harmonization_file=HARMONIZATION_CONF_FILE, skip_events=False,
-             separate_raws=False, partition_key=None, skip_or_replace=False):
+             separate_raws=False, partition_key=None, skip_or_replace=False,
+             use_jsonb=False):
     FIELDS = {}
     sql_lines = []
 
@@ -170,7 +171,7 @@ def generate(harmonization_file=HARMONIZATION_CONF_FILE, skip_events=False,
         elif value['type'] == 'UUID':
             dbtype = 'UUID'
         elif value['type'] in ('JSON', 'JSONDict'):
-            dbtype = 'json'
+            dbtype = 'jsonb' if use_jsonb else 'json'
         else:
             raise ValueError('Unknown type %r.' % value['type'])
 
@@ -212,6 +213,8 @@ def main():
                         help="Path to the harmonization file")
     parser.add_argument("--skip-or-replace", default=False, action="store_true",
                         help="Add IF NOT EXISTS or REPLACE directive to created schemas")
+    parser.add_argument("--jsonb", default=False, action="store_true",
+                        help="Use JSONB type to represent dictionary fields")
     args = parser.parse_args()
 
     OUTPUTFILE = args.outputfile
@@ -229,6 +232,7 @@ def main():
                         separate_raws=args.separate_raws,
                         partition_key=args.partition_key,
                         skip_or_replace=args.skip_or_replace,
+                        use_jsonb=args.jsonb,
                         )
         print("INFO - Writing %s file" % OUTPUTFILE)
         fp.write(psql)

intelmq/bots/outputs/smtp_batch/output.py

@@ -246,9 +246,8 @@ def prepare_mails(self):
             lines.extend(json.loads(str(message, encoding="utf-8")) for message in messages)
 
             # prepare rows for csv attachment
-            threshold = datetime.datetime.now() - datetime.timedelta(
-                days=self.ignore_older_than_days) if getattr(self, 'ignore_older_than_days',
-                                                             False) else False
+            threshold = self.ignore_older_than_days and \
+                datetime.datetime.now() - datetime.timedelta(days=self.ignore_older_than_days)
 
             # TODO: worthy to generate on the fly https://github.com/certtools/intelmq/pull/2253#discussion_r1172779620
             fieldnames = set()
@@ -338,7 +337,7 @@ def build_mail(self, mail, send=False, override_to=None):
             return (Envelope(text)
                     .attach(path=mail.path, name=attachment_name + '.zip')
                     .from_(email_from).to(email_to)
-                    .bcc([] if intended_to else getattr(self, 'bcc', []))
+                    .bcc(not intended_to and self.bcc or [])
                     .subject(subject)
                     .smtp(self.smtp_server)
                     .signature(self.gpg_key, self.gpg_pass)

intelmq/lib/bot.py

@@ -308,10 +308,25 @@ def __handle_sighup(self):
             self.shutdown()  # disconnects, stops threads etc
         except Exception:
             self.logger.exception('Error during shutdown of bot.')
-        self.logger.handlers = []  # remove all existing handlers
+        self.__cleanup_logging_handlers()
         self.__sighup.clear()
         self.__init__(self.__bot_id_full, sighup_event=self.__sighup, standalone=self._standalone)
 
+    def __cleanup_logging_handlers(self):
+        # thread-safe removing of handlers and closing opened files
+        handlers_list = self.logger.handlers[:]
+        for handler in handlers_list:
+            try:
+                self.logger.removeHandler(handler)
+                # ensure all log files are closed to prevent caching them in RAM
+                if isinstance(handler, logging.FileHandler):
+                    handler.close()
+            except:
+                # Logger should still be safe to use even without handlers
+                # In addition, we do not want any side issue to break execution
+                #  - we are about to reinitialize logging.
+                self.logger.exception("Error while cleaning up logging handlers")
+
     def init(self):
         pass