REL: 1.2.3: CSV special case corrections, extra data in notifications, documentation

• Corrections to handling of extra data in mailgen template preview
• Corrections for special cases in CSV data
• Extended Documentation

Backend

• convert example data to Event to handle extra data in database insert:
  for the mailgen preview, convert the example data to an IntelMQ event
  and before submitting it to the database, to a dict with extra as a
  string.
  Previously, extra-data provided by the user could not be inserted,
  raising an error

Frontend

• frontend: detect if csv data header can be used as field name and use it:
  if the csv column name (table header) contains a legitimate IntelMQ
  field name, use this one and skip the further sanitation steps
• frontend: handle CSV data with no body, but only with a header:
  if the CSV data contains just one line and the header option is active,
  the data is body/content-less
  handle these special cases by aborting the parse and disabling the
  spinner

Documentation

• docs: document format spec (mailgen) influence
• docs: change URLs to rendered sphinx docs in README

1.2.2: Validate Template Preview Data

The user data, (first line of parsed input data), used for the template content validation and preview, validated (sanitized) by IntelMQ before using it for the template preview, otherwise syntactically incorrect values could lead to PostgreSQL errors. Invalid fields are ignored and replaced by example data.

1.2.0 Single Template Input

Frontend

• Redesign to a single template input. The multi-template editor is hidden and can be activated with the parameter mailgen_multi_templates_enabled
• Pass assigned columns to the backend for notification preview and data submission
• Minor style enhancement in email preview
• Minor: add more spacing between the explanation text and the subject
• Target groups: add select all/none buttons
• Separate function for converting data to table

Backend

• Allow single template input and pass to mailgen
• Do not send data to the redis queue if validation/submission with bots is active (would be a double-submit)
• Refactor format_spec creation, more checks/sanitation
• Template preview: use first line of data instead of fully relying on example data, use the first line of input data if available
• Use assigned columns as default table format: the columns assigned by the user in the frontend are used as default table format for mailgen
• Compatibility changes on deprecated pkg_resources / importlib replacement, make it compatible on both Ubuntu 20.04 and 22.04, update tests and GitHub Workflows

Documentation

• Add section about the behaviour of the Frontend's "Submit/Validate with Bots" parameter

1.1.0 UI Enhancements

Frontend

• Notification preview: decode quoted-printable email body
• Change the display of validation errors in table
  • color the row orange for errors in this row,
  • color a cell red for an error in this cell
  • color the Actions-cell (first column) red for an unspecific error (e.g. missing field) and show a tooltip
  • show the tooltip for cells only for errors on this cell
• Make Table more compact (smaller padding)
• Show small icon to show processed row instead of long text
• Add warning on duplicate field assignments after detection
• Link docs in navbar
• Clean field names for defaults from CSV header and for entered field names:
  • trim (remove surrounding whitespace)
  • all lower case
  • replace whitespaces with underscore
  • remove all other disallowed characters
• Field assignment selection: select on tab
• packages:
  • upgrade csv parser papaparse
  • replace obsolete babel polyfill with core-js
  • remove unused requirements mutationobserver-shim, popper.js, portal-vue

Backend

• Data validation: group error messages for validation per field name in a row
• format row-process error log with newlines for easier readability
• only parse time-field if value is not empty
• read configfile with encoding

Package

• build-depend on hug to run the tests during build
• set NODE_OPTIONS=--openssl-legacy-provider
• require either yarn or yarnpkg
• package dependency fixes

Tests

• Add GitHub workflow to build debian packages
• Add GitHub workflow for pycodestyle

Documentation

• Add GitHub workflow to build and publish documentation
• Ass missing change log sections for 1.0.0, 1.0.1, 1.0.2
• Add screenshots
• Switch from single Markdown files Sphinx

1.0.2 Bugfix release

backend: fix method arguments causing falcon errors
backend: remove obsolete, long-running DB queries
backend: fix falcon status code calls
client: fix wrong modal on mailgen log button

1.0.1: Template validations, UI optimizations

template name validation on name change
backend: verbose error message for empty template in preview
client: improve default column field names
client: template deletion safety question
client+backend: live feedback template syntax check
frontend, mailgen: optimize UI components placement
frontend: disable classification fields in dryrun

1.0.0 Mailgen integration

This is the first stable release with complete integration of the IntelMQ Mailgen workflow