From 03c4e0a68362924f0778e50e5529150ce170994d Mon Sep 17 00:00:00 2001 From: aaronkaplan Date: Thu, 29 Feb 2024 21:25:32 +0100 Subject: [PATCH 01/12] update NEWS.md in preparation for 3.3.0 --- CHANGELOG.md | 2 +- NEWS.md | 63 ++++++++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 55 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 07af41a83..a1ed8a2a0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ -3.3.0 (unreleased) +3.3.0 ------------------ ### Configuration diff --git a/NEWS.md b/NEWS.md index eb77d3d2d..f0ae78b4c 100644 --- a/NEWS.md +++ b/NEWS.md @@ -10,24 +10,69 @@ This file lists all changes which have an affect on the administration of IntelM Please refer to the change log for a full list of changes. -3.3.0 Feature release (unreleased) ---------------------------------- +3.3.0 Feature release +--------------------- ### Documentation -The documentation is now available at [docs.intelmq.org](https://docs.intelmq.org/). Documentation has been updated and restructured into User, Administrator and Developer Guide. It provides modern look with various quality of life improvements. +The documentation is now available at [docs.intelmq.org](https://docs.intelmq.org/). Documentation has been updated and restructured into User, Administrator and Developer Guide. It provides modern look with various quality of life improvements. Big thanks to to @gethvi. +We now have a slick, modern mkdocs based documentation. Please do check it out! -### Requirements -### Tools +### Bots +#### Shadowserver dynamic parser / collector -### Data Format +**Note well**: if you use shadowserver feeds, **please read this section carefully**. -### Configuration +Thanks to shadowserver (@elsif2), we have a new dynamic shadowserver reports API integration. What does it do? +It connects to the [Shadowserver API](https://www.shadowserver.org/what-we-do/network-reporting/api-documentation/), +requests a list of all the reports for a specific country and processes the ones that are new. -### Libraries +Motivation for this change: -### Postgres databases +Shadowserver adds new scans on a nearly weekly basis. IntelMQ's release cycle and the need for a stable release could not keep up with this high intensity of shadowserver parser changes. +We therefore (thanks to @eslif2) move the shadowserver reports collector and parser to a new, dynamic system. It can: + + - fetch the shadowserver schema from shadowserver (https://interchange.shadowserver.org/intelmq/v1/schema) + - dynamically collect new reports (see also https://docs.intelmq.org/latest/user/bots/?h=shadow#shadowserver-reports-api) + - parse the new reports + +**Note well**: if your IntelMQ system runs in an airgapped environment or if it may only reach out to specific IPs/sites, you should read the notes here: +https://docs.intelmq.org/latest/user/bots/#shadowserver. +You will need to download shadowserver-schema.json periodically yourself in this case. + +**Note well:**: since dynamic changes are a bit tricky, we defined that there is a schema contract: + +> Schema contract +> +> Once set in the schema, the classification.identifier, classification.taxonomy, and classification.type fields will remain static for a specific report. + +This makes things deterministic again. + +#### Alienvault OTX + +Fix of a bug where a certain condition would have always evaluated to False. (PR#2449 by qux-bbb. Thanks) + +#### AMQP +Quite a few changes (thanks to Kamil, @gethvi) on AMQP + +#### Obsoleted bots + +- `intelmq.bots.parsers.netlab_360.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný) +- `intelmq.bots.parsers.webinspektor.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný) +- `intelmq.bots.parsers.sucuri.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný) + +### General changes and bug fixes + +Digital Trust Center fixed a bug where the config was loaded twice in intelmqctl which created quite some speedups. Thanks! +This speeds up IntelMQ API calls. + +### Data Format + +Shadowserver dynamic parser (see above). + +### General remarks +The full list of changes can be seen in the CHANGELOG.md file. 3.2.1 Bugfix release (2023-08-23) --------------------------------- From 631d897a5082642df6c1d560e8ec87275018c147 Mon Sep 17 00:00:00 2001 From: aaronkaplan Date: Thu, 29 Feb 2024 21:44:27 +0100 Subject: [PATCH 02/12] preparing release --- debian/changelog | 4 ++-- intelmq/version.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/debian/changelog b/debian/changelog index 19c1c38a6..8ccfa35e1 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,6 +1,6 @@ -intelmq (3.2.2a1-1) UNRELEASED; urgency=medium +intelmq (3.3.0-1) stable; urgency=medium - * + * 3.3.0 Feature release -- Sebastian Wagner Tue, 05 Sep 2023 15:05:07 +0200 diff --git a/intelmq/version.py b/intelmq/version.py index 45127749e..f6b97c93d 100644 --- a/intelmq/version.py +++ b/intelmq/version.py @@ -2,5 +2,5 @@ # # SPDX-License-Identifier: AGPL-3.0-or-later -__version_info__ = (3, 2, 2, 'a1') +__version_info__ = (3, 3, 0, '') __version__ = '.'.join(map(str, __version_info__)) From c3e9e12b06304c8a8d844d754e20ccec6126f2ca Mon Sep 17 00:00:00 2001 From: aaronkaplan Date: Thu, 29 Feb 2024 22:40:36 +0100 Subject: [PATCH 03/12] add release date --- CHANGELOG.md | 2 +- NEWS.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a1ed8a2a0..28c4f4c77 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ -3.3.0 +3.3.0 (2024-03-01) ------------------ ### Configuration diff --git a/NEWS.md b/NEWS.md index f0ae78b4c..fc678a421 100644 --- a/NEWS.md +++ b/NEWS.md @@ -10,7 +10,7 @@ This file lists all changes which have an affect on the administration of IntelM Please refer to the change log for a full list of changes. -3.3.0 Feature release +3.3.0 Feature release (2024-03-01) --------------------- ### Documentation From c5352e48befccf369c0d35f4a02e4c873cbde7e5 Mon Sep 17 00:00:00 2001 From: aaronkaplan Date: Thu, 29 Feb 2024 22:41:39 +0100 Subject: [PATCH 04/12] add release date again --- debian/changelog | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian/changelog b/debian/changelog index 8ccfa35e1..db5441263 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,4 +1,4 @@ -intelmq (3.3.0-1) stable; urgency=medium +intelmq (3.3.0-1) stable; urgency=medium Fr, 01 Mar 2024 00:00:00 +0200 * 3.3.0 Feature release From 2d7ef2e806b34ee227032b454ad2a05e241893fb Mon Sep 17 00:00:00 2001 From: aaronkaplan Date: Thu, 29 Feb 2024 22:42:28 +0100 Subject: [PATCH 05/12] add -1 --- intelmq/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/intelmq/version.py b/intelmq/version.py index f6b97c93d..7e9cc7e9e 100644 --- a/intelmq/version.py +++ b/intelmq/version.py @@ -2,5 +2,5 @@ # # SPDX-License-Identifier: AGPL-3.0-or-later -__version_info__ = (3, 3, 0, '') +__version_info__ = (3, 3, 0, '1') __version__ = '.'.join(map(str, __version_info__)) From cf3a50d35c56a7a05ca9283e1ad0e0edc7dc7a94 Mon Sep 17 00:00:00 2001 From: aaronkaplan Date: Thu, 29 Feb 2024 22:43:02 +0100 Subject: [PATCH 06/12] following sebix's advice --- intelmq/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/intelmq/version.py b/intelmq/version.py index 7e9cc7e9e..ccb58cc79 100644 --- a/intelmq/version.py +++ b/intelmq/version.py @@ -2,5 +2,5 @@ # # SPDX-License-Identifier: AGPL-3.0-or-later -__version_info__ = (3, 3, 0, '1') +__version_info__ = (3, 3, 0) __version__ = '.'.join(map(str, __version_info__)) From 267980a3e19c1ef9bcec829f93945b0895e369a5 Mon Sep 17 00:00:00 2001 From: aaronkaplan Date: Thu, 29 Feb 2024 22:45:09 +0100 Subject: [PATCH 07/12] wrong debian/changelog --- debian/changelog | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/debian/changelog b/debian/changelog index db5441263..0a7215baa 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,8 +1,8 @@ -intelmq (3.3.0-1) stable; urgency=medium Fr, 01 Mar 2024 00:00:00 +0200 +intelmq (3.3.0-1) stable; urgency=medium * 3.3.0 Feature release - -- Sebastian Wagner Tue, 05 Sep 2023 15:05:07 +0200 + -- Aaron Kaplan Fr, 01 Mar 2024 00:00:00 +0200 intelmq (3.2.1-2) stable; urgency=medium From bc8968d23b6358905081ca6a57800a31fb266125 Mon Sep 17 00:00:00 2001 From: aaronkaplan Date: Thu, 29 Feb 2024 22:49:20 +0100 Subject: [PATCH 08/12] Place for documenting the list of supported OSes changed --- docs/dev/release.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dev/release.md b/docs/dev/release.md index 25e0dd753..05775decb 100644 --- a/docs/dev/release.md +++ b/docs/dev/release.md @@ -17,7 +17,7 @@ different. - Check the upgrade functions in `intelmq/lib/upgrades.py`. - Close the milestone on GitHub and move any open issues to the next one. -- `docs/user/installation.rst`: Update supported operating systems. +- `docs/admin/installation/linux-packages.md`: Update supported operating systems. ## Documentation From 3064c2c1101846fdf0f23fbd452d73e652885d9b Mon Sep 17 00:00:00 2001 From: aaronkaplan Date: Thu, 29 Feb 2024 22:59:34 +0100 Subject: [PATCH 09/12] fix docu bug. wrong path --- docs/dev/release.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dev/release.md b/docs/dev/release.md index 05775decb..976b0a521 100644 --- a/docs/dev/release.md +++ b/docs/dev/release.md @@ -39,7 +39,7 @@ Eventually adapt the default log levels if necessary. Should be INFO for stable ### IntelMQ Manager - `intelmq_manager/version.py`: Update the version. -- `intelmq_manager/static/images/js/about.js`: Update the version. +- `intelmq_manager/static/js/about.js`: Update the version. ## Commit, push, review and merge From 4db2c5cb68318f3e59f4129ea741fdc5ffca20e7 Mon Sep 17 00:00:00 2001 From: aaronkaplan Date: Fri, 1 Mar 2024 11:30:54 +0100 Subject: [PATCH 10/12] update docs. The `master` branch is gone. We only have `develop` as main/master now. --- docs/dev/release.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/dev/release.md b/docs/dev/release.md index 976b0a521..7d9983b04 100644 --- a/docs/dev/release.md +++ b/docs/dev/release.md @@ -43,18 +43,17 @@ Eventually adapt the default log levels if necessary. Should be INFO for stable ## Commit, push, review and merge -Commit your changes in a separate branch, the final commit message -should start with `REL:`. Push and create a pull request to maintenance -and after that from maintenance to master. Someone else should review -the changes. Eventually fix them, make sure the `REL:` is the last -commit, you can also push that one at last, after the reviews. +Commit your changes in a separate branch, the final commit message should start +with `REL:`. Push and create a pull request to the `develop` branch. Someone +else should review the changes. Eventually fix them, make sure the `REL:` is +the last commit, you can also push that one at last, after the reviews. Why a separate branch? Because if problems show up, you can still force-push to that one, keeping the release commit the latest one. ## Tag and release -Tag the commit with `git tag -s version HEAD`, merge it into master, +Tag the commit with `git tag -s version HEAD`, merge it into `develop`, push the branches *and* the tag. The tag is just `a.b.c`, not prefixed with `v` (that was necessary only with SVN a long time ago...). From d2330abebc42edc7b719823259735141d26a92a2 Mon Sep 17 00:00:00 2001 From: aaronkaplan Date: Fri, 1 Mar 2024 14:21:48 +0100 Subject: [PATCH 11/12] timestamp formatting --- debian/changelog | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian/changelog b/debian/changelog index 0a7215baa..ac9ddbe81 100644 --- a/debian/changelog +++ b/debian/changelog @@ -2,7 +2,7 @@ intelmq (3.3.0-1) stable; urgency=medium * 3.3.0 Feature release - -- Aaron Kaplan Fr, 01 Mar 2024 00:00:00 +0200 + -- Aaron Kaplan Fr, 01 Mar 2024 14:19:00 +0100 intelmq (3.2.1-2) stable; urgency=medium From 3ec669d0d674a606d0c62a8def0f8b27a7953e87 Mon Sep 17 00:00:00 2001 From: aaronkaplan Date: Fri, 1 Mar 2024 17:25:32 +0100 Subject: [PATCH 12/12] REL: release 3.3.0 --- docs/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/index.md b/docs/index.md index cf097897c..1b55da37f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -9,6 +9,7 @@ ![IntelMQ](docs/static/images/Logo_Intel_MQ.svg) + # Introduction **IntelMQ** is a solution for IT security teams (CERTs & CSIRTs, SOCs