Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Added new expert plugin - extra_tags #2476

Closed
wants to merge 1 commit into from

Conversation

manuel-subredu
Copy link

@manuel-subredu manuel-subredu commented Mar 7, 2024

  • added a new plugin that allows manipulation of extra_tags. Main use case
    is to allow usage of same pipeline for both dev and prod setups but be able
    to differentiate between messages and select a different path for your
    dev stage (ex: route dev messages to local directory for examination rather
    then production database)

Description

ExtraTags expert plugin aims to ease the testing of an existing pipeline and allow
easy routing of the messages in development phase. Main use case:

  1. use extra_tags plugin right after a parser (or input) and mark the messages as
    beloning to development stage {'stage': 'dev'}
  2. insert a prod-dev-routing decision right before final output and with filter
    plugin route non-production messages to various backends to inspection

Testing

(.venv) ➜  intelmq git:(develop) ✗ pytest intelmq/tests/bots/experts/extra_tags -s
=========================================================================================================== test session starts ============================================================================================================
platform darwin -- Python 3.12.0, pytest-8.0.2, pluggy-1.4.0
rootdir: ws/extra-tags/intelmq
configfile: pytest.ini
plugins: cov-4.1.0, requests-mock-1.11.0
collected 5 items

intelmq/tests/bots/experts/extra_tags/test_expert.py ...True
..

---------- coverage: platform darwin, python 3.12.0-final-0 ----------
Coverage XML written to file coverage.xml


============================================================================================================ 5 passed in 2.01s =============================================================================================================
(.venv) ➜  intelmq git:(develop)

* added a new plugin that allows manipulation of extra_tags. Main use case
  is to allow usage of same pipeline for both dev and prod setups but be able
  to differentiate between messages and select a different path for your
  dev stage (ex: route dev messages to local directory for examination rather
  then production database)
@manuel-subredu manuel-subredu marked this pull request as ready for review March 7, 2024 04:41
@kamil-certat
Copy link
Contributor

Hey, thanks for the commitment! I've looked through your code, and I think it's somehow similar to the sieve bot. Have you tried it before? I feel like it may be very near to accomplish what you want, although I'm not sure if it supports manipulating a dict inside a field.

May I ask you to have a look at the sieve bot, and see if it's either already solve the problem, or would be relatively easily possible to extend it? I think it may be nice to extend it as it has much more features to manipulate an event :)

@manuel-subredu
Copy link
Author

Mea culpa ! I thought that Sieve is somehow related to mail area and didn't looked into the specifics. You are right, sieve expert includes the functionality I actually needed. I think we can drop this pull request. Sorry for the noise.

@kamil-certat
Copy link
Contributor

No problem!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants