Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle type parameter as a string and match by type instead of filena… #2495

Merged
merged 3 commits into from
May 5, 2024

Conversation

elsif2
Copy link
Collaborator

@elsif2 elsif2 commented Apr 18, 2024

The current version expects the types parameter to be a list:

parameters:
      types: [blocklist]

If the parameter is provided as a string instead all report types are matched.

This update adds support for the types parameter to be either a string or a list.

This update also utilizes the type field returned by the API to match the requested types instead of a sub-string match on the filename.

@elsif2 elsif2 linked an issue Apr 18, 2024 that may be closed by this pull request
@elsif2 elsif2 assigned sebix and unassigned sebix Apr 18, 2024
Copy link
Member

@sebix sebix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_____________ TestShadowServerAPICollectorBot.test_report_content ______________
...
ERROR    test-bot:bot.py:406 Bot has found a problem.
Traceback (most recent call last):
  File "/home/runner/work/intelmq/intelmq/intelmq/lib/bot.py", line 363, in start
    self.process()
  File "/home/runner/work/intelmq/intelmq/intelmq/bots/collectors/shadowserver/collector_reports_api.py", line 134, in process
    reportslist = self._reports_list()
  File "/home/runner/work/intelmq/intelmq/intelmq/bots/collectors/shadowserver/collector_reports_api.py", line 114, in _reports_list
    reports = [report for report in reports if any(report['type'] == rtype for rtype in self.types)]
  File "/home/runner/work/intelmq/intelmq/intelmq/bots/collectors/shadowserver/collector_reports_api.py", line 114, in <listcomp>
    reports = [report for report in reports if any(report['type'] == rtype for rtype in self.types)]
  File "/home/runner/work/intelmq/intelmq/intelmq/bots/collectors/shadowserver/collector_reports_api.py", line 114, in <genexpr>
    reports = [report for report in reports if any(report['type'] == rtype for rtype in self.types)]
KeyError: 'type'

Can you please also adapt the docs and add a short changelog entry?
Thanks!

@elsif2
Copy link
Collaborator Author

elsif2 commented Apr 18, 2024

The documentation looks fine as-is:

(optional, string/array of strings) An array of strings (or a list of comma-separated values) with the names of report types you want to process. If you leave this empty, all the available reports will be downloaded and processed (i.e. 'scan', 'drones', 'intel', 'sandbox_connection', 'sinkhole_combined'). The possible report types are equivalent to the file names defined the the schema. Please see the Supported Reports of the Shadowserver parser for details.

@elsif2 elsif2 requested a review from sebix April 18, 2024 16:07
@sebix sebix added this to the 3.3.1 milestone May 5, 2024
@sebix sebix added bug Indicates an unexpected problem or unintended behavior component: bots labels May 5, 2024
@sebix sebix merged commit bc941c6 into develop May 5, 2024
31 checks passed
@sebix sebix deleted the 2493-shadowserver-api-bot branch May 5, 2024 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Indicates an unexpected problem or unintended behavior component: bots
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Shadowserver API bot
2 participants