feat: Shodan host-lookup task (passive recon via the Shodan SDK)#1233
feat: Shodan host-lookup task (passive recon via the Shodan SDK)#1233ocervell wants to merge 10 commits into
Conversation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
…puts Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
📝 WalkthroughWalkthroughThis PR adds a new Shodan integration to secator: a ChangesShodan Task Integration
Estimated code review effort: 3 (Moderate) | ~30 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant yielder
participant ShodanSDK
participant Mapper
Caller->>yielder: iterate(target, operation)
yielder->>yielder: resolve api_key (opt → CONFIG → env)
alt operation = host
yielder->>ShodanSDK: api.host(ip, history, minify)
ShodanSDK-->>yielder: host dict or APIError
yielder->>Mapper: _map_host(h)
else operation = dns
yielder->>ShodanSDK: api.dns.domain_info()
ShodanSDK-->>yielder: dns info or APIError
yielder->>Mapper: _map_dns(domain, info)
else operation = search
yielder->>ShodanSDK: api.search(query, limit)
ShodanSDK-->>yielder: matches
yielder->>Mapper: _map_banner(b)
end
Mapper-->>Caller: Ip, Subdomain, Port, Technology, Vulnerability, Tag, Record
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
secator/tasks/shodan.py (1)
139-142: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy liftUse a public-suffix-aware parser for
domain.The last-two-labels heuristic turns values like
foo.co.ukintoco.uk, soSubdomain.domainbecomes wrong for common multi-label suffixes. Please switch this helper to an existing PSL-aware parser/helper before downstream grouping depends on it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@secator/tasks/shodan.py` around lines 139 - 142, The _registered_domain helper in ShodanTask uses a naive last-two-labels split, which misclassifies domains with multi-label public suffixes. Update this static method to use an existing public-suffix-aware parser/helper already available in the codebase so it returns the correct registrable domain for cases like foo.co.uk, and keep downstream grouping logic pointed at this corrected domain value.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@secator/tasks/shodan.py`:
- Around line 16-29: The shodan task is using PythonRunner instead of the repo’s
Command task contract, so it should be rewritten to match other entries under
secator/tasks/. Update the shodan class to subclass Command and define the
expected cmd, input_type, output_types, install_cmd, and item_loaders surface
instead of the current PythonRunner-specific setup. Keep the existing task
metadata and options, but move any parsing logic into item_loaders so the task
integrates consistently with the framework.
- Line 22: The install hint in shodan task is too loose and can fetch
unsupported versions. Update the install command in shodan.py (the install_cmd
used by the Shodan task) so it matches the version constraint already declared
in pyproject.toml, pinning shodan to the supported <2 range. Keep the change
localized to the task’s install hint string so the built-in install path stays
aligned with the dependency policy.
- Around line 93-95: The Shodan host vulnerability branch in shodan.py currently
yields only the CVE ID from h.get('vulns'), dropping attached metadata. Update
the vuln handling in the same area as the host parsing logic that builds
Vulnerability objects so it mirrors the banner-path normalization, extracting
any Shodan metadata from each vuln entry and populating cvss_score and summary
on the yielded finding instead of discarding them.
In `@tests/unit/test_shodan.py`:
- Around line 21-25: The _run helper in test_shodan.py imports output_types
symbols it never uses, which triggers Flake8 F401. Remove the unused
secator.output_types import from _run and keep only the symbols actually
referenced by the test helper, leaving shodan._map_host and _load_fixture
unchanged.
---
Nitpick comments:
In `@secator/tasks/shodan.py`:
- Around line 139-142: The _registered_domain helper in ShodanTask uses a naive
last-two-labels split, which misclassifies domains with multi-label public
suffixes. Update this static method to use an existing public-suffix-aware
parser/helper already available in the codebase so it returns the correct
registrable domain for cases like foo.co.uk, and keep downstream grouping logic
pointed at this corrected domain value.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 7ddaa95f-965e-4a1c-b390-a591d93a77b5
📒 Files selected for processing (8)
pyproject.tomlsecator/config.pysecator/tasks/shodan.pysecator/utils_test.pytests/fixtures/shodan_output.jsontests/integration/inputs.pytests/integration/outputs.pytests/unit/test_shodan.py
| @task() | ||
| class shodan(PythonRunner): | ||
| """Passive host recon via the Shodan API (ports, services, CVEs, hostnames).""" | ||
| input_types = [HOST, IP] | ||
| output_types = [Ip, Subdomain, Port, Technology, Vulnerability, Tag] | ||
| tags = ['shodan', 'recon', 'osint', 'passive'] | ||
| install_cmd = 'pip install shodan' | ||
| opts = { | ||
| # Empty default + runtime fallback (never a CONFIG default — it would leak | ||
| # the configured key into the secator-api UI form, like the `ai` task). | ||
| 'api_key': {'type': str, 'default': '', 'help': 'Shodan API key (defaults to configured key)'}, | ||
| 'history': {'is_flag': True, 'default': False, 'help': 'Include historical (non-current) banners'}, | ||
| 'minify': {'is_flag': True, 'default': False, 'help': 'Only ports + general host info (no banners)'}, | ||
| } |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift
Use the repo’s Command task contract here.
This new tool is wired up as a PythonRunner, so it never defines the Command-style cmd/input_type/item_loaders surface the repo expects for new entries under secator/tasks/. Please align the implementation with that contract before more tasks copy this pattern.
As per coding guidelines, secator/tasks/*.py should “Use Command subclass in secator/tasks/ to integrate new tools, defining cmd, input_type, output_types, and install_cmd attributes” and “Implement custom output parsers via item_loaders in task Command subclasses”.
🧰 Tools
🪛 Ruff (0.15.20)
[warning] 19-19: Mutable default value for class attribute
(RUF012)
[warning] 20-20: Mutable default value for class attribute
(RUF012)
[warning] 21-21: Mutable default value for class attribute
(RUF012)
[warning] 23-29: Mutable default value for class attribute
(RUF012)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@secator/tasks/shodan.py` around lines 16 - 29, The shodan task is using
PythonRunner instead of the repo’s Command task contract, so it should be
rewritten to match other entries under secator/tasks/. Update the shodan class
to subclass Command and define the expected cmd, input_type, output_types,
install_cmd, and item_loaders surface instead of the current
PythonRunner-specific setup. Keep the existing task metadata and options, but
move any parsing logic into item_loaders so the task integrates consistently
with the framework.
Source: Coding guidelines
| input_types = [HOST, IP] | ||
| output_types = [Ip, Subdomain, Port, Technology, Vulnerability, Tag] | ||
| tags = ['shodan', 'recon', 'osint', 'passive'] | ||
| install_cmd = 'pip install shodan' |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Repo root: '; pwd
printf '\nFiles matching shodan task / dependency hints:\n'
git ls-files | rg '(^|/)(shodan|requirements|pyproject|setup|Pipfile|poetry|flake8|tasks/.*\.py)$'
printf '\n--- secator/tasks/shodan.py ---\n'
if [ -f secator/tasks/shodan.py ]; then
cat -n secator/tasks/shodan.py
fi
printf '\n--- dependency references to shodan ---\n'
rg -n --hidden --no-ignore-vcs '\bshodan\b|shodan < 2|shodan<2|pip install shodan' . \
-g '!**/.git/**' -g '!**/node_modules/**' -g '!**/dist/**' -g '!**/build/**' -g '!**/venv/**' -g '!**/.venv/**'Repository: freelabz/secator
Length of output: 11136
Pin the Shodan install hint to match the dependency. pyproject.toml already constrains shodan to <2, but secator/tasks/shodan.py:22 still advertises pip install shodan, so the built-in install path can pull an unsupported release.
Proposed fix
- install_cmd = 'pip install shodan'
+ install_cmd = 'pip install "shodan<2"'📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| install_cmd = 'pip install shodan' | |
| install_cmd = 'pip install "shodan<2"' |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@secator/tasks/shodan.py` at line 22, The install hint in shodan task is too
loose and can fetch unsupported versions. Update the install command in
shodan.py (the install_cmd used by the Shodan task) so it matches the version
constraint already declared in pyproject.toml, pinning shodan to the supported
<2 range. Keep the change localized to the task’s install hint string so the
built-in install path stays aligned with the dependency policy.
| for cve in (h.get('vulns') or []): | ||
| yield Vulnerability(name=cve, id=cve, matched_at=ip_str, ip=ip_str, | ||
| provider='shodan', confidence='low', tags=['shodan']) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Preserve Shodan host vuln metadata
h['vulns'] can include Shodan metadata, but this branch emits only the CVE ID. Mirror the banner-path normalization here so top-level findings keep cvss_score and summary instead of dropping them.
🧰 Tools
🪛 Flake8 (7.3.0)
[error] 95-95: continuation line under-indented for visual indent
(E128)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@secator/tasks/shodan.py` around lines 93 - 95, The Shodan host vulnerability
branch in shodan.py currently yields only the CVE ID from h.get('vulns'),
dropping attached metadata. Update the vuln handling in the same area as the
host parsing logic that builds Vulnerability objects so it mirrors the
banner-path normalization, extracting any Shodan metadata from each vuln entry
and populating cvss_score and summary on the yielded finding instead of
discarding them.
| def _run(self): | ||
| from secator.output_types import Ip, Subdomain, Port, Technology, Vulnerability, Tag | ||
| from secator.tasks.shodan import shodan | ||
| task = shodan.__new__(shodan) | ||
| return list(task._map_host(_load_fixture(), '10.0.0.1', 'host1.example.com')) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Drop the unused output_types import from _run().
Line 22 is unused and already trips Flake8 F401, so this helper won’t stay lint-clean as written.
As per coding guidelines, **/*.py should use Flake8 with the repo’s configured settings.
🧰 Tools
🪛 Flake8 (7.3.0)
[error] 22-22: 'secator.output_types.Ip' imported but unused
(F401)
[error] 22-22: 'secator.output_types.Subdomain' imported but unused
(F401)
[error] 22-22: 'secator.output_types.Port' imported but unused
(F401)
[error] 22-22: 'secator.output_types.Technology' imported but unused
(F401)
[error] 22-22: 'secator.output_types.Vulnerability' imported but unused
(F401)
[error] 22-22: 'secator.output_types.Tag' imported but unused
(F401)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/unit/test_shodan.py` around lines 21 - 25, The _run helper in
test_shodan.py imports output_types symbols it never uses, which triggers Flake8
F401. Remove the unused secator.output_types import from _run and keep only the
symbols actually referenced by the test helper, leaving shodan._map_host and
_load_fixture unchanged.
Sources: Coding guidelines, Linters/SAST tools
…lity + total tag)
Add shodan step (operation: dns, if: opts.passive) to domain_recon tasks._group so passive recon runs include Shodan DNS records alongside dnsx. Also fix two dispatch bugs found during smoke testing: - Add STRING to input_types so free-text search queries are not filtered out by the framework's type guard before reaching the task. - Move operation validation before the API-key check so unknown operations yield a clean 'Unknown Shodan operation' Error regardless of key config.
Update: multi-mode expansion (host + DNS + search)This PR now extends the host-lookup task into a single multi-mode
Refactor extracted a shared Verification: full unit suite 945 passed, Known minors (for your call — non-blocking)
🤖 Generated with Claude Code |
CI installs `.[dev,ai]` only, so the `shodan` SDK wasn't present in secator's pipx venv — the unit tests that import shodan (error-path + search) and the shodan integration task failed with "No module named 'shodan'". Add `shodan` to the install extras (mirrors how `ai`/litellm is included) so the task and its tests run in CI. Also remove the shodan entry from the integration inputs/outputs: a live `api.host()` integration run needs a real SHODAN_API_KEY (network), which CI has no secret for, so it can only ever FAILURE. The mocked unit tests are the offline coverage; a live integration test can be added later behind a key. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
…API_KEY) The test_tasks loop auto-discovers every registered task and falls back to the generic HOST input when a task has no INPUTS_TASKS entry, so simply dropping the shodan input still ran shodan on 'localhost' and failed without a live SHODAN_API_KEY. Skip it explicitly like msfconsole. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
There was a problem hiding this comment.
🧹 Nitpick comments (1)
secator/tasks/shodan.py (1)
255-258: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winNaive TLD parsing breaks on multi-label public suffixes.
_registered_domaintakes the last two labels, which is wrong for suffixes likeco.uk,com.au, etc. (e.g.foo.example.co.uk→co.uk). This feedsSubdomain.domainin_map_host/_run_search/_run_dnsand the suffix-stripping logic in_shodan_resolve(Line 140), so results can be grouped/resolved incorrectly for such domains.Please confirm whether the project already depends on (or is open to adding) a public-suffix-aware library such as
tldextractfor this; if a lightweight in-house heuristic is intentional for v1, consider documenting the limitation.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@secator/tasks/shodan.py` around lines 255 - 258, The _registered_domain helper in ShodanTask is using a naive last-two-label split, which breaks domain grouping and resolution for multi-label public suffixes like co.uk or com.au. Update _registered_domain (and any callers such as _map_host, _run_search, _run_dns, and the suffix stripping in _shodan_resolve) to use a public-suffix-aware approach, preferably via tldextract if the project can depend on it. If adding a dependency is not desired for v1, keep the current heuristic but document its limitation clearly where _registered_domain is defined.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@secator/tasks/shodan.py`:
- Around line 255-258: The _registered_domain helper in ShodanTask is using a
naive last-two-label split, which breaks domain grouping and resolution for
multi-label public suffixes like co.uk or com.au. Update _registered_domain (and
any callers such as _map_host, _run_search, _run_dns, and the suffix stripping
in _shodan_resolve) to use a public-suffix-aware approach, preferably via
tldextract if the project can depend on it. If adding a dependency is not
desired for v1, keep the current heuristic but document its limitation clearly
where _registered_domain is defined.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: a4fc0ba3-f9b8-4b92-aabf-cd410f322bf1
📒 Files selected for processing (7)
.github/actions/install/action.ymlsecator/configs/workflows/domain_recon.yamlsecator/tasks/shodan.pytests/fixtures/shodan_dns_output.jsontests/fixtures/shodan_search_output.jsontests/integration/test_tasks.pytests/unit/test_shodan.py
✅ Files skipped from review due to trivial changes (1)
- tests/fixtures/shodan_search_output.json
1 participant
Adds a
shodanPythonRunner task: a passive Shodan host lookup (api.host(ip)via the Shodan Python SDK) that maps the response to native, low-confidence secator findings.What it does
Given a target IP or hostname (hostnames resolved via
socket.gethostbyname), it emits:extra_data)confidence='low'confidence='low'Findings are low-confidence because Shodan data is from its last crawl, not actively verified (same convention as fuzzing tasks).
Scope (v1)
Host lookup only — no search/count/scan, no CIDR input (documented non-goals). The client init + mapping are factored so
searchcould be added later without a mode-dispatch rewrite.Config / deps
ShodanAddon(CONFIG.addons.shodan.api_key, mirrorsVulnersAddon); key resolution is opt → config →SHODAN_API_KEYenv. Theapi_keyopt defaults to''(never a CONFIG default — that would leak the key into the secator-api UI form).shodan < 2extras group (pip install secator[shodan]).Error handling
Missing key / import error →
Error(return);No information available→Warning+ zero findings; otherAPIError→Error; DNS failure →Errorfor that input; all per-input errors continue with the remaining inputs.Tests
get_mock_context(mocks the SDK + DNS, no network); mapping assertions + low-confidence checks + error-branch tests.inputs.py/outputs.pyentries (live run needsSHODAN_API_KEY).secator test lintclean,secator x shodan <ip>CLI render verified.Known minor (accepted)
_registered_domainis naive last-two-labels (no Public Suffix List) — acceptable for low-confidence passive OSINT; not worth a PSL dependency.Design + plan:
gke-admin/docs/superpowers/specs/2026-06-29-shodan-integration-design.mdand the matching plan.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Documentation