fix(adversarial): LLM classifier skips the shared-memory and broadcast channels#29
Conversation
…M classifier AdversarialLLMLayer.aevaluate collected only direct_prompt, memory_reads, tool_responses, retrieved_documents, and peer_messages for LLM classification. The regex scanner in AdversarialLayer also scans the two v0.8 multi-agent channels (shared_memory, broadcast_messages), but the LLM never saw them. Since the regex taxonomy scores ~0 on action-oriented injection, those channels had no effective coverage for the exact cascade-compromise (T-CASCADE) case they were added to defend. Collect both channels into the classifier target list, matching the content keys and channel labels the regex scanner already uses.
requie
left a comment
There was a problem hiding this comment.
Confirmed the gap and the fix. The regex scanner (_detect_channel_threats) covers seven channels; aevaluate only fed five to the LLM, so shared_memory and broadcast_messages never reached the classifier that’s supposed to catch what the regex can’t. The added code mirrors the scanner exactly: same topology_context source, same content-key fallbacks, and same channel labels, including the broadcast_messages key mapping to the broadcast_channels label. That last part matters because the dedup map is keyed on channel label; a mismatch would have produced duplicate ThreatAssessments. You got it right. After this change the LLM covers all seven channels; I found no remaining gap.
One change requested before merge: commit the stub-based test you describe in the PR body. tests/test_adversarial_llm.py currently has zero coverage of channel targeting. A test that stubs _call_claude_classify, feeds a context with all seven channels populated, and asserts each one reaches the classifier would pin this permanently.
One note, not a blocker and not caused by this PR: targets are classified one sequential awaited LLM call each, and the multi-agent buffers cap at 1000 entries, so worst-case aevaluate latency and cost grow linearly with attacker-writable content. The peer_messages loop already had this shape. Tracking separately.
The problem
agentegrity has two ways to catch a malicious instruction hidden in text: a regex matcher (fast, but ~0% on real action-injection, per the module's own docstring) and an LLM classifier (the one that actually works).
The regex scanner checks seven channels, including the two multi-agent ones added in v0.8:
shared_memoryandbroadcast_messages. ButAdversarialLLMLayer.aevaluateonly sent five channels to the LLM. Those two never reached it.Impact: a compromised peer drops an action-injection into shared memory; the reading agent's regex misses it (~0%) and the LLM never sees it, so it passes clean. That's the exact cross-agent cascade (T-CASCADE) the v0.8 channels were meant to defend, with a hole right where the strong detector should be.
The fix
Collect
shared_memoryandbroadcast_messagesinto the classifier's target list, using the same content keys and channel labels the regex scanner already uses.Verified
With the LLM call stubbed to record inputs, both channel payloads now reach the classifier (previously only the direct prompt did). Adversarial suite: 41 passed, 10 skipped (async tests need
pytest-asyncio, unrelated).