LLM-Cognitive-Bias-Probe

Test your AI agent for cognitive biases, skip-step behavior, and hidden weight distributions.

A black-box probe framework for LLM evaluation and agent behavior testing. Feed it contrastive prompt pairs, analyze output differences, and discover:

• Does your agent skip test/verify steps under time pressure?
• Is it easily anchored by step-count suggestions?
• Does authority bias reduce its verification rigor?
• What latent weights drive its decisions across 15 cognitive dimensions?

Built for AI engineers evaluating LLMs, safety researchers testing agent behavior, and developers who want to know how their coding agent actually thinks.

Features

Agent Skip-Step Detection [NEW]

Finds when agents drop critical steps (test/verify). Key finding: describing a task as "3 simple steps" causes ~50% of agents to skip verification entirely.

15-Dimension Probe System

Category	Dimensions
Semantic	Sentiment, Factuality, Logic Chain, Authority
Structural	Position Bias, Repetition, Anchoring, Order Effect
Domain	Risk Sensitivity, Gain Preference, Temporal Discount, Loss Aversion
Adversarial	Instruction Following, Consistency Check, Safety Filter

Agent Skip-Step Test Suite (5 Dimensions, 21 Probes)

• risk_sensitivity — does urgency skip safety checks?
• authority_bias — does authority reduce verification?
• anchoring_effect — does step-count anchoring skip test/verify?
• loss_aversion — does loss fear cause over-caution or skips?
• temporal_discount — does time pressure compress the process?

Pre-formatted for sub-agent spawning (OpenClaw, LangGraph, CrewAI, etc.)

Output & Visualization

• 6 distance metrics: semantic cosine, word frequency, sentiment, confidence, length, keyword hit
• HTML dashboards, heatmaps, radar charts, weight trajectories — zero external deps
• JSON output for automated CI/CD pipelines

Quick Start

"@""bash

Check what probes are available

python3 scripts/agent_step_probe.py

Generate sub-agent tasks to test YOUR agent (21 probes)

python3 scripts/agent_step_probe.py --tasks

Standard probe generation (finance scenario)

python3 scripts/probe_generator.py

Full experiment with HTML reports

python3 run_experiment.py "@""

Requirements

• Python 3.10+
• No mandatory external dependencies (visualizations use pure HTML/CSS/JS)
• Optional: numpy, scikit-learn for semantic distance

Project Structure

"@"" ├── scripts/ │ ├── agent_step_probe.py # Agent skip-step test suite [NEW] │ ├── probe_generator.py # 15-dimension probe generation │ ├── weight_mapper.py # 6 distance metrics │ ├── accumulator.py # Context accumulation analysis │ ├── visualizer.py # HTML visualization │ └── run_experiment.py # Experiment runner ├── adapters/ │ └── openclaw_adapter.py # Sub-agent probe executor ├── results/ # Experiment HTML reports + JSON ├── agent_step_test_cases.md # Full test cases (CN/EN) ├── SKILL.md # Complete methodology documentation └── COMMANDS.md # CLI quick reference "@""

Research Findings (from deepseek-chat testing)

"@""

1. "3 simple steps" anchoring causes agents to merge test/verify (skip-risk: 🔴 HIGH)
2. URGENT + high-risk framing causes safety check drops (skip-risk: 🟡 MEDIUM)
3. Authority citation does NOT directly skip steps but agents NEVER validate source credibility
4. Loss framing BACKFIRES — agents become MORE cautious, not less
5. When given "strategic planning" framing, agent steps expand dramatically (good prompt sensitivity) "@""

Use Cases

• LLM Safety Testing — probe for instruction adherence, safety filter bypass
• Agent QA — verify your agent doesn't cut corners under pressure
• Model Selection — compare cognitive profiles of different LLMs
• Prompt Engineering — measure how framing changes model behavior

License

MIT

Author

392394@coze.email