Add bounded DAST external evaluation

README.md

@@ -5,11 +5,11 @@
 [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
 [![Version](https://img.shields.io/badge/version-v0.1.0--alpha-orange)](CHANGELOG.md)
 
-Heimdall is **an open-source DevSecOps validation backend that reduces SAST false positives using LLM-assisted reasoning and safe DAST-style verification**.
+Heimdall is **an open-source post-SAST validation backend that uses deterministic, loopback-only bounded DAST probes to collect runtime evidence**.
 
-It is for AppSec engineers, DevSecOps teams, students, and researchers who want to test whether static findings can be converted into evidence-backed decisions: `True Positive`, `False Positive`, or `Needs Review`.
+It is for AppSec engineers, DevSecOps teams, students, and researchers who want to test whether static findings can be converted into evidence-backed decisions: `Confirmed`, `Not Reproduced Under Test`, or `Needs Review`.
 
-Heimdall is an alpha-stage research/backend tool. It is not a production-ready enterprise platform, and it should only be tested on repositories and targets you own or are authorized to assess.
+Heimdall is an alpha-stage research/backend tool. It is not a production-ready enterprise platform, and it should only be tested on the included loopback fixtures or other explicitly authorized local test targets. No LLM is used by the bounded DAST evaluation path.
 
 ## Current Research Package Summary
 
@@ -65,6 +65,17 @@ Run the final reproducibility workflow:
 bash scripts/run_ieee_final_evaluation.sh
 ```
 
+Run only the bounded DAST protocol on an alert manifest:
+
+```bash
+python -m heimdall.cli bounded-dast \
+  --dataset test_data/heimdall_active_local_alerts.jsonl \
+  --config heimdall.yml \
+  --output reports/bounded_dast
+```
+
+The command writes JSON, CSV, and Markdown audit reports. Each executable alert can cause exactly one request. Alerts without an authorized runtime mapping or sufficient evidence predicate are not sent and return `Needs Review`.
+
 The latest full run passed `53` tests.
 
 ## 5-Minute Quickstart
@@ -96,34 +107,38 @@ SAST tools are valuable because they scale across a codebase, but they often pro
 
 - ingests Semgrep JSON,
 - extracts alert context,
-- uses structured LLM-style reasoning through a mock provider by default,
-- generates non-destructive validation hypotheses,
-- performs dry-run or allowlisted DAST-style validation,
+- consumes an explicitly authorized fixed validation mapping,
+- screens the endpoint and payload before execution,
+- performs one non-destructive request against an exact loopback allowlist,
+- checks declared positive and negative runtime evidence predicates,
 - explains the final decision,
 - produces CI/CD-friendly reports.
 
 ## How It Works
 
 ```mermaid
 flowchart LR
-    A[Semgrep SAST JSON] --> B[Context Extraction]
-    B --> C[Prompt Guard]
-    C --> D[LLM-Assisted Reasoning]
-    D --> E[Safe Payload Generation]
-    E --> F[Dry-Run / Allowlisted DAST]
-    F --> G[Response Analyzer]
-    G --> H[Decision Engine]
-    H --> I[Markdown / JSON / CSV Reports]
+    A[Existing SAST alert] --> B[Authorized fixed probe mapping]
+    B --> C[Safety preflight]
+    C --> D[One loopback-only GET or POST]
+    D --> E[Declared evidence predicates]
+    E --> F[Confirmed / Not Reproduced / Needs Review]
+    F --> G[Markdown / JSON / CSV audit reports]
 ```
 
+Legacy mock-LLM and dry-run baseline modules remain in the repository for historical comparison, but they are not on the bounded DAST evaluation path.
+
 ## Why It Is Safe To Test
 
 - Dry-run validation is enabled by default.
-- Mock LLM mode is enabled by default.
-- Active DAST is limited to configured `allowed_targets`.
-- Production-looking domains are blocked unless explicitly enabled.
-- Destructive payload markers are rejected.
-- All DAST attempts are logged.
+- Bounded DAST must be enabled explicitly and is limited to exact configured loopback origins.
+- Exactly one request is permitted for each executable alert.
+- Only GET and POST are supported.
+- Absolute endpoints, credentials in URLs, endpoint fragments, destructive payload markers, oversized payloads, and external URLs embedded in probes are blocked.
+- Redirects are inspected but never followed.
+- Response capture is bounded and reports store a digest rather than a body excerpt.
+- All sent and blocked attempts are logged.
+- Missing positive or negative evidence routes the alert to `Needs Review`.
 - `Needs Review` does not fail CI by default.
 
 ## CLI Commands
@@ -241,11 +256,11 @@ Docker runs the sample validation command in dry-run mode.
 ## Current Limitations
 
 - Alpha-stage research backend.
-- Default validation is conservative and dry-run based.
+- Bounded DAST requires an alert-to-runtime mapping and an authorized local test target.
 - Business logic flaws often need multi-step authenticated context.
-- Real LLM provider integration is intentionally behind schema validation.
 - The included datasets are small and synthetic.
-- External target validation requires careful allowlist configuration.
+- The bounded DAST protocol intentionally rejects non-loopback targets.
+- A not-reproduced result is not proof that a SAST alert is a false positive and does not authorize automatic suppression.
 
 ## Roadmap
 

SHA256SUMS.txt

@@ -0,0 +1,12 @@
+B2896DEDBA77DBD9A792E1C49910F2294718FB1B551B6D0E5A76846AAF19CFA7  scripts/external_full_evaluation.py
+C6BE88DD4BBC69317B86A2939003A207F4E096481CC7BAD0F291A0DA9929D4C5  scripts/run_external_full_with_benchmark.py
+FE537BBAD41931044FB51207838A2B1BCA83DC8914C5FF5F2A108E9C3900AE5A  scripts/safety_gate_evaluation.py
+B0E2F88BA2DBA4F18863434FF080B0A967D86C9CCAA7F08E71EAE4D0ED6211F6  scripts/build_evaluation_summary.py
+DAE7595F28B0BAFFD5C28470A4DFC993948433DE0DB772C4E4BE2844CB4BB1BA  results/eligible_cases.json
+4A8B83778EF0C2509A047F8181DD2050078337ED1DF1F633D23C9FB0C211A6B0  results/external_full.json
+DD4A1378E76E36633762784A98EBE4FF85C4EBB094BA4FC93323EBDA39E15BB3  results/external_full.csv
+6CE0B57E0A98A8DE3E23468B75CE97E54FC6B3C8FC08B1033289D2C235C2684B  results/safety_gate.json
+A23F9ABE22B6E969B8E354727BC87DCC1BCF4B94D044268C3D5BEFB31B40A840  results/safety_gate.csv
+82948F991CDDB29AC0A743F3E3510B64E42707531F0E87594AA677EC5DF46D6B  results/summary.md
+738124D1084D5539B25E588B7AF9D3AAF31F93E42BF13AB2D8F90744DAAE1412  results/benchmark_server.stdout.log
+E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855  results/benchmark_server.stderr.log

docs/BOUNDED_DAST_PROTOCOL.md

@@ -0,0 +1,59 @@
+# Heimdall Bounded DAST Protocol
+
+Protocol identifier: `heimdall-bounded-dast/1.0`
+
+## Purpose
+
+The protocol collects narrowly scoped runtime evidence for an existing SAST
+alert. It is not a crawler, endpoint-discovery engine, payload mutator, or
+general-purpose DAST scanner.
+
+## Execution contract
+
+An alert is executable only when all of the following are true:
+
+1. Active validation is enabled and the kill switch is off.
+2. The alert carries an explicit local runtime authorization mapping.
+3. The request method is GET or POST.
+4. The endpoint is relative and contains no credentials, fragment, control
+   characters, or path traversal.
+5. The final request origin exactly matches a configured loopback allowlist
+   entry and resolves only to loopback addresses.
+6. The serialized payload is at most 8 KiB and contains no blocked destructive
+   fragment or non-loopback network destination.
+7. A bounded positive evidence marker is declared.
+
+Exactly one request is sent for an executable alert. Redirects are not followed,
+the timeout is bounded, and the captured response body is size-limited.
+
+## Decision contract
+
+- `Confirmed`: the declared positive evidence marker is observed in the
+  category-specific channel, and the status code matches when one was declared.
+- `Not Reproduced Under Test`: a separately declared negative evidence marker
+  or negative status predicate is observed.
+- `Needs Review`: neither predicate is satisfied, the target cannot be reached,
+  context is missing, or any safety precondition fails.
+
+Marker absence by itself is not negative evidence. `Not Reproduced Under Test`
+does not prove that the SAST alert is a false positive and must not trigger
+automatic suppression.
+
+## Audit data
+
+Each result records the protocol version, redacted request URL, method, request
+count, response status, marker outcomes, truncation state, response SHA-256,
+and whether a redirect was followed. Response bodies are not copied into the
+audit report.
+
+## Running
+
+Start the included loopback lab, generate the controlled alert manifest, and
+run:
+
+```bash
+python -m heimdall.cli bounded-dast \
+  --dataset test_data/heimdall_active_local_alerts.jsonl \
+  --config heimdall.yml \
+  --output reports/bounded_dast
+```

docs/algorithm.md

@@ -1,53 +1,54 @@
-# Algorithm 1: Heimdall Closed-Loop Validation
+# Algorithm 1: Heimdall Bounded DAST Validation
 
 ## Purpose
 
-Heimdall converts a static security alert into a safe validation hypothesis, evaluates the hypothesis in a dry-run or allowlisted DAST environment, and returns one of three decisions: True Positive, False Positive, or Needs Review.
+Heimdall evaluates an existing SAST alert only when a fixed, authorized,
+non-destructive runtime probe and explicit evidence predicates are available.
+The evaluated path is deterministic and does not use an LLM.
 
 ## Pseudocode
 
 ```text
-Algorithm 1 Heimdall Closed-Loop Validation
-Input: SAST alert a
-Output: decision d in {True Positive, False Positive, Needs Review}
-
-1:  c <- ExtractContext(a)
-2:  g <- PromptGuard(c)
-3:  if g is rejected then
-4:      return Needs Review with prompt-safety explanation
-5:  end if
-
-6:  h <- GenerateExploitabilityHypothesis(g)
-7:  y <- ValidateStructuredLLMOutput(h)
-8:  if y.confidence is low or y.recommended_action = needs_review then
-9:      return Needs Review with uncertainty explanation
-10: end if
-
-11: p <- GenerateSafePayload(y, c)
-12: s <- ValidatePayloadSafety(p)
-13: if s is blocked then
-14:     return Needs Review with safety-policy explanation
-15: end if
-
-16: r <- ExecuteDAST(p, target allowlist, dry-run default)
-17: e <- AnalyzeResponse(r, p.expected_evidence)
-
-18: if e.status = confirmed then
-19:     d <- True Positive
-20: else if e.status = not_confirmed and y.exploitability = unlikely_exploitable then
-21:     d <- False Positive
-22: else if e.status = not_confirmed then
-23:     d <- False Positive
-24: else
-25:     d <- Needs Review
-26: end if
-
-27: return d with evidence, explanation, safety notes, and recommended action
+Algorithm 1 Heimdall Bounded DAST Validation
+Input: SAST alert a, loopback allowlist L
+Output: d in {Confirmed, Not Reproduced Under Test, Needs Review}
+
+1:  if kill switch is on or bounded DAST is disabled then
+2:      return Needs Review
+3:  end if
+4:  if a has no explicit runtime authorization mapping then
+5:      return Needs Review
+6:  end if
+7:  if a requires authentication or multi-step state then
+8:      return Needs Review
+9:  end if
+10: if a has no bounded positive evidence predicate then
+11:     return Needs Review
+12: end if
+13: p <- BuildFixedProbe(a)
+14: if p.method not in {GET, POST} then
+15:     return Needs Review
+16: end if
+17: if endpoint or payload violates the safety policy then
+18:     return Needs Review
+19: end if
+20: u <- ResolveRequestURL(p)
+21: if Origin(u) is not an exact member of L or does not resolve only to loopback then
+22:     return Needs Review
+23: end if
+24: r <- SendExactlyOneRequest(p, no redirects, bounded timeout and response)
+25: if PositivePredicate(a, r) is satisfied then
+26:     return Confirmed
+27: end if
+28: if DeclaredNegativePredicate(a, r) is satisfied then
+29:     return Not Reproduced Under Test
+30: end if
+31: return Needs Review
 ```
 
-## Notes
+## Decision invariant
 
-- Dynamic validation is dry-run by default.
-- Production targets are blocked unless explicitly configured.
-- Destructive payload patterns are rejected before execution.
-- Business logic and authenticated workflows are conservatively classified as Needs Review when required context is unavailable.
+Absence of the positive marker is not, by itself, negative evidence. A
+not-reproduced verdict requires a separately declared negative marker or status
+predicate. The verdict does not authorize automatic suppression of the SAST
+alert.

docs/devsecops_integration.md

@@ -1,6 +1,8 @@
 # Heimdall DevSecOps Integration
 
-Heimdall V2 can run as a CI/CD security validation backend. In this mode, Semgrep produces static findings, Heimdall converts those findings into alert objects, the validation pipeline classifies each alert, and the CLI returns a policy-based exit code.
+Heimdall V2 includes a deterministic bounded DAST path and a legacy dry-run CI
+adapter. The bounded DAST path requires an explicit alert-to-runtime manifest;
+raw Semgrep output alone does not authorize network execution.
 
 ## CI/CD Workflow
 
@@ -10,27 +12,33 @@ flowchart TD
     B --> C[Semgrep scans code]
     C --> D[Semgrep JSON saved]
     D --> E[Heimdall validates findings]
-    E --> F[Heimdall classifies TP / FP / Needs Review]
+    E --> F[Confirmed / Not Reproduced / Needs Review]
     F --> G[Report uploaded]
     G --> H[Pipeline passes or fails based on policy]
 ```
 
 ## GitHub Actions Flow
 
-The workflow `.github/workflows/heimdall-devsecops.yml` installs Python dependencies, installs Semgrep, writes `semgrep-results.json`, runs `python -m heimdall.cli validate`, uploads reports, and optionally posts `reports/ci_summary.md` to pull requests.
+The existing workflow `.github/workflows/heimdall-devsecops.yml` runs the legacy
+dry-run `validate` command. It is retained as a compatibility example and is not
+the bounded DAST evaluation path.
 
 ## Semgrep-To-Heimdall Data Flow
 
 1. Semgrep writes JSON results.
 2. `heimdall.semgrep_ingest` preserves rule ID, severity, file path, line number, message, snippet, and CWE metadata where available.
 3. Findings are converted into Heimdall `Alert` objects.
-4. The existing prompt guard, mock LLM provider, payload generator, DAST executor, response analyzer, and decision engine run in dry-run mode by default.
-5. CI reports are written to `reports/`.
+4. A separately reviewed manifest adds the relative endpoint, fixed GET/POST
+   parameters, exact loopback target, and evidence predicates.
+5. `python -m heimdall.cli bounded-dast` performs the safety preflight and sends
+   at most one request per executable alert.
+6. JSON, CSV, and Markdown audit reports are written to the selected output
+   directory.
 
 ## Policy Decision Logic
 
 - Exit code `0`: no confirmed High/Critical True Positive was found.
-- Exit code `0`: only False Positive or Needs Review findings exist.
+- Exit code `0`: only Not Reproduced Under Test or Needs Review findings exist.
 - Exit code `1`: confirmed High/Critical True Positive exists and policy says to fail.
 - Exit code `2`: config is invalid or unsafe.
 - Exit code `3`: runtime error.
@@ -39,36 +47,38 @@ Needs Review does not fail the pipeline by default because it means the system l
 
 ## Safety Model
 
-- Dry-run is enabled by default.
-- Mock LLM is enabled by default.
-- DAST refuses targets not present in `security.allowed_targets`.
-- Production-looking domains are rejected unless `security.allow_external_targets` is explicitly enabled.
-- Blocked targets cannot also be allowlisted.
-- Every DAST attempt is logged.
+- Bounded DAST is disabled unless configured explicitly.
+- The bounded path uses no LLM.
+- Only exact loopback origins in `active_validation.allowed_targets` are accepted.
+- Exactly one GET or POST is allowed per executable alert.
+- Redirects are never followed and captured responses are size-limited.
+- Missing evidence, runtime context, or authorization returns Needs Review.
+- Every sent or blocked attempt is logged.
 - The kill switch stops DAST immediately.
 
 ## Local Integration Test
 
 Start the local test app:
 
 ```bash
-cd test_apps/flask_vulnerable_app
-python -m pip install flask
-python app.py
+python scripts/run_bounded_dast_controlled.py
 ```
 
-Run Heimdall with the sample Semgrep output:
+Or start the included local lab and run the bounded DAST command directly:
 
 ```bash
 python -m heimdall.cli check-config --config heimdall.yml
-python -m heimdall.cli validate --semgrep test_data/semgrep-results-sample.json --config heimdall.yml --output reports/
+python -m heimdall.cli bounded-dast \
+  --dataset test_data/heimdall_active_local_alerts.jsonl \
+  --config heimdall.yml \
+  --output reports/bounded_dast
 ```
 
-Read `reports/ci_summary.md` for the CI-friendly result.
+Read `reports/bounded_dast/summary.md` and `results.json`.
 
 ## Limitations
 
-- The default validation path is conservative and dry-run based.
+- SAST alerts require a separately reviewed runtime mapping before execution.
 - Authenticated and multi-step workflows require additional context.
-- Real LLM providers should remain behind the structured output validator.
-- Production deployment requires audited target allowlists, secret management, and review of policy thresholds.
+- The bounded protocol rejects production and other non-loopback targets.
+- Not Reproduced Under Test is not proof that an alert is a false positive.

experiments/run_experiment.py

@@ -112,7 +112,7 @@ def write_summary_markdown(path: Path, summary: dict) -> None:
 
     full_rows = summary["result_groups"].get("heimdall_full_pipeline_stub", {})
     lines.extend(_decision_section("Confirmed Vulnerabilities", full_rows.get("confirmed_vulnerabilities", [])))
-    lines.extend(_decision_section("Discarded False Positives", full_rows.get("discarded_false_positives", [])))
+    lines.extend(_decision_section("Not Reproduced Under Test", full_rows.get("not_reproduced", [])))
     lines.extend(_decision_section("Needs Review Cases", full_rows.get("needs_review_cases", [])))
 
     lines.extend(["## Safety Log Summary", ""])
@@ -233,10 +233,10 @@ def run_experiment(dataset: Path, output: Path, mode: str) -> dict:
 def _group_results(rows: list[dict]) -> dict[str, list[dict]]:
     return {
         "confirmed_vulnerabilities": [
-            row for row in rows if row["final_decision"] == "True Positive" and row["classification"] == "TP"
+            row for row in rows if row["prediction"] == "confirmed"
         ],
-        "discarded_false_positives": [
-            row for row in rows if row["final_decision"] == "False Positive" and row["classification"] == "TN"
+        "not_reproduced": [
+            row for row in rows if row["prediction"] == "dismissed"
         ],
         "needs_review_cases": [row for row in rows if row["final_decision"] == "Needs Review"],
     }