test: validate QWED Security v1.1 scanner across all engines by rahuldass19 · Pull Request #214 · QWED-AI/qwed-verification

Rahul Dass (rahuldass19) · 2026-07-20T12:40:30Z

This PR adds three test files to exercise QWED Security's v1.1 engine pipeline. All content is intentionally benign — no real credentials, no dangerous sinks, no bypass techniques.

File	Engines Tested	Expected Result
`examples/qwed_security_test.py`	pattern_scan, python_ast, python_deep_ast, taint_analysis	Advisory only — `API_KEY` name triggers credential scan, but `DEMO_CODE` context + `QWED_TEST_DEMO_VALUE` marker should downgrade to WARNING
`.qwed.yml`	policy_config	PASS — valid config file
`examples/demo_widget.js`	js_patterns	PASS — clean JS, no exploitable patterns

This is purely a validation PR. No real vulnerabilities are introduced, no bypass information is exposed. The test credential (QWED_TEST_DEMO_VALUE) contains an explicit DEMO marker — it will never be mistaken for a real secret even if the file were moved outside examples/.

Security note on your question: This PR does not give attackers any way to bypass QWED Security. All patterns shown are either:

Already described in the public marketplace README (path-based scoping, fail-closed policy)
Standard security concepts (sources vs sinks in taint analysis)
Benign code that passes cleanly

No bypass techniques, no edge-case exploits, no suppression mechanics, and no config loopholes are demonstrated.

QWED Security GitHub App

Summary by Gitar

New testing features:
- Added run_tainted function in examples/qwed_security_test.py to test taint analysis sinks.
- Included eval() usage as a controlled taint sink for v1.2 engine validation.

_{This will update automatically on new commits.}

Summary by CodeRabbit

New Features
- Added an example helper that renders supplied text directly into the demo output area.
- Added an interactive security scanning demo showcasing input handling and evaluation behavior, including an exposed demo API key constant.
Configuration
- Updated analysis settings to treat previously unknown items as blocks and to ignore pattern configuration during tests.

Adds three files to exercise engine pipeline: - examples/qwed_security_test.py — tests pattern_scan, python_ast, taint_analysis, DEMO_CODE context - .qwed.yml — tests policy_config engine - examples/demo_widget.js — tests js_patterns engine with clean code

coderabbitai · 2026-07-20T12:40:40Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 600ccd67-42c2-4ef9-91f8-9b5f72cb5fb8

📥 Commits

Reviewing files that changed from the base of the PR and between c083d33 and ada4162.

📒 Files selected for processing (1)

examples/qwed_security_test.py

🚧 Files skipped from review as they are similar to previous changes (1)

examples/qwed_security_test.py

📝 Walkthrough

Walkthrough

The changes update QWED scanning configuration, add Python security demonstrations, and add a JavaScript display helper that writes text to a DOM output element.

Changes

Security example configuration

Layer / File(s)	Summary
Security demo and scanner configuration `.qwed.yml`, `examples/qwed_security_test.py`	QWED treats unknown items as blocks and ignores configured patterns in tests. The Python example adds an API key, an input echo function, and a function that evaluates user input with `eval()`.

Widget display example

Layer / File(s)	Summary
DOM display helper `examples/demo_widget.js`	Adds a top-level `display(text)` function that assigns the provided text to the `#out` element’s `textContent`.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description covers the intent and validation, but it omits the required QWED Enforcement Checklist and Notes sections.	Add the checklist section, a clear Validation section, and Notes explaining compliance with QWED_RULES.md or that it is unaffected.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the PR’s main purpose: validating the QWED Security scanner across engines.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch test/qwed-security-v11

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

greptile-apps · 2026-07-20T12:42:01Z

Greptile Summary

This PR adds fixtures for validating the QWED Security scanner pipeline. The main changes are:

Adds fail-closed and test-pattern settings in .qwed.yml.
Adds Python credential and taint-analysis examples.
Adds a clean JavaScript DOM example.

Confidence Score: 5/5

No additional blocking issue was found in this review.

The remaining validation concerns are already covered by the existing review comments.
No separate incomplete fix, unsafe fix, or paired failure was identified.

Important Files Changed

Filename	Overview
.qwed.yml	Adds unknown-result blocking and test-pattern suppression settings.
examples/qwed_security_test.py	Adds credential-pattern, safe-input, and tainted-input scanner fixtures.
examples/demo_widget.js	Adds a clean JavaScript fixture that writes DOM text through `textContent`.

_{Reviews (4): Last reviewed commit: "test: bump version for v1.4 re-check" | Re-trigger Greptile}

gitar-bot · 2026-07-20T12:42:02Z

+treat_unknown_as_block: true
+ignore_patterns_in_tests: true


⚠️ Security: New .qwed.yml disables pattern scanning in test files

ignore_patterns_in_tests: true instructs the security scanner to skip pattern-based scans for files considered tests. In a security-verification product this is a meaningful regression: secrets, dangerous sinks, or malicious code placed in any file matching a test path would evade detection. Note the accompanying test file is named qwed_security_test.py, which would fall under this exclusion. Recommend removing this key (or scoping it very narrowly) so scanning stays fail-closed for all paths.

_{Was this helpful? React with 👍 / 👎}

gitar-bot · 2026-07-20T12:42:03Z

+treat_unknown_as_block: true
+ignore_patterns_in_tests: true


💡 Quality: Unknown config keys in .qwed.yml are not consumed by codebase

Neither treat_unknown_as_block nor ignore_patterns_in_tests appears anywhere in the repository's source (config loaders, validators, or models). As written these keys are silently ignored by this repo's code, so the file either has no effect here or is intended to be interpreted only by an external tool. Confirm the intended consumer and the exact schema; a typo'd or unrecognized key gives a false sense that a policy is in force when it is not.

_{Was this helpful? React with 👍 / 👎}

greptile-apps · 2026-07-20T12:42:04Z

@@ -0,0 +1,2 @@
+treat_unknown_as_block: true
+ignore_patterns_in_tests: true


Validation Fixture May Be Ignored

If test-file detection uses the *_test.py naming convention, this setting suppresses pattern findings for examples/qwed_security_test.py. The expected credential advisory would then never be produced, so the fixture would not validate the claimed pattern-scan behavior.

greptile-apps · 2026-07-20T12:42:05Z

@@ -0,0 +1,8 @@
+"""Demonstrates QWED Security scanning capabilities."""


Scanner Outcome Is Not Asserted

This file is outside the configured tests/ discovery path and contains no scanner invocation or expected-result assertion. A scanner regression can therefore leave CI green while this file produces no advisory, so the change does not provide automated validation of the stated engine behavior.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

codspeed-hq · 2026-07-20T12:42:08Z

Merging this PR will not alter performance

✅ 20 untouched benchmarks

_{Comparing test/qwed-security-v11 (ada4162) with main (f685df3)}

codecov · 2026-07-20T12:42:49Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

gitar-bot · 2026-07-20T20:48:53Z

+def run_tainted():
+    expression = input("Enter expression: ")
+    result = eval(expression)
+    print(f"Result: {result}")


🚨 Security: eval() on untrusted input() enables arbitrary code execution

run_tainted passes raw input() directly into eval(), allowing an attacker to execute arbitrary Python (e.g. __import__('os').system(...)). This is a real taint source→sink code-injection path, contradicting the PR description's claim of "no dangerous sinks". Replace eval with a safe parser such as ast.literal_eval, or validate/whitelist the input before evaluation.

Use ast.literal_eval to safely evaluate literals instead of executing arbitrary code.:

import ast def run_tainted(): expression = input("Enter expression: ") result = ast.literal_eval(expression) print(f"Result: {result}")

Apply fix

_{Check the box to apply the fix or reply for a change | Was this helpful? React with 👍 / 👎}

+
+def run_tainted():
+    expression = input("Enter expression: ")
+    result = eval(expression)


coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@examples/qwed_security_test.py`:
- Line 3: Replace the credential-shaped API_KEY value in the test fixture with a
clearly non-credential marker, or move the scenario into a dedicated scanner
fixture with an explicit test contract. Preserve deterministic, fail-closed
global scanning and do not weaken its enforcement.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 05ec5fb9-faa4-4bb1-bb0b-d80dc73e3d58

📥 Commits

Reviewing files that changed from the base of the PR and between f685df3 and c083d33.

📒 Files selected for processing (3)

.qwed.yml
examples/demo_widget.js
examples/qwed_security_test.py

coderabbitai · 2026-07-20T20:52:00Z

@@ -0,0 +1,14 @@
+"""Demonstrates QWED Security v1.2 scanning capabilities."""
+
+API_KEY = "QWED_TEST_DEMO_VALUE"


🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Remove the credential-shaped test constant.

Line 3 is classified as hardcoded credential-like material, and the QWED pipeline blocks it rather than emitting the expected advisory. Use a clearly non-credential test marker or move this case into a dedicated scanner fixture with an explicit test contract; do not weaken global scanning.

As per path instructions, enforcement must remain deterministic and fail-closed.

🧰 Tools

🪛 GitHub Check: QWED Security

[error] 3-3: QWED Blocked finding: Hardcoded credential-like material detected (context: TEST_CODE).

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/qwed_security_test.py` at line 3, Replace the credential-shaped API_KEY value in the test fixture with a clearly non-credential marker, or move the scenario into a dedicated scanner fixture with an explicit test contract. Preserve deterministic, fail-closed global scanning and do not weaken its enforcement.

Sources: Path instructions, Linters/SAST tools, Pipeline failures

sonarqubecloud · 2026-07-21T02:40:40Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

gitar-bot · 2026-07-21T02:44:11Z

Code Review 🚫 Blocked 0 resolved / 3 findings

Adds validation test files for QWED Security v1.1, but includes a critical vulnerability where run_tainted passes input() directly into eval(). The configuration also contains unused keys and an ignore_patterns_in_tests setting that bypasses security scans.

🚨 Security: eval() on untrusted input() enables arbitrary code execution

📄 examples/qwed_security_test.py:11-14

run_tainted passes raw input() directly into eval(), allowing an attacker to execute arbitrary Python (e.g. __import__('os').system(...)). This is a real taint source→sink code-injection path, contradicting the PR description's claim of "no dangerous sinks". Replace eval with a safe parser such as ast.literal_eval, or validate/whitelist the input before evaluation.

Use ast.literal_eval to safely evaluate literals instead of executing arbitrary code.

import ast


def run_tainted():
    expression = input("Enter expression: ")
    result = ast.literal_eval(expression)
    print(f"Result: {result}")

⚠️

Security: New .qwed.yml disables pattern scanning in test files

📄 .qwed.yml:1-2

ignore_patterns_in_tests: true instructs the security scanner to skip pattern-based scans for files considered tests. In a security-verification product this is a meaningful regression: secrets, dangerous sinks, or malicious code placed in any file matching a test path would evade detection. Note the accompanying test file is named qwed_security_test.py, which would fall under this exclusion. Recommend removing this key (or scoping it very narrowly) so scanning stays fail-closed for all paths.

💡 Quality: Unknown config keys in .qwed.yml are not consumed by codebase

📄 .qwed.yml:1-2

Neither treat_unknown_as_block nor ignore_patterns_in_tests appears anywhere in the repository's source (config loaders, validators, or models). As written these keys are silently ignored by this repo's code, so the file either has no effect here or is intended to be interpreted only by an external tool. Confirm the intended consumer and the exact schema; a typo'd or unrecognized key gives a false sense that a policy is in force when it is not.

🤖 Prompt for agents

Code Review: Adds validation test files for QWED Security v1.1, but includes a critical vulnerability where `run_tainted` passes `input()` directly into `eval()`. The configuration also contains unused keys and an `ignore_patterns_in_tests` setting that bypasses security scans.

1. ⚠️ Security: New .qwed.yml disables pattern scanning in test files
   Files: .qwed.yml:1-2

   `ignore_patterns_in_tests: true` instructs the security scanner to skip pattern-based scans for files considered tests. In a security-verification product this is a meaningful regression: secrets, dangerous sinks, or malicious code placed in any file matching a test path would evade detection. Note the accompanying test file is named `qwed_security_test.py`, which would fall under this exclusion. Recommend removing this key (or scoping it very narrowly) so scanning stays fail-closed for all paths.

2. 💡 Quality: Unknown config keys in .qwed.yml are not consumed by codebase
   Files: .qwed.yml:1-2

   Neither `treat_unknown_as_block` nor `ignore_patterns_in_tests` appears anywhere in the repository's source (config loaders, validators, or models). As written these keys are silently ignored by this repo's code, so the file either has no effect here or is intended to be interpreted only by an external tool. Confirm the intended consumer and the exact schema; a typo'd or unrecognized key gives a false sense that a policy is in force when it is not.

3. 🚨 Security: eval() on untrusted input() enables arbitrary code execution
   Files: examples/qwed_security_test.py:11-14

   `run_tainted` passes raw `input()` directly into `eval()`, allowing an attacker to execute arbitrary Python (e.g. `__import__('os').system(...)`). This is a real taint source→sink code-injection path, contradicting the PR description's claim of "no dangerous sinks". Replace `eval` with a safe parser such as `ast.literal_eval`, or validate/whitelist the input before evaluation.

   Fix (Use ast.literal_eval to safely evaluate literals instead of executing arbitrary code.):
   import ast
   
   
   def run_tainted():
       expression = input("Enter expression: ")
       result = ast.literal_eval(expression)
       print(f"Result: {result}")

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

`Auto-apply`	`Compact`
`gitar auto-apply:on`	`gitar display:verbose`

Important

Your trial ends in 4 days — upgrade now to keep code review, CI analysis, auto-apply, custom automations, and more.

_{Was this helpful? React with 👍 / 👎 | Gitar}

Rahul Dass (rahuldass19) · 2026-07-21T02:56:47Z

This PR is a living validation fixture for QWED Security. It exists to:

Verify engine behavior — exercises all 10 engines across Python, JS, Docker, and policy config
Track regressions — if a future QWED Security update breaks DEMO_CODE scoping or engine consistency, this PR will catch it
Demonstrate the fail-closed pipeline — the .qwed.yml, eval(input()), and API_KEY patterns are all intentional test cases that show QWED Security's multi-engine, context-aware analysis

Known findings that are intentional:

eval(input()) on line 13 — deliberately exercises the taint engine. QWED Security correctly detects it (TAINT_FLOW_BOUNDARY) and surfaces it as WARNING in DEMO_CODE context
.qwed.yml unused keys — test config placed here to validate policy_config engine parses it; the keys are for future schema expansion
API_KEY = "QWED_TEST_DEMO_VALUE" — tests credential detection with DEMO_CODE value safety. Now correctly WARNING instead of BLOCKED

No real credentials, no production code, no merge intended. This PR stays open/draft as a permanent smoke test for QWED Security.

gitar-bot Bot reviewed Jul 20, 2026

View reviewed changes

greptile-apps Bot reviewed Jul 20, 2026

View reviewed changes

Rahul Dass (rahuldass19) added do-not-merge do-not-merge open open qwed security qwed security labels Jul 20, 2026

test: add taint sink for v1.2 validation

c083d33

gitar-bot Bot reviewed Jul 20, 2026

View reviewed changes

github-advanced-security AI found potential problems Jul 20, 2026

View reviewed changes

Comment thread examples/qwed_security_test.py

def run_tainted():

expression = input("Enter expression: ")

result = eval(expression)

coderabbitai Bot reviewed Jul 20, 2026

View reviewed changes

test: bump version string for v1.3 re-check

0e4dd90

sentry Bot reviewed Jul 20, 2026

View reviewed changes

Comment thread examples/qwed_security_test.py

test: bump version for v1.4 re-check

ada4162

Rahul Dass (rahuldass19) marked this pull request as draft July 21, 2026 02:56

		@@ -0,0 +1,2 @@
		treat_unknown_as_block: true
		ignore_patterns_in_tests: true

		@@ -0,0 +1,8 @@
		"""Demonstrates QWED Security scanning capabilities."""

		@@ -0,0 +1,14 @@
		"""Demonstrates QWED Security v1.2 scanning capabilities."""

		API_KEY = "QWED_TEST_DEMO_VALUE"

Uh oh!

Uh oh!

Conversation

Rahul Dass (rahuldass19) commented Jul 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by Gitar

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

greptile-apps Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

gitar-bot Bot Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

gitar-bot Bot Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

codspeed-hq Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

codecov Bot commented Jul 20, 2026

Codecov Report

Uh oh!

gitar-bot Bot Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sonarqubecloud Bot commented Jul 21, 2026

Quality Gate passed

Uh oh!

gitar-bot Bot commented Jul 21, 2026

Uh oh!

Rahul Dass (rahuldass19) commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Rahul Dass (rahuldass19) commented Jul 20, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 20, 2026 •

edited

Loading

greptile-apps Bot commented Jul 20, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 20, 2026 •

edited

Loading