Repository navigation
[Feature]: Gate complexity and test strength with PSComplexity and PSMutant (PR on changed files, weekly on the whole tree) #91
Description
Problem statement
The build already fails on any PSScriptAnalyzer Error/Warning, imports the built module under both powershell.exe and pwsh.exe with Set-StrictMode -Version Latest, enforces comment-based help and runs ~1,165 Pester test cases. None of that measures:
- whether a test would catch a bug. A passing suite proves the code ran. It does not tell a test that asserts real behaviour from one that keeps passing when the condition under it is inverted.
- whether a function has become too tangled to reason about. There is no complexity measurement at all.
Two Fortigi modules answer exactly those questions:
| Module | Version (pinned) | Answers |
|---|---|---|
| PSComplexity | 0.5.1, latest on the PowerShell Gallery | Cyclomatic and SonarSource cognitive complexity per unit, from the AST, with a CI gate, a ratcheting baseline and -ChangedFile scoping. |
| PSMutant | 0.5.0, latest on the PowerShell Gallery | Mutation testing: injects small faults (-eq→-ne, $true→$false, a dropped -not, N→N+1) in a sandbox and checks that Pester fails. -ChangedFile scopes a run to a PR. |
| Pester | 6.2.0 | Test runner. Still ships a net462 build and declares PowerShell 5.1 as its minimum, so the Windows PowerShell leg keeps working. |
Versions checked 2026-10-10. PSComplexity's main already carries a 0.5.2 (SARIF fields for Azure DevOps, no score change), but it is not tagged or published yet, so 0.5.1 stays the pin.
Both modules require PowerShell 7.0+. Every new task skips itself under Windows PowerShell 5.1, so the powershell leg of PR validation is unaffected.
Proposed solution
The bar is enforced, not report-only (this replaces the earlier report-only plan). It is applied at two scopes:
| Scope | When | On failure | |
|---|---|---|---|
| PR gate | Only the files the PR changed | /validate, pwsh leg, after the tests |
Validation fails |
| Weekly gate | The whole tree | Monday 02:23 UTC, plus manual dispatch | A bug labelled quality-gate is filed per gate; later failing runs comment on it; it closes itself when the gate passes again |
The bar.
- Complexity: every unit ≤ 15 cyclomatic and ≤ 15 cognitive. The 32 units already over that are recorded in a committed
complexity-baseline.json. A recorded unit may not get worse, any other unit must meet the ceilings, and an entry that stops describing the code (improved, fixed, renamed) fails until it is lowered. The baseline ratchets down only. - Mutation: score ≥
thresholds.breakinpsmutant.config.json, set at 75% from the first measurement (see the analysis comment). Only ever raised.
Phase 0: pin the toolchain, one module list
-
Build/InstallModules.ps1is the only module list;Build/build.ps1calls it instead of keeping its own. - Pester 6.2.0, PSComplexity 0.5.1 and PSMutant 0.5.0 pinned with
-RequiredVersion.build.ps1imports the pinned Pester explicitly, because 5.1 ships Pester 3.4.0 inbox. - Pester 6.2.0 checked against the suite: run against the source module on Linux, Pester 5.7.1 and 6.2.0 produce the identical set of passed and failed tests (940 passed; the 176 failures are all Windows-only APIs).
- psake, PSDeploy, PSScriptAnalyzer and ThreadJob are still unpinned or floor-only. Pinning them needs the current gallery versions confirmed; left as a follow-up.
Phase 1: complexity gate
-
Task Complexity: whole tree, againstcomplexity-baseline.json, JSON report and SARIF underbuildoutput/quality/. -
Task ComplexityChanged: changed files only. - Workaround for an upstream defect: PSComplexity 0.5.1 (and current
main) throws when-ChangedFileand-BaselineFileare combined. Every baseline entry outside the changed files reads as "renamed or moved". The task narrows the baseline to the changed files first, so the ratchet still holds for the code under review. This should be reported upstream in Fortigi/PSComplexity.
Phase 2: mutation gate
The original blocker, mutants being invisible because the tests load the built artifact, does not apply to PSMutant's runs. PSMutant invokes the mapped test files without -Data, so each file falls back to its default -ModulePath, which is the source OmadaWeb.PS.psm1, and that file already dot-sources Public/ and Private/. A mutant spliced into a source file inside the sandbox is exactly what the tests load.
-
psmutant.config.jsonat the repo root. Themutate→testsmap is generated byBuild/Update-MutationConfig.ps1: a source file is covered by every test file underTests/UnitandTests/Integrationthat names one of its functions.Tests/E2Eis never mapped. Files no test names go under_untestedand are not mutated, so they cannot inflate or deflate the score.-Checkruns on every PR, so the map cannot go stale. -
workers: 1, deliberately. Three suites share process-wide state that parallel runspaces would race on, and those races score as kills:StrictMode.Tests.ps1writes$Env:OMADAWEBPS_STRICTMODE,Install-EdgeDriver.Tests.ps1writes$Env:TEMP/$Env:TMP, and the integration tests bind fixed port 9080. -
Task Mutatefor the whole tree andTask MutateChangedfor changed files. Both run underOMADAWEBPS_STRICTMODE=1, like the Test task.
Phase 3: pipelines
-
pr-validation.yml: new step./Build/build.ps1 -Task QualityChangedon the pwsh leg (MutationConfigCheck,ComplexityChanged,MutateChanged). It reads the existingPR_CHANGED_FILES, which was previously written and never read. Each gate skips itself, saying so, when nothing it measures changed. Reports are uploaded as theQualityReportsartifact. -
quality-weekly.yml: aComplexityjob and aMutatejob in parallel on windows-latest. Each writes a verdict file. Analertjob (the only one withissues: write) opens, comments on or closes one bug per gate. A gate that never reached a verdict (install failure, red baseline suite) turns the run red but files no bug, because a run that measured nothing says nothing about the code. -
CONTRIBUTING.mddocuments the gates, the generated map and the baseline workflow.
Open points and risks
- Not yet run on Windows. Everything above was exercised on Linux pwsh 7.6 with a patched copy of the module, because the module refuses to load off Windows. 60 of the 96 mapped files could be evaluated there. The other 36 depend on Windows-only tests (Authenticode, CIM, WinForms,
pwsh.exe, HttpListener). Before relying on the PR gate, run it once on a throwaway PR that touches a module file, and run the weekly workflow by hand after merge. - Source-mode baseline. PSMutant refuses to run unless the mapped tests are green against the source module. The earlier version of this issue noted two snippet tests that fail in source mode. If they still do on Windows,
MutateChangedblocks any PR touching the files they cover, and the weeklyMutatereaches no verdict. Fix or re-map them. - File-level, not line-level. Both tools judge a touched file as a whole. A one-line change to a weak file (for example
Clear-OmadaWebCache.ps1at 45.5%) has to bring that file to the floor. That is intentional (boy-scout rule), and it is the cost of this design. - Floor calibration. 75% comes from the Linux-evaluable subset (78.8%). Recalibrate
thresholds.breakfrom the first full Windows run. Only raise it. - Runtime. On Linux the 546 mutants took 6 minutes serially. The full Windows set includes integration suites and will be slower. The weekly job has the 360-minute ceiling. A PR touching
Invoke-OmadaRequest.ps1runs 18 covering suites per mutant and will be the slowest PR case.
Out of scope / follow-ups
- Isolating the three suites above so
workerscan go above 1. - Uploading
complexity.sarifto GitHub code scanning. - Adding the opt-in operators (
ConditionForcing,ReturnValue,ConditionalBoundary). They roughly double the mutant count and would need the floor reset. - The 11 source files no test names (listed under
_untestedinpsmutant.config.json). - Mirroring this in [Feature]: Gate complexity and test strength with PSComplexity 0.5.1 and PSMutant 0.5.0 on Pester 6.2.0 — PR gate on changed code, weekly whole-tree run that files bugs OmadaSqlTroubleshooter#109.
Implementation: branch ccr-5a55522e-x59sy1.
Activity
mvaneijken commentedon Sep 11, 2026
Sibling proposal: Fortigi/OmadaSqlTroubleshooter#109 — same three phases, same pinned versions. Phases 0 and 1 are near-identical and should be ported in whichever direction lands first, as the workflows were in dd90afa. Phase 2 is where the two repos genuinely diverge: that repo's tests dot-source individual source files, which is exactly the per-file scoping PSMutant wants; here the build-artifact test contract has to be resolved first.
[-][Feature]: Measure complexity and test strength in the build with PSComplexity and PSMutant (report-only)[/-][+][Feature]: Gate complexity and test strength with PSComplexity and PSMutant (PR on changed files, weekly on the whole tree)[/+]mvaneijken commentedon Oct 10, 2026
First analysis —
main@92ee2f6, PSComplexity 0.5.1, PSMutant 0.5.0, Pester 6.2.0Measured on Linux, pwsh 7.6.6. PSComplexity reads the AST only, so its numbers are platform-independent. PSMutant has to run the tests, and the module only loads on Windows, so mutation was measured on a patched copy with the Windows environment faked, and only over the files whose covering tests are green there. Treat the mutation score as a partial, indicative reading, not the project's score. The first weekly run on Windows is the real baseline.
Complexity (whole module, 218 units)
complexity-baseline.json)Worst ten (cyclomatic / cognitive):
Invoke-OmadaRequestGet-DataFromWebDriverInitialize-WebView2Invoke-WebView2MicrosoftLoginOmadaWeb.PS.psm1<script-body>Start-WebView2LoginGet-WebView2CookieResolve-EntraSignInScreenConvertTo-RedactedLogValueInstall-WebView2The debt is concentrated in the request pipeline and the browser sign-in flows: 8 units above cognitive 60, against a median of ≈2.
Invoke-OmadaRequestalone is about 10× the ceiling and is the obvious refactoring target. It is also the function the most test files cover (18), so a refactor there is well protected.Test strength (mutation)
Mapping: 96 source files are named by at least one test and are mapped. 11 are named by none, so they are not mutated:
Close-EdgeDriver,Confirm-FileHash,ConvertTo-OmadaExpiryMoment,Get-EdgeProfile,Get-OmadaLegacyCookieCacheFile,Get-OmadaResponseStatusCode,Install-LockedPackageGroup,Reset-Timer,Start-EdgeDriver,Start-EdgeDriverLogin,Test-OmadaTransientFailure. Some of these, for exampleTest-OmadaTransientFailureandConfirm-FileHash, are security- or retry-relevant and are only exercised indirectly.Evaluated subset: 60 of the 96 files. The other 36 are covered by tests that need Windows (Authenticode, CIM, WinForms,
pwsh.exe, HttpListener), and PSMutant refuses to run on a red baseline.coveredLinesOnly)filesWithNoCandidate(vacuous 100%)filesWithNoMutants(coverage removed every candidate)Initialize-Webview2(92 candidates, 0 covered),Get-WebView2Cookie(29, 0 covered) andGet-NuGetLibraryFolderWeakest files (score, survived/total):
Invoke-WindowsAuthentication.ps1,Unprotect-OmadaSessionPayload.ps1Install-Selenium.ps1Get-InstalledModuleInfo.ps1,Reset-LoginAutomationState.ps1Clear-OmadaWebCache.ps1Invoke-WebEdgeDriverFramework.ps1Stop-OmadaLogin.ps1New-OmadaWebCacheItem.ps1Resolve-OmadaNativeCommand.ps1Select-EntraMfaMethod.ps1ConvertTo-RedactedLogString.ps1Patterns in the survivors:
Unprotect-OmadaSessionPayload.ps1:53-ne→-eqsurvives. That condition guardsZeroFreeBSTRon the decrypted plaintext infinally. Inverted, the plaintext stays in unmanaged memory, and no test notices. This is secret-handling hygiene, not functional behaviour, so it may need a targeted test (assert the zero-and-free is called) rather than a behavioural one.Clear-OmadaWebCache.ps1:112/139.if ($Force -and -not $WhatIfPreference)can be inverted unnoticed, which means no test asserts that-Forcesuppresses confirmation without disabling-WhatIf. The count condition at line 139 survives too.Stop-OmadaLogin.ps1:136–183. Every-not [string]::IsNullOrWhiteSpace(...)around the error-detail fields survives. The tests do not assert which fields appear in the message.Select-EntraMfaMethod.ps1:61–79. All priority constants (10…70, 999) surviveN → N+1. These are likely equivalent mutants (only the order matters), so they are candidates forequivalentswith a reason rather than for more tests.ConvertTo-RedactedLogString.ps1, 11 survivors. That is significant for a redaction function, where a missed branch means a secret in a log.Pester 6.2.0
Against the source module on Linux, Pester 5.7.1 and 6.2.0 produce identical results: 1,165 tests, 940 passed, 176 failed (all Windows-only APIs), 29 skipped, and a byte-identical list of failing tests. The test files use no Pester-5-only API (
Assert-MockCalledand similar). On Windows the/validaterun on the implementation branch is the confirmation.Findings outside the two metrics
workers: 1.-ChangedFilecombined with-BaselineFilethrows on every baseline entry outside the changed files, contradicting its README. Reproduced on 0.5.1 and currentmain; worked around inComplexityChanged.Conclusion
Complexity is fine for 85% of units, with a hard core of about 8 very large functions in the request and sign-in paths. On the parts that could be measured, the tests catch about 4 in 5 injected faults. The gaps sit in places that matter: plaintext cleanup after session decryption, the
-Force/-WhatIfinteraction in cache clearing, and log redaction. The floor is set at 75% from this reading; the first Windows weekly run should set it properly.Generated by Claude Code
Write the whole verdict to the step summary, and fix a hard-to-parse …
Assert the requested backoff delays, not the wall clock between retries
Gate complexity and test strength with PSComplexity and PSMutant (#119)