qualitymax-grader

Lighthouse for test quality. Grade your Playwright tests A-F with a single command.

Live Example

Actual grading output from the included example files:

File	Grade	Score	Key Issues
good-test.spec.ts	A	100/100	None
mediocre-test.spec.ts	B	88/100	waitForTimeout, no test.step()
bad-test.spec.ts	D	53/100	No imports, no assertions, waitForTimeout, fragile selectors
empty-test.spec.ts	F	35/100	No imports, no assertions, too short, no navigation

$ qualitymax-grader examples/*.spec.ts --verbose

  qualitymax-grader v0.2.0

  examples/bad-test.spec.ts        D  (53/100)  ✗  Missing @playwright/test import
    ✗ Has @playwright/test import 0/15  Missing @playwright/test import
       ^ Add: import { test, expect } from '@playwright/test';
    ✓ Has test structure 15/15
    ✗ Has assertions 0/20  No assertions found (expect() calls)
       ^ Add: await expect(page.locator('...')).toBeVisible();
    ✓ Balanced braces 10/10
    ✓ Reasonable length 10/10
    ✓ No TypeScript syntax in .js 5/5
    ✓ Uses page.goto() 5/5
    ✗ Has test.step() blocks 0/5  No test.step() blocks -- add steps for readability
       ^ Wrap actions in: await test.step('Step name', async () => { ... });
    ✓ No markdown fences 5/5
    ✓ Meaningful actions 10/10
    ✗ No waitForTimeout -4  Uses waitForTimeout (2x) -- flaky anti-pattern
    ✗ Selector stability -3  Fragile selectors detected (XPath, nth-child)

  examples/empty-test.spec.ts      F  (35/100)  ✗  Missing @playwright/test import
    ✗ Has @playwright/test import 0/15
    ✓ Has test structure 15/15
    ✗ Has assertions 0/20
    ✗ Reasonable length 0/10  Very short (5 lines)
    ✗ Uses page.goto() 0/5
    ✗ Has test.step() blocks 0/5
    ✗ Meaningful actions 0/10

  examples/good-test.spec.ts       A  (100/100)  ✓
    ✓ Has @playwright/test import 15/15
    ✓ Has test structure 15/15
    ✓ Has assertions 20/20
    ✓ Balanced braces 10/10
    ✓ Reasonable length 10/10
    ✓ No TypeScript syntax in .js 5/5
    ✓ Uses page.goto() 5/5
    ✓ Has test.step() blocks 5/5
    ✓ No markdown fences 5/5
    ✓ Meaningful actions 10/10
    ✓ Selector stability 5/5

  examples/mediocre-test.spec.ts   B  (88/100)  ✓
    ✓ Has @playwright/test import 15/15
    ✓ Has test structure 15/15
    ✗ Has assertions 10/20  Only 1 assertion (minimum 2 recommended)
    ✓ Balanced braces 10/10
    ✓ Reasonable length 10/10
    ✓ No TypeScript syntax in .js 5/5
    ✓ Uses page.goto() 5/5
    ✗ Has test.step() blocks 0/5  No test.step() blocks
    ✓ No markdown fences 5/5
    ✓ Meaningful actions 10/10
    ✗ No waitForTimeout -2  Uses waitForTimeout (1x)
    ✓ Selector stability 5/5

  Summary: 4 files graded | Average: C (69/100) | Passed: 2 | Failed: 2

Quick Start

npx qualitymax-grader tests/**/*.spec.ts

Example Output

  qualitymax-grader v0.2.0

  tests/login.spec.ts          A  (92/100)  ✓
  tests/checkout.spec.ts       D  (38/100)  ✗ no assertions, only navigation
  tests/search.spec.ts         B  (78/100)  ✓
  tests/dashboard.spec.ts      F  (12/100)  ✗ empty boilerplate

  Summary: 4 files graded | Average: C (55/100) | Passed: 2 | Failed: 2

  Fix low grades automatically → https://qualitymax.io

What It Checks

Check	Points	Description
@playwright/test import	15	Has proper import { test, expect }
Test structure	15	Uses test() or test.describe()
Assertions	20	Has expect() calls (0 for none, 10 for 1, 20 for 2+)
Balanced braces	10	{ and } counts match
Reasonable length	10	Between 10-300 lines
No TypeScript in .js	5	No type annotations in plain JS files
Navigation	5	Uses page.goto() to navigate
Step blocks	5	Uses test.step() for readability
No markdown fences	5	Raw code, no ``` wrappers
Meaningful actions	10	Sufficient page.* and expect() calls
No waitForTimeout	-2/ea	Anti-pattern penalty (max -5)
Selector stability	+5/-3	Bonus for data-testid, penalty for XPath/nth-child
No garbage selectors	-3/ea	Bare getByRole() without name qualifier

Grade Scale

Grade	Score Range
A	90 - 100
B	75 - 89
C	60 - 74
D	40 - 59
F	0 - 39

CLI Options

--min-grade <grade>   Minimum passing grade (A/B/C/D/F). Default: B
--format <fmt>        Output format: text (default), json, junit, sarif
--json                Alias for --format json
--verbose, -v         Show per-check breakdown for each file
--fix                 Show fix suggestions for each failing file
--help, -h            Show help

Output Formats

# Default pretty text
qualitymax-grader tests/**/*.spec.ts

# JSON (for CI scripting)
qualitymax-grader tests/**/*.spec.ts --json

# JUnit XML (for CI test report integrations)
qualitymax-grader tests/**/*.spec.ts --format junit > grade-report.xml

# SARIF v2.1.0 (for GitHub Code Scanning / IDE integration)
qualitymax-grader tests/**/*.spec.ts --format sarif > grade-report.sarif

CI Integration

GitHub Action (Reusable)

- uses: Quality-Max/qualitymax-grader@v0
  with:
    test-dir: 'tests/**/*.spec.ts'
    min-grade: 'B'

The action installs the grader, runs it against your test files, and fails the step if any test falls below the minimum grade.

GitHub Actions (Manual Setup)

name: Test Quality Gate

on: [push, pull_request]

jobs:
  grade:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Grade Playwright tests
        run: npx qualitymax-grader tests/**/*.spec.ts --min-grade B

      - name: Upload grade report (JUnit)
        if: always()
        run: npx qualitymax-grader tests/**/*.spec.ts --format junit > test-grades.xml

      - name: Upload SARIF to GitHub Code Scanning
        if: always()
        run: |
          npx qualitymax-grader tests/**/*.spec.ts --format sarif > test-grades.sarif

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-grades
          path: |
            test-grades.xml
            test-grades.sarif

The CLI exits with code 1 if any test falls below --min-grade, failing your CI pipeline.

Library Usage

const { gradeTest, scoreToGrade, analyzeSelectors } = require('qualitymax-grader');

const code = fs.readFileSync('test.spec.ts', 'utf-8');
const result = gradeTest(code, 'test.spec.ts');
// result: { score, grade, checks, issues }

console.log(result.grade); // 'A'
console.log(result.score); // 95
console.log(result.checks); // [{ id, name, maxPoints, earned, passed, issue, suggestion }]

You can also pass options to disable specific checks:

const result = gradeTest(code, 'test.spec.ts', {
  checks: {
    steps: { enabled: false },
    navigation: { enabled: false },
  },
});

Playwright Reporter

Add the reporter to your playwright.config.ts:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['qualitymax-grader/reporter', { minGrade: 'B' }],
  ],
});

Options:

• minGrade - Minimum grade to pass (default: 'B')
• verbose - Show per-check breakdown (default: false)
• json - Output JSON instead of pretty text (default: false)

Configuration

Create a .qualitymaxrc.json file in your project root (or any parent directory):

{
  "minGrade": "B",
  "checks": {
    "steps": { "enabled": false },
    "navigation": { "enabled": false }
  },
  "ignore": ["**/generated/**"]
}

• minGrade - Default minimum grade (CLI --min-grade overrides this)
• checks - Disable specific checks by setting enabled: false
• ignore - Glob patterns for files to skip

The grader walks up from the current directory looking for .qualitymaxrc.json, similar to how ESLint finds its config.

Selector Stability Scoring

Selectors are scored by reliability:

Selector Pattern	Score	Stability
getByRole with name	+3	Best
data-testid, data-test	+2	Good
getByText, getByLabel	+1	Okay
CSS class selector	0	Neutral
nth-child, :first-child	-2	Fragile
XPath	-3	Worst

---

Want to auto-fix low grades? Let AI rewrite your tests with stable selectors, proper assertions, and step blocks.

qualitymax.io

License

Apache 2.0 - see LICENSE