[Bug]: E2E test fails with plugin load error "Cannot read properties of undefined (reading 'find')"

[sjnims]

Bug Description

When running E2E tests, the "runs full evaluation pipeline for all component types" test fails during Stage 3 plugin verification with the error:

Plugin failed to load: Plugin load failed: Cannot read properties of undefined (reading 'find')

Important context: All unit tests pass. This issue only manifests during E2E test execution with real API calls and the full plugin loading flow.

Steps to Reproduce

1. Run RUN_E2E_TESTS=true npm test -- tests/e2e/
2. Observe the "runs full evaluation pipeline for all component types" test
3. See the plugin load error during Stage 3

Expected Behavior

The plugin at tests/fixtures/valid-plugin should load successfully during E2E test execution, just as it does in unit tests.

Actual Behavior

Plugin verification fails with:

[INFO] Plugin failed to load: /Users/.../tests/fixtures/valid-plugin
  Error: Plugin load failed: Cannot read properties of undefined (reading 'find')
  Type: unknown
  Hint: Check logs for detailed error information

The test fails even after retry, causing the E2E test suite to report 1 failure.

Environment

• OS: macOS
• Node.js: v25.x
• Test runner: Vitest

Error Logs / CLI Output

[ERROR] Plugin failed to load: Plugin load failed: Cannot read properties of undefined (reading 'find')

============================================================
STAGE: STAGE 3: EXECUTION (5 items)
============================================================
[INFO] Verifying plugin load...
[INFO] Plugin failed to load: /Users/stevenims/Projects/cc-plugin-eval/tests/fixtures/valid-plugin
  Error: Plugin load failed: Cannot read properties of undefined (reading 'find')
  Type: unknown
  Hint: Check logs for detailed error information

Additional Context

• Unit tests for Stage 3 execution pass without issues
• The plugin structure appears valid (Stage 1 analysis succeeds)
• This suggests either:
  1. An E2E test setup/configuration issue
  2. A real integration bug that only manifests with the full SDK flow
  3. A race condition or environment-specific issue in E2E conditions

Investigation should determine whether this is a test infrastructure problem or an actual code bug in the execution stage.

---

🤖 Created with Claude Code

[claude]

Problem Assessment

Is the problem clearly defined?

Yes. The issue is well-defined:

• Error: Cannot read properties of undefined (reading 'find') during Stage 3 plugin verification
• Location: src/stages/3-execution/plugin-loader.ts:343
• Scope: Only occurs in E2E tests with real SDK calls; all unit tests pass
• Test: "runs full evaluation pipeline for all component types" in tests/e2e/pipeline.test.ts

What appears to be the root cause?

The error occurs at src/stages/3-execution/plugin-loader.ts:343:

const loadedPlugins = initMsg.plugins;
const targetPlugin = loadedPlugins.find(
  (p) => p.path === pluginPath || p.path.endsWith(pluginPath),
);

The code assumes initMsg.plugins is always defined, but in real SDK scenarios, the plugins field may be undefined when:

1. The SDK encounters an error during plugin loading
2. The system init message arrives in an unexpected state
3. There's a version mismatch between the SDK and our expectations

Unit tests pass because mocks explicitly set the plugins field (see tests/integration/stages/3-execution/plugin-loader.test.ts:62), but the real SDK may not provide this field in error conditions.

Which stage(s) of the pipeline does this affect?

Stage 3: Execution - specifically the plugin load verification step that happens before executing scenarios.

Proposed Solution

Defensive null check

Add a defensive check for undefined plugins array:

Location: src/stages/3-execution/plugin-loader.ts:336-354

function processInitMessage(
  initMsg: SDKSystemMessage,
  pluginPath: string,
  timings: PluginLoadTimings,
): PluginLoadResult {
  // Check if our plugin is in the loaded plugins array
  const loadedPlugins = initMsg.plugins;

  // Guard against undefined plugins array
  if (\!loadedPlugins || \!Array.isArray(loadedPlugins)) {
    return createFailedResult(
      pluginPath,
      `Plugin load failed: SDK init message missing plugins array`,
      "unknown",
      timings,
    );
  }

  const targetPlugin = loadedPlugins.find(
    (p) => p.path === pluginPath || p.path.endsWith(pluginPath),
  );
  // ... rest of function
}

Tradeoffs

Pros:

• Minimal code change (2-3 lines)
• Prevents crash and provides clearer error message
• Maintains backward compatibility
• No impact on performance

Cons:

• Doesn't address the why - just handles the symptom
• May mask underlying SDK issues

Risks and Edge Cases

1. SDK version compatibility: This may indicate a breaking change in the Claude Agent SDK. We should verify the SDK version and check release notes.

2. Error propagation: If the SDK is trying to communicate a plugin load error but we're not handling it properly, we should also check for error messages earlier in the stream.

3. MCP-related failures: The test has mcp_servers: false, but the code at line 358 still processes initMsg.mcp_servers. If both plugins and mcp_servers are undefined, there may be a broader SDK state issue.

Alternative Approaches

1. Enhanced error context from SDK messages

Instead of just checking for undefined, collect all error messages from the SDK stream before the init message and include them in the failure result.

Pros:

• Captures full error context from SDK
• Helps diagnose why plugins is undefined
• Better debugging experience

Cons:

• More complex change
• Requires updating function signatures

2. Type-safe SDK message handling

Add TypeScript guards to ensure init message structure matches expectations:

function isValidInitMessage(msg: SDKSystemMessage): boolean {
  return (
    Array.isArray(msg.plugins) &&
    Array.isArray(msg.mcp_servers) &&
    Array.isArray(msg.tools) &&
    Array.isArray(msg.slash_commands)
  );
}

// In verifyPluginLoad:
if (isSystemMessage(message)) {
  if (\!isValidInitMessage(message)) {
    return createFailedResult(
      pluginPath,
      `SDK init message missing required fields`,
      "unknown",
      timings,
    );
  }
  const initResult = processInitMessage(message, pluginPath, timings);
  // ...
}

Pros:

• Type-safe validation
• Clear contract for what we expect
• Self-documenting code

Cons:

• Requires understanding SDK type definitions
• May need to handle optional fields more gracefully

3. Investigate SDK version/compatibility

Check if this is a known SDK issue by reviewing SDK version and release notes.

Implementation Considerations

Files affected

1. Primary change: src/stages/3-execution/plugin-loader.ts:336-354 (processInitMessage function)
2. Type definitions: Potentially src/stages/3-execution/sdk-client.ts if we add type guards
3. Tests: tests/integration/stages/3-execution/plugin-loader.test.ts - add test case for undefined plugins

Testing requirements

Unit tests

Add test case to tests/integration/stages/3-execution/plugin-loader.test.ts:

it("handles init message with undefined plugins array", async () => {
  const queryFn = (input: QueryInput): QueryObject => {
    const initMsg = {
      type: "system" as const,
      subtype: "init" as const,
      session_id: "test-session",
      tools: [],
      slash_commands: [],
      plugins: undefined,  // Explicitly undefined
      mcp_servers: [],
      // ... other required fields
    };

    return {
      [Symbol.asyncIterator]: async function* () {
        yield initMsg;
      },
      // ... other methods
    };
  };

  const result = await verifyPluginLoad({
    pluginPath: validPluginPath,
    config: createMockExecutionConfig(),
    queryFn,
  });

  expect(result.loaded).toBe(false);
  expect(result.error).toContain("missing plugins array");
  expect(result.error_type).toBe("unknown");
});

Breaking changes or migration concerns

None - This is a bug fix that makes the code more defensive. No API changes, no state migration needed.

Impact on state/resume capability

None - This fix happens during plugin verification before any state is written. Resume functionality is unaffected.

Recommendation

Recommended approach

Hybrid solution combining defensive programming and enhanced diagnostics:

1. Immediate fix: Add type guard validation in verifyPluginLoad before calling processInitMessage
2. Enhanced diagnostics: Add debug logging to capture SDK message structure when validation fails
3. Investigation: Check SDK version and add detailed logging to E2E tests

This provides:

• Immediate crash prevention
• Better error messages for debugging
• Type safety for future maintenance
• Diagnostic data to understand the root cause

Implementation steps

1. Add type guard in src/stages/3-execution/plugin-loader.ts
2. Update verifyPluginLoad to validate before processing
3. Add unit test for undefined plugins scenario
4. Add E2E logging to capture actual SDK response structure
5. Run E2E tests to verify the fix
6. Document SDK version compatibility if this is a known SDK behavior

Follow-up issues

1. Investigate SDK init message contract - Document what fields are guaranteed vs optional in SDK system init messages
2. Add SDK version compatibility matrix - Track which SDK versions are tested and known to work
3. Enhanced E2E error reporting - Add structured logging for SDK message streams in E2E tests to help diagnose integration issues

[sjnims]

PR submitted: #312