Skip to content

Conversation

kuiwang02
Copy link
Contributor

@kuiwang02 kuiwang02 commented Oct 21, 2025

Add Comprehensive AGENTS.md Guide for OLM v0 QE Test Extension Migration

Why / Problem Statement

The OLM v0 QE Test Extension project (tests-extension/) is a specialized testing framework for validating OLM v0 functionality in OpenShift clusters. As the project grows with migrated test cases from openshift-tests-private and new test development, there is a critical need for:

  1. AI Agent Guidance: AI coding assistants (like Claude Code) need comprehensive context about:

    • Test framework architecture and patterns
    • Migration guidelines from tests-private to tests-extension
    • Suite definitions and label requirements
    • Code quality standards and anti-patterns to avoid
  2. Developer Onboarding: New QE engineers and contributors need clear documentation covering:

    • Test structure patterns for different cluster topologies (Microshift, HyperShift, SNO, etc.)
    • Suite selection logic (parallel, serial, extended, candidate, etc.)
    • Build workflow and local testing procedures
  3. Consistency: Without documented standards, test cases may be written inconsistently, leading to:

    • Incorrect labeling (Extended, ReleaseGate, StressTest, etc.)
    • Missing cleanup logic causing resource leaks
    • Non-idiomatic patterns that fail in CI

This PR addresses these needs by providing a single, authoritative guide that serves both AI agents and human developers.

What / Solution

Added comprehensive documentation to guide AI agents and developers working with the OLM v0 QE Test Extension:

Changes Made

  • tests-extension/AGENTS.md:1-1179 - Comprehensive AI agent guide covering:

    • Project Scope (lines 1-86): Working directory context, applicability rules
    • Project Overview (lines 88-102): Purpose, framework, characteristics
    • Test Case Sources (lines 103-128): Origin vs tests-private migration, suite selection logic
    • Suite Definitions (lines 130-280): Detailed breakdown of all test suites with qualifiers
    • Test Architecture (lines 494-652): Standard test patterns, skip functions, cluster detection
    • Migration Guide (lines 332-493): Code changes needed, label requirements
    • Local Development (lines 654-734): Build workflow, testing procedures, PR requirements
    • Test File Organization (lines 838-868): Mapping between old and new test files
    • Anti-Patterns (lines 969-1014): Common mistakes to avoid
    • Product Code References (lines 1070-1119): Links to OLM source code for debugging
  • tests-extension/CLAUDE.md:1 - Pointer file directing to AGENTS.md for Claude Code integration

Key Technical Decisions

  1. AGENTS.md Format: Chose comprehensive markdown over separate docs because:

    • AI agents perform better with complete context in a single file
    • Easier to maintain than multiple scattered documents
    • Serves dual purpose: AI guidance + human reference
  2. Context Awareness Section: Added explicit rules for when to apply this guidance (lines 26-73) to prevent:

    • AI agents from applying test-specific patterns to product code
    • Confusion when working in different parts of the monorepo
  3. Suite Definition Details: Included exact qualifier expressions (lines 130-280) because:

    • Suite definitions in cmd/main.go may change over time
    • Understanding filter logic is critical for proper labeling
    • Helps developers choose the right suite for new tests
  4. Migration Guide Completeness: Documented ALL code changes needed from tests-private migration:

    • Function name changes (exutil.By()g.By())
    • Type prefix changes (all types need olmv0util. prefix)
    • Constant changes (all booleans need exutil. prefix)
    • Prevents common migration errors
  5. Anti-Patterns Section: Explicitly called out problematic patterns with examples:

    • No o.Expect in wait.Poll loops
    • No quotes in test titles (causes XML parse failures)
    • No logic in g.Describe blocks
    • These are actual issues found in migration work
  6. Test Structure Patterns: Provided three separate patterns for:

    • Standard tests (skip on Microshift)
    • Microshift-only tests
    • HyperShift management tests
    • Each has different setup requirements

Testing

This documentation has been validated through:

  • Manual review: Verified all sections are accurate and complete
  • Cross-referencing: Checked against actual codebase structure
    • Suite definitions match cmd/main.go:1-319
    • Test patterns match test/qe/specs/*.go
    • Migration guide matches actual migrated code
  • AI Agent Testing: Verified Claude Code can load and use this context effectively
  • File paths: All referenced paths are correct relative to tests-extension/

Additional Context

Why AGENTS.md Now?

This documentation is being added at a critical juncture:

  1. Test Migration: Hundreds of test cases migrating from tests-private to tests-extension
  2. Suite Expansion: New suite types being added (extended/candidate/stress)
  3. Team Growth: New QE engineers onboarding to the project
  4. AI Adoption: Increased use of AI coding assistants in test development

Documentation Philosophy

This AGENTS.md follows emerging best practices for AI-era codebases:

  • Comprehensive over Concise: AI agents benefit from complete context
  • Explicit over Implicit: State rules directly rather than expecting inference
  • Examples over Descriptions: Show actual code patterns, not just descriptions
  • Anti-patterns Documented: Explicitly state what NOT to do

Relationship to Other Documentation

  • README.md: User-facing quick start and usage guide
  • AGENTS.md: AI agent + developer comprehensive reference (this PR)
  • CLAUDE.md: Pointer for Claude Code integration

Future Enhancements

While this PR provides comprehensive documentation, future additions may include:

  • Additional examples as new patterns emerge
  • Updates to suite definitions as they evolve in cmd/main.go
  • New sections for additional test types (stress tests, upgrade tests, etc.)

Note: This documentation is specifically for the tests-extension/ directory and should NOT be applied when working with product code in other parts of the operator-framework-olm repository.

Assisted-by: Claude Code

/cc @jianzhangbjz @Xia-Zhao-rh @bandrade

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Oct 21, 2025
@openshift-ci-robot
Copy link

openshift-ci-robot commented Oct 21, 2025

@kuiwang02: This pull request references OCPQE-30995 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.21.0" version, but no target version was set.

In response to this:

Add Comprehensive AGENTS.md Guide for OLM v0 QE Test Extension Migration

Why / Problem Statement

The OLM v0 QE Test Extension project (tests-extension/) is a specialized testing framework for validating OLM v0 functionality in OpenShift clusters. As the project grows with migrated test cases from openshift-tests-private and new test development, there is a critical need for:

  1. AI Agent Guidance: AI coding assistants (like Claude Code) need comprehensive context about:
  • Test framework architecture and patterns
  • Migration guidelines from tests-private to tests-extension
  • Suite definitions and label requirements
  • Code quality standards and anti-patterns to avoid
  1. Developer Onboarding: New QE engineers and contributors need clear documentation covering:
  • Test structure patterns for different cluster topologies (Microshift, HyperShift, SNO, etc.)
  • Suite selection logic (parallel, serial, extended, candidate, etc.)
  • Build workflow and local testing procedures
  1. Consistency: Without documented standards, test cases may be written inconsistently, leading to:
  • Incorrect labeling (Extended, ReleaseGate, StressTest, etc.)
  • Missing cleanup logic causing resource leaks
  • Non-idiomatic patterns that fail in CI

This PR addresses these needs by providing a single, authoritative guide that serves both AI agents and human developers.

What / Solution

Added comprehensive documentation to guide AI agents and developers working with the OLM v0 QE Test Extension:

Changes Made

  • tests-extension/AGENTS.md:1-1179 - Comprehensive AI agent guide covering:

  • Project Scope (lines 1-86): Working directory context, applicability rules

  • Project Overview (lines 88-102): Purpose, framework, characteristics

  • Test Case Sources (lines 103-128): Origin vs tests-private migration, suite selection logic

  • Suite Definitions (lines 130-280): Detailed breakdown of all test suites with qualifiers

  • Test Architecture (lines 494-652): Standard test patterns, skip functions, cluster detection

  • Migration Guide (lines 332-493): Code changes needed, label requirements

  • Local Development (lines 654-734): Build workflow, testing procedures, PR requirements

  • Test File Organization (lines 838-868): Mapping between old and new test files

  • Anti-Patterns (lines 969-1014): Common mistakes to avoid

  • Product Code References (lines 1070-1119): Links to OLM source code for debugging

  • tests-extension/CLAUDE.md:1 - Pointer file directing to AGENTS.md for Claude Code integration

Key Technical Decisions

  1. AGENTS.md Format: Chose comprehensive markdown over separate docs because:
  • AI agents perform better with complete context in a single file
  • Easier to maintain than multiple scattered documents
  • Serves dual purpose: AI guidance + human reference
  1. Context Awareness Section: Added explicit rules for when to apply this guidance (lines 26-73) to prevent:
  • AI agents from applying test-specific patterns to product code
  • Confusion when working in different parts of the monorepo
  1. Suite Definition Details: Included exact qualifier expressions (lines 130-280) because:
  • Suite definitions in cmd/main.go may change over time
  • Understanding filter logic is critical for proper labeling
  • Helps developers choose the right suite for new tests
  1. Migration Guide Completeness: Documented ALL code changes needed from tests-private migration:
  • Function name changes (exutil.By()g.By())
  • Type prefix changes (all types need olmv0util. prefix)
  • Constant changes (all booleans need exutil. prefix)
  • Prevents common migration errors
  1. Anti-Patterns Section: Explicitly called out problematic patterns with examples:
  • No o.Expect in wait.Poll loops
  • No quotes in test titles (causes XML parse failures)
  • No logic in g.Describe blocks
  • These are actual issues found in migration work
  1. Test Structure Patterns: Provided three separate patterns for:
  • Standard tests (skip on Microshift)
  • Microshift-only tests
  • HyperShift management tests
  • Each has different setup requirements

Testing

This documentation has been validated through:

  • Manual review: Verified all sections are accurate and complete
  • Cross-referencing: Checked against actual codebase structure
  • Suite definitions match cmd/main.go:1-319
  • Test patterns match test/qe/specs/*.go
  • Migration guide matches actual migrated code
  • AI Agent Testing: Verified Claude Code can load and use this context effectively
  • File paths: All referenced paths are correct relative to tests-extension/

Additional Context

Why AGENTS.md Now?

This documentation is being added at a critical juncture:

  1. Test Migration: Hundreds of test cases migrating from tests-private to tests-extension
  2. Suite Expansion: New suite types being added (extended/candidate/stress)
  3. Team Growth: New QE engineers onboarding to the project
  4. AI Adoption: Increased use of AI coding assistants in test development

Documentation Philosophy

This AGENTS.md follows emerging best practices for AI-era codebases:

  • Comprehensive over Concise: AI agents benefit from complete context
  • Explicit over Implicit: State rules directly rather than expecting inference
  • Examples over Descriptions: Show actual code patterns, not just descriptions
  • Anti-patterns Documented: Explicitly state what NOT to do

Relationship to Other Documentation

  • README.md: User-facing quick start and usage guide
  • AGENTS.md: AI agent + developer comprehensive reference (this PR)
  • CLAUDE.md: Pointer for Claude Code integration

Future Enhancements

While this PR provides comprehensive documentation, future additions may include:

  • Additional examples as new patterns emerge
  • Updates to suite definitions as they evolve in cmd/main.go
  • New sections for additional test types (stress tests, upgrade tests, etc.)

Note: This documentation is specifically for the tests-extension/ directory and should NOT be applied when working with product code in other parts of the operator-framework-olm repository.

Assisted-by: Claude Code

/cc @jianzhangbjz @Xia-Zhao-rh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Oct 21, 2025

@kuiwang02: This pull request references OCPQE-30995 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.21.0" version, but no target version was set.

In response to this:

Add Comprehensive AGENTS.md Guide for OLM v0 QE Test Extension Migration

Why / Problem Statement

The OLM v0 QE Test Extension project (tests-extension/) is a specialized testing framework for validating OLM v0 functionality in OpenShift clusters. As the project grows with migrated test cases from openshift-tests-private and new test development, there is a critical need for:

  1. AI Agent Guidance: AI coding assistants (like Claude Code) need comprehensive context about:
  • Test framework architecture and patterns
  • Migration guidelines from tests-private to tests-extension
  • Suite definitions and label requirements
  • Code quality standards and anti-patterns to avoid
  1. Developer Onboarding: New QE engineers and contributors need clear documentation covering:
  • Test structure patterns for different cluster topologies (Microshift, HyperShift, SNO, etc.)
  • Suite selection logic (parallel, serial, extended, candidate, etc.)
  • Build workflow and local testing procedures
  1. Consistency: Without documented standards, test cases may be written inconsistently, leading to:
  • Incorrect labeling (Extended, ReleaseGate, StressTest, etc.)
  • Missing cleanup logic causing resource leaks
  • Non-idiomatic patterns that fail in CI

This PR addresses these needs by providing a single, authoritative guide that serves both AI agents and human developers.

What / Solution

Added comprehensive documentation to guide AI agents and developers working with the OLM v0 QE Test Extension:

Changes Made

  • tests-extension/AGENTS.md:1-1179 - Comprehensive AI agent guide covering:

  • Project Scope (lines 1-86): Working directory context, applicability rules

  • Project Overview (lines 88-102): Purpose, framework, characteristics

  • Test Case Sources (lines 103-128): Origin vs tests-private migration, suite selection logic

  • Suite Definitions (lines 130-280): Detailed breakdown of all test suites with qualifiers

  • Test Architecture (lines 494-652): Standard test patterns, skip functions, cluster detection

  • Migration Guide (lines 332-493): Code changes needed, label requirements

  • Local Development (lines 654-734): Build workflow, testing procedures, PR requirements

  • Test File Organization (lines 838-868): Mapping between old and new test files

  • Anti-Patterns (lines 969-1014): Common mistakes to avoid

  • Product Code References (lines 1070-1119): Links to OLM source code for debugging

  • tests-extension/CLAUDE.md:1 - Pointer file directing to AGENTS.md for Claude Code integration

Key Technical Decisions

  1. AGENTS.md Format: Chose comprehensive markdown over separate docs because:
  • AI agents perform better with complete context in a single file
  • Easier to maintain than multiple scattered documents
  • Serves dual purpose: AI guidance + human reference
  1. Context Awareness Section: Added explicit rules for when to apply this guidance (lines 26-73) to prevent:
  • AI agents from applying test-specific patterns to product code
  • Confusion when working in different parts of the monorepo
  1. Suite Definition Details: Included exact qualifier expressions (lines 130-280) because:
  • Suite definitions in cmd/main.go may change over time
  • Understanding filter logic is critical for proper labeling
  • Helps developers choose the right suite for new tests
  1. Migration Guide Completeness: Documented ALL code changes needed from tests-private migration:
  • Function name changes (exutil.By()g.By())
  • Type prefix changes (all types need olmv0util. prefix)
  • Constant changes (all booleans need exutil. prefix)
  • Prevents common migration errors
  1. Anti-Patterns Section: Explicitly called out problematic patterns with examples:
  • No o.Expect in wait.Poll loops
  • No quotes in test titles (causes XML parse failures)
  • No logic in g.Describe blocks
  • These are actual issues found in migration work
  1. Test Structure Patterns: Provided three separate patterns for:
  • Standard tests (skip on Microshift)
  • Microshift-only tests
  • HyperShift management tests
  • Each has different setup requirements

Testing

This documentation has been validated through:

  • Manual review: Verified all sections are accurate and complete
  • Cross-referencing: Checked against actual codebase structure
  • Suite definitions match cmd/main.go:1-319
  • Test patterns match test/qe/specs/*.go
  • Migration guide matches actual migrated code
  • AI Agent Testing: Verified Claude Code can load and use this context effectively
  • File paths: All referenced paths are correct relative to tests-extension/

Additional Context

Why AGENTS.md Now?

This documentation is being added at a critical juncture:

  1. Test Migration: Hundreds of test cases migrating from tests-private to tests-extension
  2. Suite Expansion: New suite types being added (extended/candidate/stress)
  3. Team Growth: New QE engineers onboarding to the project
  4. AI Adoption: Increased use of AI coding assistants in test development

Documentation Philosophy

This AGENTS.md follows emerging best practices for AI-era codebases:

  • Comprehensive over Concise: AI agents benefit from complete context
  • Explicit over Implicit: State rules directly rather than expecting inference
  • Examples over Descriptions: Show actual code patterns, not just descriptions
  • Anti-patterns Documented: Explicitly state what NOT to do

Relationship to Other Documentation

  • README.md: User-facing quick start and usage guide
  • AGENTS.md: AI agent + developer comprehensive reference (this PR)
  • CLAUDE.md: Pointer for Claude Code integration

Future Enhancements

While this PR provides comprehensive documentation, future additions may include:

  • Additional examples as new patterns emerge
  • Updates to suite definitions as they evolve in cmd/main.go
  • New sections for additional test types (stress tests, upgrade tests, etc.)

Note: This documentation is specifically for the tests-extension/ directory and should NOT be applied when working with product code in other parts of the operator-framework-olm repository.

Assisted-by: Claude Code

/cc @jianzhangbjz @Xia-Zhao-rh @bandrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@kuiwang02
Copy link
Contributor Author

/cc @bandrade

@openshift-ci openshift-ci bot requested a review from bandrade October 21, 2025 01:11
@jianzhangbjz
Copy link
Contributor

/approve
/lgtm

@openshift-ci openshift-ci bot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Oct 21, 2025
@kuiwang02
Copy link
Contributor Author

/verified by @kuiwang02

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Oct 21, 2025
@openshift-ci-robot
Copy link

@kuiwang02: This PR has been marked as verified by @kuiwang02.

In response to this:

/verified by @kuiwang02

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD bdc0d75 and 2 for PR HEAD fa5bff9 in total

@openshift-ci-robot openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Oct 21, 2025
@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 21, 2025
@kuiwang02
Copy link
Contributor Author

/verified by @kuiwang02

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Oct 21, 2025
@openshift-ci-robot
Copy link

@kuiwang02: This PR has been marked as verified by @kuiwang02.

In response to this:

/verified by @kuiwang02

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@jianzhangbjz
Copy link
Contributor

/approve
/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 21, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Oct 21, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jianzhangbjz, kuiwang02

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD bdc0d75 and 2 for PR HEAD 92bbdca in total

1. **Start Claude Code from tests-extension directory**:
```bash
cd tests-extension/
# Then launch Claude Code from this directory
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to add the specific Command here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no specfic command. just claude

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated. thanks

# Then launch Claude Code from this directory
```
2. **Load AGENTS.md**: When prompted by Claude Code to load the AGENTS.md file, select **Yes** to enable comprehensive context about the OLM v0 QE test framework.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How to load this AGENT? Thanks!
image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no. it is agent.md, not agent. it is diffrent thing

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, thanks! I got mixed up between the sub-agent and AGENT.md. We should use the /memory command to check whether Claude has successfully loaded the CLAUDE.md file.
image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated. thanks

@openshift-ci-robot openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Oct 21, 2025
@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 21, 2025
@kuiwang02
Copy link
Contributor Author

/verified by @kuiwang02

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Oct 21, 2025
@openshift-ci-robot
Copy link

@kuiwang02: This PR has been marked as verified by @kuiwang02.

In response to this:

/verified by @kuiwang02

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@jianzhangbjz
Copy link
Contributor

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 21, 2025
@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD bdc0d75 and 2 for PR HEAD a63e791 in total

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD a0b88f9 and 1 for PR HEAD a63e791 in total

@jianzhangbjz
Copy link
Contributor

/retest-required

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Oct 22, 2025

@kuiwang02: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/verify-deps a63e791 link true /test verify-deps

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants