Optimize CI Pipeline to Reduce Runtime

[gkorland]

Optimize CI Pipeline to Reduce Runtime

Goal

Analyze and optimize the CI/CD pipeline configuration(s) in this repository to significantly reduce overall CI runtime, while maintaining correctness and test coverage.

Context

Our CI pipeline is taking longer than necessary. We want to reduce wall-clock time for CI runs by applying industry best practices for CI performance optimization.

Instructions for the Coding Agent

Please analyze all CI/CD configuration files in this repository (e.g., .github/workflows/*.yml, .circleci/config.yml, Jenkinsfile, .gitlab-ci.yml, azure-pipelines.yml, etc.) and apply the following optimizations where applicable:

1. Caching

• Add or improve caching for package managers (npm, pip, maven, gradle, cargo, go modules, bundler, etc.).
• Add or improve caching for build artifacts, compiled outputs, and intermediate files.
• Ensure cache keys are based on lockfiles or dependency hashes for correctness.

2. Parallelism

• Split large test suites into parallel jobs using matrix strategies or test-splitting tools.
• Run independent jobs concurrently instead of sequentially where possible.
• Use needs: / dependency graphs to ensure jobs only wait on actual dependencies.

3. Job & Step Optimization

• Remove redundant or unnecessary steps (e.g., duplicate installs, unnecessary checkouts).
• Combine small sequential steps that can be merged.
• Use shallow clones (fetch-depth: 1) for checkouts where full history is not required.
• Skip unnecessary jobs on certain triggers (e.g., skip deploy on PRs, skip heavy tests on docs-only changes) using path filters or conditional execution.

4. Docker & Container Optimization

• Use pre-built Docker images with dependencies baked in, instead of installing them at runtime.
• Use multi-stage builds to reduce image size where applicable.
• Cache Docker layers where possible.

5. Dependency Installation

• Use frozen/locked installs (npm ci, pip install --no-deps, bundle install --frozen, etc.) which are faster than resolver-based installs.
• Avoid installing dev/test dependencies in jobs that don't need them.

6. Tool & Action Versions

• Pin to latest stable versions of actions/tools which may include performance improvements.
• Use native/built-in features of the CI platform over custom scripts when faster.

7. Fail Fast

• Enable fail-fast strategies so the pipeline stops early on failure.
• Move linting and static analysis to run first/in parallel since they are typically fast and catch errors early.

Constraints

• Do not reduce test coverage or remove meaningful tests.
• Do not break existing functionality — all currently passing checks must continue to pass.
• Preserve all existing triggers (push, PR, schedule, etc.) unless a change is clearly safe and beneficial.
• Ensure all changes are backward-compatible with the current branch structure and workflows.

Deliverables

• Modified CI configuration file(s) with optimizations applied.
• A summary (in the PR description) of each optimization made and its expected