Feat/rag test suite - #166
Draft
maryamtahhan wants to merge 4 commits into
Draft
Conversation
GuideLLM v0.7.0 introduced breaking CLI changes, moving from env-var-based configuration to structured CLI args. This adds a version-aware abstraction that detects the GuideLLM version and generates the appropriate configuration: env vars for v0.6.x, CLI args for v0.7.x. Key changes: - Bump GuideLLM v0.6.0 → v0.7.1 and vLLM v0.20.0 → v0.25.1 across all source, config, test, and documentation files - Add version branching to the benchmark_guidellm Ansible role for both container mode (v0.6.x env vars vs v0.7.x command: parameter) and host mode (v0.3/v0.4-v0.6/v0.7+ CLI branches) - Add version-aware get_command() and get_env_vars() to GuideLLMLoadGen Python class with _is_v7_or_later property - Add tests for both v0.7.x (CLI args) and v0.6.x (env vars) paths Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>
Add IETF metadata fields (SUT boundary, tokenizer, load model, container images, clock sync, precision, quantization) to test-metadata.json in both playbook paths. Enable poisson/constant profiles. Add P90/P99.9 percentiles to Prometheus metrics template. Raise max_requests to 10000 for IETF sample requirements. Add context-length scaling workloads (1k/4k/8k). Create IETF alignment doc with terminology mapping and compliance status. Update metrics, reporting, overview, testing-phases, and getting-started docs. Fix results.md tree to match actual directory structure. Switch markdownlint pre-commit hook from Docker to native variant and fix pre-existing lint violations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>
Tree structures in 6 documentation files referenced non-existent directories, files, and playbooks. Fixed to match actual repo layout. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>
Three-phase RAG suite measuring context length sensitivity (1K/4K/8K input tokens), variable retrieval distributions, and prefix caching impact. Supports core count sweeps to characterize how RAG prefill cost scales with CPU configuration. Key changes: - New workloads: rag_short, rag_medium, rag_long, rag_var - Dedicated playbook: llm-benchmark-rag-suite.yml with core sweep - Dynamic variable workload lookup in concurrent-load playbook - Model matrix updates with per-model KV cache sizes and eligibility - RAG-appropriate concurrency levels [1,2,4,8,16] Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
depends on #165