ChatQnA Example with OpenAI-Compatible Endpoint

[edlee123]

Description

Allows ChatQnA to be used with thousands of OpenAI-like endpoints e.g. OpenRouter.ai, Hugging Face, Denvr, and improve the developer experience to use OPEA quickly even on low resource environments. Using endpoints would not require spinning up a inference service.

Key Changes Made:

1. Created ChatQnA/docker_compose/intel/cpu/xeon/README_endpoint_openai.md: instructions to spin up example.
2. Created ChatQnA/docker_compose/intel/cpu/xeon/compose_endpoint_openai.yaml: replaces vLLM with an opeai-like endpoint

Also:

• Fixed align_generator function to properly detect and skip chunks where content is null in open-ai like endpoints. Previously it'd show the null json in the UI.
• Added better error handling and debug logging for easier troubleshooting of endpoint issues.

Issues

Helps address issue:

• 1972
• 1973

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)
• Others (enhancement, documentation, validation, etc.)

Dependencies

No new third party dependencies, but uses recent updated textgen component:
https://github.com/opea-project/

GenAIComps/pull/1395 (the component was updated to filter null inputs. Some openai standard null inputs are disallowed by OpenRouter.ai).

Tests

Tested on:
OpenRouter.ai: anthropic/claude-3.7-sonnet
Denvr: meta-llama/Llama-3.1-70B-Instruct

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

None

[Copilot]

Pull Request Overview

This PR enables ChatQnA to run against OpenAI-compatible endpoints via a new Docker Compose setup and enhances the core Python logic to better handle and log streaming responses.

• Added compose_endpoint_openai.yaml for spinning up ChatQnA with OPEA text-generation service backed by OpenAI-like endpoints.
• Updated chatqna.py to improve logging, skip null-content chunks in the generator, and bolster error handling.
• Fixed align_generator to filter out invalid or null JSON and added debug/error logs.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
ChatQnA/docker_compose/intel/cpu/xeon/compose_endpoint_openai.yaml	New Compose file replacing vLLM with an OpenAI-compatible endpoint and related services.
ChatQnA/chatqna.py	Enhanced input/output alignment, added logging, and refactored align_generator.

Comments suppressed due to low confidence (4)

ChatQnA/docker_compose/intel/cpu/xeon/compose_endpoint_openai.yaml:98

• [nitpick] Component name 'OpeaTextGenService' is inconsistent with other OPEA naming conventions; consider using uppercase 'OPEA' or aligning with existing style.

LLM_COMPONENT_NAME: OpeaTextGenService

ChatQnA/docker_compose/intel/cpu/xeon/compose_endpoint_openai.yaml:104

• [nitpick] The comment says 'Add OpenRouter API Key' but the variable is 'OPENAI_API_KEY'; clarify the env var or comment to avoid confusion.

OPENAI_API_KEY: ${OPENAI_API_KEY} # Add OpenRouter API Key

ChatQnA/docker_compose/intel/cpu/xeon/compose_endpoint_openai.yaml:166

• [nitpick] BACKEND_SERVICE_NAME 'chatqna' does not match the container name 'chatqna-xeon-backend-server'; consider aligning these to prevent misconfiguration.

- BACKEND_SERVICE_NAME=chatqna

ChatQnA/chatqna.py:189

• The updated align_generator logic introduces new behaviors (skipping null content, error handling) but lacks corresponding unit tests; consider adding tests to cover these scenarios.

def align_generator(self, gen, **kwargs):

[edlee123]

I accepted a Co-pilot commit but wouldn't let me sign the commit after that. Will create a new PR.