ChatQnA Example with OpenAI-Compatible Endpoint

[edlee123]

Description

Allows ChatQnA to be used with thousands of OpenAI-like endpoints e.g. OpenRouter.ai, Hugging Face, Denvr, and improve the developer experience to use OPEA quickly even on low resource environments (does not require spinning up a inference service).

Key Changes Made

1. Created ChatQnA/docker_compose/intel/cpu/xeon/README_endpoint_openai.md: instructions to spin up example.
2. Created ChatQnA/docker_compose/intel/cpu/xeon/compose_endpoint_openai.yaml: replaces vLLM with an opeai-like endpoint .

Also:

• Fixed align_generator function to properly detect and skip chunks where content is null in open-ai like endpoints. Previously it'd show the null json in the UI.
• Added better error handling and debug logging for easier troubleshooting of endpoint issues.

Issues

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)
• Others (enhancement, documentation, validation, etc.)

Dependencies

Tests

Tested on:
OpenRouter.ai: anthropic/claude-3.7-sonnet
Denvr: meta-llama/Llama-3.1-70B-Instruct

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

None

[Copilot]

Pull Request Overview

This PR extends ChatQnA to support OpenAI-compatible endpoints and improves robustness by enhancing environment variable handling, skipping null content chunks, and adding detailed logging for debugging.

• Introduces LLM_MODEL_ID validation in frontend and dynamically sets the model in requests.
• Refactors align_generator, align_inputs, and align_outputs to skip null or malformed content and use structured logging.
• Adds new Docker Compose configurations and READMEs for spinning up ChatQnA and GraphRAG services with external endpoints.

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
GraphRAG/ui/svelte/src/lib/network/chat/Network.ts	Validates and injects LLM_MODEL_ID into text requests
GraphRAG/graphrag.py	Replaced prints with logger, skipped null chunks
GraphRAG/docker_compose/intel/*	Added LLM_MODEL_ID env and endpoint README/composes
ChatQnA/docker_compose/intel/cpu/xeon/compose_endpoint_openai.yaml	Swapped vLLM for an OpenAI-compatible textgen service
ChatQnA/chatqna.py	Added structured logging to alignment methods
GraphRAG/example_data/programming_languages{,2}.txt + README.md	Expanded example datasets and documentation

Comments suppressed due to low confidence (3)

GraphRAG/graphrag.py:203

• Avoid bare except: which catches all exceptions; specify the exception type to prevent masking unexpected errors.

except:

GraphRAG/graphrag.py:131

• The new JSON parsing logic in align_generator should be covered by unit tests for TGI, OpenAI, and malformed chunks to prevent regressions.

def align_generator(self, gen, **kwargs):

GraphRAG/graphrag.py:141

• This if appears over-indented relative to its surrounding lines and may introduce a syntax error; verify the indentation block.

if start == -1 or end <= start:

[Copilot]

The time, uuid, and typing imports are unused; consider removing them to reduce clutter.

Suggested change

	import time
	import uuid
	from typing import Dict, List, Union
	# Removed unused imports: time, uuid, typing

[Copilot]

[nitpick] Throwing an error at module import may break SSR or bundling; consider checking LLM_MODEL_ID inside the fetchTextStream function or providing a fallback.

Suggested change

	if (!env.LLM_MODEL_ID) {
	throw new Error("LLM_MODEL_ID environment variable must be set");
	}
	const LLM_MODEL_ID = env.LLM_MODEL_ID;
	export async function fetchTextStream(query: string) {
	let payload = {};
	let url = "";
	export async function fetchTextStream(query: string) {
	let payload = {};
	let url = "";
	const LLM_MODEL_ID = env.LLM_MODEL_ID || "default-model-id"; // Fallback value
	if (!env.LLM_MODEL_ID) {
	console.warn("LLM_MODEL_ID environment variable is not set. Using fallback value.");
	}

[Copilot]

[nitpick] It would be helpful to add a healthcheck for the textgen-service-endpoint-openai to ensure the service is ready before backend startup.

[edlee123]

i mixed two branches together. going to make a clean PR.