Benchmark #11
Closed
Benchmark #11
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…it; create hybrid-retriever.py file; update requirements.txt for new dependencies
…nit_collection method and enhance as_langchain_vectorstore for hybrid retrieval; improve BaseRetriever documentation and remove hybrid-retriever.py file.
…ctor run method to return dense and sparse vectors; update test script to integrate new pipeline functionality and add metadata handling for documents.
… logging for collection creation and document insertion; refactor insert_documents method for improved clarity. Update EmbeddingPipeline to use new splitter method. Modify SparseEmbedder to default to CUDA. Add hybrid_retriever.py file.
…nitialization and embedding retrieval; improve logging and document preparation in test_embedding_pipeline.
…ctor_size parameter for improved clarity.
…ds to retrieve client and collection name. Update EmbeddingPipeline and SparseEmbedder to use Embeddings instead of BaseEmbedder. Add QdrantHybridRetriever for hybrid retrieval functionality and update test scripts accordingly.
… docstrings for clarity.
… hybrid-retriever
Hybrid retriever is functional and ready to deploy to development branch
Hybrid retriever
…er; add text processing pipeline for PDF documents
…; implement metadata handling and enrich documents for upload
…rocessing test script
…ctory in test script
…date requirements for PyMuPDF
…g, and memory updating; add retriever routing logic and logging
…ment generator and retriever nodes
… requirements.txt with dependency version upgrades
- Added 'split' parameter to StackOverflowAdapter for better data segmentation. - Introduced 'dense_embedding' and 'sparse_embedding' fields in ChunkMeta for improved embedding metadata. - Updated EmbeddingPipeline to directly assign embeddings to respective fields. - Made allowed characters in DocumentValidator more permissive for HTML/code content.
- Introduced a comprehensive Quick Start Guide for implementing an MLOps pipeline for RAG systems, covering project initialization, dataset adapters, configuration, processing components, and testing. - Implemented a CSV dataset adapter for reading and converting CSV data into documents. - Created a configuration schema for managing dataset, chunking, embedding, and vector store settings. - Developed core processing components including a document chunker and an embedding pipeline. - Added a simple CLI interface for ingestion with logging and configuration handling. - Implemented a sparse embedding mechanism and integrated it into the embedding pipeline. - Added inspection script for analyzing vector structures in Qdrant. - Created smoke tests for validating ingestion processes and vector store uploads. - Added test script for verifying sparse embedding serialization. - Updated existing configurations for stackoverflow datasets to support hybrid and dense embedding strategies.
… framework - Created `experimental.yml` for testing new components in the retrieval pipeline. - Added `hybrid_multistage.yml` for hybrid retrieval with multi-stage reranking. - Implemented tests for the new answer-focused adapter in `test_new_adapter.py`. - Developed advanced reranking tests in `test_advanced_rerankers.py`. - Introduced answer retrieval tests in `test_answer_retrieval.py`. - Demonstrated retrieval pipeline extensibility in `test_extensibility.py`. - Showcased modular pipeline features in `test_modular_pipeline.py`. - Added a comprehensive test runner in `run_all_tests.py`. - Updated agent retrieval tests to support configurable pipelines in `test_agent_retrieval.py`.
- Implemented unit tests for the RetrievalPipeline, RetrievalResult, and associated components (Retriever, Reranker, Filter) in `test_retrieval_pipeline.py`. - Created mock classes for testing purposes to simulate retrieval, reranking, and filtering behaviors. - Added tests for basic functionality, component addition/removal, and pipeline execution with various configurations. - Introduced tests for the RetrievalPipelineFactory to validate pipeline creation with dense and hybrid configurations. - Added minimal and example tests for the SOSum adapter to ensure basic functionality without heavy dependencies. - Implemented smoke tests for the ingestion process and overall system quality checks. - Updated the test runner to include new tests and organized the test structure for better clarity.
- Updated langchain-core to version 0.3.75 - Added new dependencies: cachetools, distro, filetype, google-ai-generativelanguage, google-api-core, google-auth, googleapis-common-protos, grpcio-status, jiter, langchain-google-genai, langchain-openai, langgraph, langgraph-checkpoint, langgraph-prebuilt, langgraph-sdk, openai, ormsgpack, proto-plus, psycopg2-binary, pyasn1, pyasn1_modules, rsa, tiktoken, xxhash - Updated existing dependencies to their latest versions test: Enhance tests for rerankers and retrieval pipeline - Refactored test cases in test_rerankers.py for better readability and maintainability - Added new tests for the RetrievalPipeline and its components in test_retrieval_pipeline.py - Improved mock implementations for better isolation in tests feat: Add debug scripts for StackOverflow adapter - Introduced debug_row_order.py to check row order and types from StackOverflow data - Added debug_stackoverflow_adapter.py to investigate issues with document reading in the StackOverflow adapter test: Implement tests for ingestion pipeline and adapter functionality - Created test_full_ingestion.py to validate the full ingestion pipeline with StackOverflow data - Added test_adapter_fix.py to verify the StackOverflow adapter produces documents correctly chore: Update test runner to include new tests - Modified run_all_tests.py to include new test files for retrieval and ingestion
- Removed legacy retriever wrapping and introduced modern retriever classes. - Updated `RetrievalPipelineFactory` to create dense, hybrid, sparse, and semantic pipelines using new retriever implementations. - Created `ModernBaseRetriever` as a base class for all retrievers, providing common functionality and configuration handling. - Implemented `QdrantDenseRetriever`, `QdrantHybridRetriever`, `QdrantSparseRetriever`, and `SemanticRetriever` with improved initialization and search methods. - Removed deprecated `router.py` and integrated routing logic into the new retriever classes. - Enhanced logging and error handling across retrievers for better debugging and monitoring. - Updated imports and module structure to reflect the new architecture.
- Enhanced the `load_config` function to include detailed error handling and logging. - Introduced `get_retriever_config`, `get_benchmark_config`, and `get_pipeline_config` functions for better configuration management. - Added `load_config_with_overrides` to support configuration overrides. - Updated `QdrantVectorDB` initialization to accept configuration parameters. - Created example scripts for unified configuration usage and retriever configuration examples. - Removed outdated retrieval pipeline configurations to streamline the codebase. - Improved logging throughout the retriever classes for better traceability.
…ality - Refactored benchmark runner to utilize unified configuration approach. - Updated imports to reflect new module structure for benchmarks and metrics. - Introduced new benchmark scripts for simple and full dataset evaluations. - Enhanced retrieval pipeline initialization to support unified config. - Created comprehensive dataset adapter for full StackOverflow dataset evaluation. - Added real data benchmark runner for testing with actual StackOverflow queries. - Updated configuration file to include new retrieval strategies and parameters. - Documented changes in CONFIG_CONSOLIDATION_COMPLETE.md and UNIFIED_CONFIG.md. - Added examples demonstrating the use of the new unified configuration system.
…issing ground truth and improving document ID extraction
…and configuration - Removed the full dataset benchmark script to streamline the benchmarking process. - Updated the real benchmark script to ensure proper imports and functionality. - Enhanced the configuration file to include new fusion methods and adjustable weights for hybrid retrieval. - Refactored dense and sparse retrievers to improve embedding initialization and search processes. - Implemented a new hybrid retriever that combines dense and sparse results using configurable fusion methods. - Deleted the synthetic dataset text processing script to clean up unused code. - Added a comprehensive test suite for all retrievers in the full benchmark pipeline to ensure reliability and performance.
…flow - Created `natural_questions.yml` for Google Natural Questions dataset with hybrid embedding strategy, chunking, validation, and evaluation settings. - Created `stackoverflow.yml` for SOSum dataset with hybrid embedding strategy, chunking, validation, and evaluation settings. - Added `stackoverflow_hybrid.yml` for hybrid dense and sparse embeddings configuration. - Introduced dataset template `dataset_template.yml` for easy dataset configuration. - Added retrieval configuration templates: `retrieval_template.yml` for agent retrieval setup. - Implemented legacy configurations for various models: `stackoverflow_bge_large.yml`, `stackoverflow_e5_large.yml`, and `stackoverflow_minilm.yml`. - Created high-performance retrieval configurations: `fast_hybrid.yml`, `modern_dense.yml`, and `modern_hybrid.yml`. - Removed outdated retriever configurations: `dense_retriever.yml`, `hybrid_retriever.yml`, `semantic_retriever.yml`, and `sparse_retriever.yml`. - Updated tests for agent retrieval and streamlined agent functionality, ensuring compatibility with new configurations.
…script - Deleted the following test files: - test_full_ingestion.py - test_modular_pipeline.py - run_all_tests.py - test_adapter_fix.py - test_agent_retrieval.py - test_retriever_direct.py - test_streamlined_agent.py - Added a new test file: test_local_setup.py - This script checks prerequisites and runs progressive tests for the pipeline.
…t.txt and updating pipeline tests to use requirements-minimal.txt
…ation structure for improved clarity and maintainability. - Deleted SYSTEM_EXTENSION_GUIDE.md, UNIFIED_CONFIG.md, agent_retrieval_upgrade_summary.md, config_reorganization_summary.md, integration_testing_setup.md, and sql_removal_summary.md. - Simplified agent graph by removing SQL-related nodes and dependencies. - Consolidated configuration files into a unified structure, enhancing usability and reducing clutter.
…s for LangChain and dotenv
…rant in requirements-minimal.txt
…rements documentation
…iever configuration
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces a production-ready, modular Retrieval-Augmented Generation (RAG) system with a configurable LangGraph agent and robust CI/CD and deployment support. The main changes include a comprehensive new workflow for pipeline testing and security, a Dockerfile for containerization, a detailed project README, and the initial implementation of the agent's modular graph and nodes.
Key highlights:
1. CI/CD and Deployment
.github/workflows/pipeline-tests.yml) with jobs for minimal, integration (with Qdrant), end-to-end (with API), and security/config validation tests, including checks for hardcoded secrets and YAML config validation.Dockerfilefor building a slim Python 3.11-based container, installing system and Python dependencies, and copying the full source code for production deployment.2. Documentation
README.mdwith a detailed overview of the system, including features, architecture, quick start, configuration examples, project structure, testing, extension guides, and migration instructions from legacy code.3. Agent Graph Implementation
agent/graph.pyusing LangGraph, with nodes for query interpretation, retrieval, generation, and memory update, all configurable via YAML.4. Modular Agent Nodes
agent/nodes/query_interpreter.py: Determines if a question needs retrieval or can be answered directly, using a prompt and robust error handling.agent/nodes/generator.py: Generates answers using context or fallback, with logging and exception handling.agent/nodes/memory_updater.py: Maintains a rolling chat history in the agent state.