Benchmark #15
Closed
Benchmark #15
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…it; create hybrid-retriever.py file; update requirements.txt for new dependencies
…nit_collection method and enhance as_langchain_vectorstore for hybrid retrieval; improve BaseRetriever documentation and remove hybrid-retriever.py file.
…ctor run method to return dense and sparse vectors; update test script to integrate new pipeline functionality and add metadata handling for documents.
… logging for collection creation and document insertion; refactor insert_documents method for improved clarity. Update EmbeddingPipeline to use new splitter method. Modify SparseEmbedder to default to CUDA. Add hybrid_retriever.py file.
…nitialization and embedding retrieval; improve logging and document preparation in test_embedding_pipeline.
…ctor_size parameter for improved clarity.
…ds to retrieve client and collection name. Update EmbeddingPipeline and SparseEmbedder to use Embeddings instead of BaseEmbedder. Add QdrantHybridRetriever for hybrid retrieval functionality and update test scripts accordingly.
… docstrings for clarity.
… hybrid-retriever
Hybrid retriever is functional and ready to deploy to development branch
Hybrid retriever
…er; add text processing pipeline for PDF documents
…; implement metadata handling and enrich documents for upload
…rocessing test script
…ctory in test script
…date requirements for PyMuPDF
…g, and memory updating; add retriever routing logic and logging
…ment generator and retriever nodes
… requirements.txt with dependency version upgrades
… framework - Created `experimental.yml` for testing new components in the retrieval pipeline. - Added `hybrid_multistage.yml` for hybrid retrieval with multi-stage reranking. - Implemented tests for the new answer-focused adapter in `test_new_adapter.py`. - Developed advanced reranking tests in `test_advanced_rerankers.py`. - Introduced answer retrieval tests in `test_answer_retrieval.py`. - Demonstrated retrieval pipeline extensibility in `test_extensibility.py`. - Showcased modular pipeline features in `test_modular_pipeline.py`. - Added a comprehensive test runner in `run_all_tests.py`. - Updated agent retrieval tests to support configurable pipelines in `test_agent_retrieval.py`.
- Implemented unit tests for the RetrievalPipeline, RetrievalResult, and associated components (Retriever, Reranker, Filter) in `test_retrieval_pipeline.py`. - Created mock classes for testing purposes to simulate retrieval, reranking, and filtering behaviors. - Added tests for basic functionality, component addition/removal, and pipeline execution with various configurations. - Introduced tests for the RetrievalPipelineFactory to validate pipeline creation with dense and hybrid configurations. - Added minimal and example tests for the SOSum adapter to ensure basic functionality without heavy dependencies. - Implemented smoke tests for the ingestion process and overall system quality checks. - Updated the test runner to include new tests and organized the test structure for better clarity.
- Updated langchain-core to version 0.3.75 - Added new dependencies: cachetools, distro, filetype, google-ai-generativelanguage, google-api-core, google-auth, googleapis-common-protos, grpcio-status, jiter, langchain-google-genai, langchain-openai, langgraph, langgraph-checkpoint, langgraph-prebuilt, langgraph-sdk, openai, ormsgpack, proto-plus, psycopg2-binary, pyasn1, pyasn1_modules, rsa, tiktoken, xxhash - Updated existing dependencies to their latest versions test: Enhance tests for rerankers and retrieval pipeline - Refactored test cases in test_rerankers.py for better readability and maintainability - Added new tests for the RetrievalPipeline and its components in test_retrieval_pipeline.py - Improved mock implementations for better isolation in tests feat: Add debug scripts for StackOverflow adapter - Introduced debug_row_order.py to check row order and types from StackOverflow data - Added debug_stackoverflow_adapter.py to investigate issues with document reading in the StackOverflow adapter test: Implement tests for ingestion pipeline and adapter functionality - Created test_full_ingestion.py to validate the full ingestion pipeline with StackOverflow data - Added test_adapter_fix.py to verify the StackOverflow adapter produces documents correctly chore: Update test runner to include new tests - Modified run_all_tests.py to include new test files for retrieval and ingestion
- Removed legacy retriever wrapping and introduced modern retriever classes. - Updated `RetrievalPipelineFactory` to create dense, hybrid, sparse, and semantic pipelines using new retriever implementations. - Created `ModernBaseRetriever` as a base class for all retrievers, providing common functionality and configuration handling. - Implemented `QdrantDenseRetriever`, `QdrantHybridRetriever`, `QdrantSparseRetriever`, and `SemanticRetriever` with improved initialization and search methods. - Removed deprecated `router.py` and integrated routing logic into the new retriever classes. - Enhanced logging and error handling across retrievers for better debugging and monitoring. - Updated imports and module structure to reflect the new architecture.
- Enhanced the `load_config` function to include detailed error handling and logging. - Introduced `get_retriever_config`, `get_benchmark_config`, and `get_pipeline_config` functions for better configuration management. - Added `load_config_with_overrides` to support configuration overrides. - Updated `QdrantVectorDB` initialization to accept configuration parameters. - Created example scripts for unified configuration usage and retriever configuration examples. - Removed outdated retrieval pipeline configurations to streamline the codebase. - Improved logging throughout the retriever classes for better traceability.
…ality - Refactored benchmark runner to utilize unified configuration approach. - Updated imports to reflect new module structure for benchmarks and metrics. - Introduced new benchmark scripts for simple and full dataset evaluations. - Enhanced retrieval pipeline initialization to support unified config. - Created comprehensive dataset adapter for full StackOverflow dataset evaluation. - Added real data benchmark runner for testing with actual StackOverflow queries. - Updated configuration file to include new retrieval strategies and parameters. - Documented changes in CONFIG_CONSOLIDATION_COMPLETE.md and UNIFIED_CONFIG.md. - Added examples demonstrating the use of the new unified configuration system.
…issing ground truth and improving document ID extraction
…and configuration - Removed the full dataset benchmark script to streamline the benchmarking process. - Updated the real benchmark script to ensure proper imports and functionality. - Enhanced the configuration file to include new fusion methods and adjustable weights for hybrid retrieval. - Refactored dense and sparse retrievers to improve embedding initialization and search processes. - Implemented a new hybrid retriever that combines dense and sparse results using configurable fusion methods. - Deleted the synthetic dataset text processing script to clean up unused code. - Added a comprehensive test suite for all retrievers in the full benchmark pipeline to ensure reliability and performance.
…flow - Created `natural_questions.yml` for Google Natural Questions dataset with hybrid embedding strategy, chunking, validation, and evaluation settings. - Created `stackoverflow.yml` for SOSum dataset with hybrid embedding strategy, chunking, validation, and evaluation settings. - Added `stackoverflow_hybrid.yml` for hybrid dense and sparse embeddings configuration. - Introduced dataset template `dataset_template.yml` for easy dataset configuration. - Added retrieval configuration templates: `retrieval_template.yml` for agent retrieval setup. - Implemented legacy configurations for various models: `stackoverflow_bge_large.yml`, `stackoverflow_e5_large.yml`, and `stackoverflow_minilm.yml`. - Created high-performance retrieval configurations: `fast_hybrid.yml`, `modern_dense.yml`, and `modern_hybrid.yml`. - Removed outdated retriever configurations: `dense_retriever.yml`, `hybrid_retriever.yml`, `semantic_retriever.yml`, and `sparse_retriever.yml`. - Updated tests for agent retrieval and streamlined agent functionality, ensuring compatibility with new configurations.
…script - Deleted the following test files: - test_full_ingestion.py - test_modular_pipeline.py - run_all_tests.py - test_adapter_fix.py - test_agent_retrieval.py - test_retriever_direct.py - test_streamlined_agent.py - Added a new test file: test_local_setup.py - This script checks prerequisites and runs progressive tests for the pipeline.
…t.txt and updating pipeline tests to use requirements-minimal.txt
…ation structure for improved clarity and maintainability. - Deleted SYSTEM_EXTENSION_GUIDE.md, UNIFIED_CONFIG.md, agent_retrieval_upgrade_summary.md, config_reorganization_summary.md, integration_testing_setup.md, and sql_removal_summary.md. - Simplified agent graph by removing SQL-related nodes and dependencies. - Consolidated configuration files into a unified structure, enhancing usability and reducing clutter.
…s for LangChain and dotenv
…rant in requirements-minimal.txt
…rements documentation
…iever configuration
…and improve connectivity checks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces the initial foundation for a production-ready, modular Retrieval-Augmented Generation (RAG) system with LangGraph agent integration. It adds core agent graph logic, modular pipeline components, a Dockerfile for containerization, a comprehensive README, and a robust CI workflow. The main themes are agent architecture, configuration, testing, and documentation.
Agent Architecture and Core Logic:
agent/graph.py, wiring together modular nodes for query interpretation, retrieval, generation, and memory updating using LangGraph and loading configuration fromconfig.yml.agent/nodes/query_interpreter.py: Determines user intent and routing for retrieval or direct answer, with robust prompt and error handling.agent/nodes/generator.py: Generates answers using context and the configured LLM, with logging and error fallback.agent/nodes/memory_updater.py: Maintains chat history in the agent state.Configuration and Containerization:
Dockerfileusing Python 3.11-slim, installing system and Python dependencies, and copying the full source code for containerized deployment.Testing and Continuous Integration:
.github/workflows/pipeline-tests.ymlto define CI jobs for minimal, integration, end-to-end, and security/config validation tests, including Qdrant service setup and YAML config validation.Documentation and Developer Guidance:
README.mdwith detailed system overview, architecture diagrams, configuration examples, usage instructions, extension guides, and migration paths from legacy code.