Benchmark #9
Closed
Benchmark #9
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…er; add text processing pipeline for PDF documents
…; implement metadata handling and enrich documents for upload
…rocessing test script
…ctory in test script
…date requirements for PyMuPDF
…g, and memory updating; add retriever routing logic and logging
…ment generator and retriever nodes
… requirements.txt with dependency version upgrades
- Implemented a comprehensive smoke testing framework in `smoke_tests.py` to verify system quality post-ingestion. - Created `uploader.py` for handling idempotent vector uploads with versioning to Qdrant, supporting dense and sparse vectors. - Developed `validator.py` for document validation and cleaning, ensuring content quality and metadata integrity. - Added a setup script `setup_sosum.sh` for quick setup of the SOSum dataset, including verification of required files and ingestion instructions.
- Implemented a minimal ingestion test for the SOSum adapter in `examples/test_sosum_minimal.py` to validate basic functionality without heavy ML dependencies. - Created a standalone SOSum processor script in `scripts/standalone_sosum_processor.py` for processing datasets without full pipeline dependencies. - Updated `ingest.py` to improve output formatting for collection status. - Added new configuration files for different embedding strategies in `pipelines/configs/`. - Updated requirements in `requirements.txt` to include new dependencies and versions. - Added sample output JSON file for processed SOSum data.
- Added 'split' parameter to StackOverflowAdapter for better data segmentation. - Introduced 'dense_embedding' and 'sparse_embedding' fields in ChunkMeta for improved embedding metadata. - Updated EmbeddingPipeline to directly assign embeddings to respective fields. - Made allowed characters in DocumentValidator more permissive for HTML/code content.
- Introduced a comprehensive Quick Start Guide for implementing an MLOps pipeline for RAG systems, covering project initialization, dataset adapters, configuration, processing components, and testing. - Implemented a CSV dataset adapter for reading and converting CSV data into documents. - Created a configuration schema for managing dataset, chunking, embedding, and vector store settings. - Developed core processing components including a document chunker and an embedding pipeline. - Added a simple CLI interface for ingestion with logging and configuration handling. - Implemented a sparse embedding mechanism and integrated it into the embedding pipeline. - Added inspection script for analyzing vector structures in Qdrant. - Created smoke tests for validating ingestion processes and vector store uploads. - Added test script for verifying sparse embedding serialization. - Updated existing configurations for stackoverflow datasets to support hybrid and dense embedding strategies.
… framework - Created `experimental.yml` for testing new components in the retrieval pipeline. - Added `hybrid_multistage.yml` for hybrid retrieval with multi-stage reranking. - Implemented tests for the new answer-focused adapter in `test_new_adapter.py`. - Developed advanced reranking tests in `test_advanced_rerankers.py`. - Introduced answer retrieval tests in `test_answer_retrieval.py`. - Demonstrated retrieval pipeline extensibility in `test_extensibility.py`. - Showcased modular pipeline features in `test_modular_pipeline.py`. - Added a comprehensive test runner in `run_all_tests.py`. - Updated agent retrieval tests to support configurable pipelines in `test_agent_retrieval.py`.
- Implemented unit tests for the RetrievalPipeline, RetrievalResult, and associated components (Retriever, Reranker, Filter) in `test_retrieval_pipeline.py`. - Created mock classes for testing purposes to simulate retrieval, reranking, and filtering behaviors. - Added tests for basic functionality, component addition/removal, and pipeline execution with various configurations. - Introduced tests for the RetrievalPipelineFactory to validate pipeline creation with dense and hybrid configurations. - Added minimal and example tests for the SOSum adapter to ensure basic functionality without heavy dependencies. - Implemented smoke tests for the ingestion process and overall system quality checks. - Updated the test runner to include new tests and organized the test structure for better clarity.
- Updated langchain-core to version 0.3.75 - Added new dependencies: cachetools, distro, filetype, google-ai-generativelanguage, google-api-core, google-auth, googleapis-common-protos, grpcio-status, jiter, langchain-google-genai, langchain-openai, langgraph, langgraph-checkpoint, langgraph-prebuilt, langgraph-sdk, openai, ormsgpack, proto-plus, psycopg2-binary, pyasn1, pyasn1_modules, rsa, tiktoken, xxhash - Updated existing dependencies to their latest versions test: Enhance tests for rerankers and retrieval pipeline - Refactored test cases in test_rerankers.py for better readability and maintainability - Added new tests for the RetrievalPipeline and its components in test_retrieval_pipeline.py - Improved mock implementations for better isolation in tests feat: Add debug scripts for StackOverflow adapter - Introduced debug_row_order.py to check row order and types from StackOverflow data - Added debug_stackoverflow_adapter.py to investigate issues with document reading in the StackOverflow adapter test: Implement tests for ingestion pipeline and adapter functionality - Created test_full_ingestion.py to validate the full ingestion pipeline with StackOverflow data - Added test_adapter_fix.py to verify the StackOverflow adapter produces documents correctly chore: Update test runner to include new tests - Modified run_all_tests.py to include new test files for retrieval and ingestion
- Removed legacy retriever wrapping and introduced modern retriever classes. - Updated `RetrievalPipelineFactory` to create dense, hybrid, sparse, and semantic pipelines using new retriever implementations. - Created `ModernBaseRetriever` as a base class for all retrievers, providing common functionality and configuration handling. - Implemented `QdrantDenseRetriever`, `QdrantHybridRetriever`, `QdrantSparseRetriever`, and `SemanticRetriever` with improved initialization and search methods. - Removed deprecated `router.py` and integrated routing logic into the new retriever classes. - Enhanced logging and error handling across retrievers for better debugging and monitoring. - Updated imports and module structure to reflect the new architecture.
- Enhanced the `load_config` function to include detailed error handling and logging. - Introduced `get_retriever_config`, `get_benchmark_config`, and `get_pipeline_config` functions for better configuration management. - Added `load_config_with_overrides` to support configuration overrides. - Updated `QdrantVectorDB` initialization to accept configuration parameters. - Created example scripts for unified configuration usage and retriever configuration examples. - Removed outdated retrieval pipeline configurations to streamline the codebase. - Improved logging throughout the retriever classes for better traceability.
…ality - Refactored benchmark runner to utilize unified configuration approach. - Updated imports to reflect new module structure for benchmarks and metrics. - Introduced new benchmark scripts for simple and full dataset evaluations. - Enhanced retrieval pipeline initialization to support unified config. - Created comprehensive dataset adapter for full StackOverflow dataset evaluation. - Added real data benchmark runner for testing with actual StackOverflow queries. - Updated configuration file to include new retrieval strategies and parameters. - Documented changes in CONFIG_CONSOLIDATION_COMPLETE.md and UNIFIED_CONFIG.md. - Added examples demonstrating the use of the new unified configuration system.
…issing ground truth and improving document ID extraction
…and configuration - Removed the full dataset benchmark script to streamline the benchmarking process. - Updated the real benchmark script to ensure proper imports and functionality. - Enhanced the configuration file to include new fusion methods and adjustable weights for hybrid retrieval. - Refactored dense and sparse retrievers to improve embedding initialization and search processes. - Implemented a new hybrid retriever that combines dense and sparse results using configurable fusion methods. - Deleted the synthetic dataset text processing script to clean up unused code. - Added a comprehensive test suite for all retrievers in the full benchmark pipeline to ensure reliability and performance.
…flow - Created `natural_questions.yml` for Google Natural Questions dataset with hybrid embedding strategy, chunking, validation, and evaluation settings. - Created `stackoverflow.yml` for SOSum dataset with hybrid embedding strategy, chunking, validation, and evaluation settings. - Added `stackoverflow_hybrid.yml` for hybrid dense and sparse embeddings configuration. - Introduced dataset template `dataset_template.yml` for easy dataset configuration. - Added retrieval configuration templates: `retrieval_template.yml` for agent retrieval setup. - Implemented legacy configurations for various models: `stackoverflow_bge_large.yml`, `stackoverflow_e5_large.yml`, and `stackoverflow_minilm.yml`. - Created high-performance retrieval configurations: `fast_hybrid.yml`, `modern_dense.yml`, and `modern_hybrid.yml`. - Removed outdated retriever configurations: `dense_retriever.yml`, `hybrid_retriever.yml`, `semantic_retriever.yml`, and `sparse_retriever.yml`. - Updated tests for agent retrieval and streamlined agent functionality, ensuring compatibility with new configurations.
…script - Deleted the following test files: - test_full_ingestion.py - test_modular_pipeline.py - run_all_tests.py - test_adapter_fix.py - test_agent_retrieval.py - test_retriever_direct.py - test_streamlined_agent.py - Added a new test file: test_local_setup.py - This script checks prerequisites and runs progressive tests for the pipeline.
…t.txt and updating pipeline tests to use requirements-minimal.txt
…ation structure for improved clarity and maintainability. - Deleted SYSTEM_EXTENSION_GUIDE.md, UNIFIED_CONFIG.md, agent_retrieval_upgrade_summary.md, config_reorganization_summary.md, integration_testing_setup.md, and sql_removal_summary.md. - Simplified agent graph by removing SQL-related nodes and dependencies. - Consolidated configuration files into a unified structure, enhancing usability and reducing clutter.
…s for LangChain and dotenv
…rant in requirements-minimal.txt
…rements documentation
…iever configuration
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces a modular, production-ready Retrieval-Augmented Generation (RAG) system with agent-based orchestration, comprehensive documentation, and CI/CD pipeline automation. The main changes include the addition of a YAML-configurable agent pipeline built on LangGraph, a detailed project README, a Dockerfile for deployment, and a robust GitHub Actions workflow for testing and validation.
Agent Pipeline Implementation
agent/graph.pyto define a modular LangGraph pipeline, integrating query interpretation, retrieval, generation, and memory update nodes, all configurable via YAML.query_interpreterfor routing/planning [1]generatorfor answer synthesis [2] andmemory_updaterfor chat history management [3].Documentation and Project Structure
README.mdoutlining features, architecture, configuration, extensibility, and usage patterns for the RAG system.Deployment and CI/CD
Dockerfileusing a slim Python 3.11 base image for efficient containerization and deployment..github/workflows/pipeline-tests.ymlfor automated testing, integration, security checks, and configuration validation in CI.