Skip to content

Refactor/src-validator #60

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
muthukumaranR wants to merge 7 commits into
base: develop
Choose a base branch
from
Draft

Refactor/src-validator #60

muthukumaranR wants to merge 7 commits into from

Conversation

@muthukumaranR
Copy link
Collaborator

@muthukumaranR muthukumaranR commented Jul 2, 2025 •
edited
Loading

Description

Title

Fix Source Validator Crossref API, Thread Safety and Performance Issues

Summary

  • Improve CrossRef API calling
  • Replace asyncio.to_thread() with ThreadPoolExecutor for CrossRef API calls
  • Add 30-second timeout to prevent hanging requests
  • Improve error handling and logging

Validation Improvements:

  • Add ISSN checksum validation (was format-only)
  • Enhance DOI format validation with stricter checks
  • Use weighted scoring instead of simple max() for confidence scores

Performance:

  • Add class-level caching for whitelist data
  • Consolidate overlapping DOI regex patterns (9 → 6 patterns)
  • Add cache management methods

@muthukumaranR
Update dependencies in pyproject.toml to include rapidfuzz, crossref-…
090ef72 
…commons, unidecode, httpx, requests-cache, orjson, and ftfy
@muthukumaranR
setup.py file to define the package metadata and dependencies
ed5f996
@muthukumaranR
Refactor scraper import and add source validator creation function
9205016 
- Updated import from `CompositeWebScraper` to `CompositeScraper` for consistency.
- Introduced `create_default_source_validator` function to facilitate the creation of a source validator with default parameters, enhancing the validation process.
@muthukumaranR
Enhance source validation logic and improve error handling
d2259b0 
- Refactored `source_validator.py` to include comprehensive DOI and ISSN validation with detailed error handling.
- Implemented a journal index for efficient fuzzy matching of source titles against a whitelist.
- Updated the `arun` method to handle validation results and exceptions more gracefully.
- Modified example scripts and tests to reflect changes from journal to source validation terminology.
- Improved logging for better debugging and tracking of validation processes.
@muthukumaranR
add init
eeae129
@muthukumaranR
Add example script for source validation pipeline
a62ff86 
- Introduced `source_validation_pipeline_test.py` to demonstrate the complete source validation process.
- Implemented functionality to search for research papers, extract DOIs, validate sources against a whitelist, and generate validation reports.
- Included sample queries for Earth Sciences and Astronomy to showcase the pipeline's capabilities.
- Added comprehensive inline comments and docstrings for clarity and maintainability.
@muthukumaranR
Add simple source validation example script
1ed64d3 
- Introduced `simple_source_validation_example.py` to demonstrate the core functionality of the source validation process without complex dependencies.
- Implemented a test case with real DOIs to validate against a whitelist and included a paper without a DOI to showcase validation failure.
- Added comprehensive inline comments and docstrings for clarity and maintainability.
- Utilized asynchronous programming to handle validation and output results effectively.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@muthukumaranR