Fix: ArXiv API Migration - OAI-PMH Implementation

[Opsmithe]

as per Issue #236 by @TimidRobot

Problem

The current arxiv_fetch.py script relies on ArXiv's Atom API which provides unreliable license information through two problematic fields:

1. <rights> field: Does not exist in the API response schema according to ArXiv API documentation
2. <summary> field: Uses text pattern matching which incorrectly identifies papers that discuss CC licenses rather than papers that are actually CC-licensed

Impact

• False positives: Papers discussing CC-licensed works are incorrectly classified as CC-licensed
• Data integrity: Unreliable license detection compromises the accuracy of commons quantification
• Reproducibility: Inconsistent results across different API responses

Example Case

Query [ALL:]("CC BY") returns paper [2008.00774v3] "Elsevier OA CC-By Corpus" which discusses CC BY works but is actually licensed under "arXiv.org - Non-exclusive license to distribute", not CC BY.

Proposed Solution

Migrate arxiv_fetch.py to use ArXiv's OAI-PMH API (https://oaipmh.arxiv.org/oai) which provides:

• Structured metadata: XML-based responses with dedicated license fields
• Reliable extraction: Direct access to <license> elements in arXiv namespace
• Better coverage: Access to historical papers with proper license metadata
• Standards compliance: OAI-PMH is a standardized protocol for metadata harvesting

Query Strategy Implemented

API Endpoint Migration

+ BASE_URL = "https://oaipmh.arxiv.org/oai"

License Extraction Method

+ # Structured XML parsing from OAI-PMH
+ def extract_license_from_xml(record_xml):
+     root = ET.fromstring(record_xml)
+     license_elem = root.find(".//{http://arxiv.org/OAI/arXiv/}license")
+     return LICENSE_MAPPING.get(license_elem.text, "Unknown")

Request Parameters

+ # OAI-PMH harvesting request  
+ params = {
+     'verb': 'ListRecords',
+     'metadataPrefix': 'arXiv',
+     'from': from_date,
+     'resumptionToken': token  # For pagination
+ }

Implementation Details

New Features

• Date-based harvesting: Focus on recent years where CC licensing is more prevalent
• Resumption token support: Handle large datasets with proper pagination
• Enhanced error handling: Robust XML parsing with namespace awareness
• License URL mapping: Standardized license identification from URLs
• Provenance tracking: Detailed metadata for audit trails

Data Structure Changes

• XML namespace handling: Proper parsing of arXiv-specific metadata elements
• License normalization: Map license URLs to standardized identifiers
• Category extraction: Use structured category fields instead of text parsing
• Author counting: Extract from structured author elements

Performance Improvements

• Reduced API calls: OAI-PMH provides batch processing capabilities
• Rate limiting compliance: Built-in delays following OAI-PMH best practices
• Retry logic: Enhanced error recovery for network issues

API Requirements

• ArXiv OAI-PMH endpoint: https://oaipmh.arxiv.org/oai
• No authentication required
• Rate limiting: 3-second delays between requests (OAI-PMH recommendation)

Checklist

• I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
• My pull request doesn't include code or content generated with AI.
• My pull request has a descriptive title (not a vague title like Update index.md).
• My pull request targets the default branch of the repository (main or master).
• My commit messages follow best practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no
  visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

[Opsmithe]

@TimidRobot I hope this fixes help address the issues with #236, ild be happy to get your reviews on the script and make changes where necessary

[TimidRobot]

Please add:

    "http://creativecommons.org/licenses/publicdomain": "CC CERTIFICATION 1.0 US",

[TimidRobot]

Please sort/order these entries

[TimidRobot]

Iftotal_fetched includes "Non-CC" papers, this message is misleading

[Opsmithe]

@TimidRobot. Thanks for this. I investigated and fixed the license filtering logic in arxiv_fetch.py that incorrectly included non-Creative Commons licenses due to substring matching. The condition "CC" in metadata["license"] would match false positives like "Non-CC" or "Apache with CC mention". Changed to metadata["license"].startswith("CC") to ensure only licenses beginning with "CC" are processed, preventing
contamination of CC license statistics with misclassified non-CC licenses.