Skip to content

feat: Adding mock responses for remote API requests in CI - #51

Merged
wumpus merged 25 commits into
commoncrawl:mainfrom
malteos:feat/improved-ci
Sep 21, 2025
Merged

wumpus merged 25 commits into
commoncrawl:mainfrom
malteos:feat/improved-ci

Conversation

@malteos

@malteos malteos commented Sep 10, 2025 •

Copy link
Copy Markdown
Collaborator

This PR introduces various changes to improve the CI pipeline. Most importantly, it introduces mock response so no remote server is called by the CI. The previous remote request made the CI unstable due to rate limits (runner IP being blocked by remote servers).

Overview of changes:

  • Mock response for remote requests (to a avoid time outs etc.)
  • Slow test cases are skip with pytest mark
  • Multi-testcases are refactored as single text methods (easier to spot failing test cases and to auto-load the mock responses)
  • Fixed wrong "False" in slow_ci() function.
  • DNS error handling for windows + macos
  • Additional unit tests for better test coverage

By default, all remote requests are mocked. To change this behaviour and actually call remote APIs (if you run this from a whitelisted IP address), the following environment variable can be set:

export DISABLE_MOCK_RESPONSES=1

If the remote APIs change, new mock data can be semi-automatically collected by setting another environment variable, running corresponding unit tests, and overwriting existing mock data in tests/data/mock_responses:

# set environment variable (DISABLE_MOCK_RESPONSES should not be set)
export SAVE_MOCK_RESPONSES=./tmp/mock_responses
    
# run the test for what mock data should be saved to $SAVE_MOCK_RESPONSES/<test_file>/<test_func>.jsonl
pytest tests/test_cli.py::test_basics

- Adding flake8 ignore for missing doc strings
- Multi-testcases are refactored to use pytest-parameterization (easier to spot failing test cases)
- Slow test cases are skip with pytest mark
- DNS error handling for windows + macos
- Temp. disabled test matrices
@wumpus

wumpus commented Sep 11, 2025

Copy link
Copy Markdown
Member

I need an IP to look in some logs. Both of these failed because the "block_proxies" script hates this particular Github CI IP.

@codecov

codecov Bot commented Sep 11, 2025 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.89%. Comparing base (e5d122a) to head (88a7ed7).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #51      +/-   ##
==========================================
+ Coverage   95.66%   95.89%   +0.22%     
==========================================
  Files           7        7              
  Lines         877      876       -1     
==========================================
+ Hits          839      840       +1     
+ Misses         38       36       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@malteos malteos changed the title Trying to improve the CI feat: Adding mock responses for remote API requests in CI Sep 15, 2025
@malteos
malteos marked this pull request as ready for review September 15, 2025 22:27
@wumpus
wumpus merged commit 3086620 into commoncrawl:main Sep 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants