Add test and benchmarks Timeout Guards [MOD-7846] #800

dor-forer · 2025-10-15T06:41:34Z

Describe the changes in the pull request

Added new TimeoutGuard class that receives a timeout on creation, and starts a thread that waits on a condition variable until the test is finished (changes the flag on destruction) or the timeout is reached.
Created a new main function that uses Google tests' TestEventListeners mechanism to start TimeoutTestListener (that uses TimeoutGuard), for every test.
Creates SetUp and TearDown functions (Which are inherited from benchmark::Fixture) in BM_VecSimGeneral to create and destroy the timeout when a benchmark is finished.
Disable the TimeoutGuard for EXPECT_EXIT test because they don't work well with the background threading, and added the TimeoutGuard manually to the test.

Which issues this PR fixes

MOD-7846

Main objects this PR modified

unit tests
benchmarks

Mark if applicable

This PR introduces API changes
This PR introduces serialization changes

jit-ci · 2025-10-15T06:41:43Z

Hi, I’m Jit, a friendly security platform designed to help developers build secure applications from day zero with an MVS (Minimal viable security) mindset.

In case there are security findings, they will be communicated to you as a comment inside the PR.

Hope you’ll enjoy using Jit.

Questions? Comments? Want to learn more? Get in touch with us.

Copilot

Pull Request Overview

This PR introduces timeout protection mechanisms for both unit tests and benchmarks to prevent indefinite hangs and improve CI reliability. The implementation uses RAII-style timeout guards that monitor test/benchmark execution and forcefully terminate if timeouts are reached.

Key changes:

Added TimeoutGuard class for general timeout protection with configurable actions
Created TimeoutTestListener for Google Test integration with automatic per-test timeout
Integrated timeout guards into benchmark fixtures using SetUp/TearDown methods

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/utils/timeout_guard.h	Core timeout implementation with RAII guards and benchmark wrapper
tests/utils/timeout_test_environment.h	Google Test listener for automatic test timeout registration
tests/utils/test_main_with_timeout.cpp	Custom main function registering global timeout protection
tests/unit/test_common.cpp	Removed manual timeout code, added timeout_guard.h include
tests/unit/CMakeLists.txt	Updated test executables to use custom main and link against gtest
tests/benchmark/bm_vecsim_general.h	Added timeout guard to benchmark fixture with SetUp/TearDown

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

tests/utils/timeout_guard.h

tests/unit/CMakeLists.txt

tests/unit/test_common.cpp

…ailure if any test failed again

codecov · 2025-10-15T09:10:14Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.45%. Comparing base (00bf35d) to head (cf512e7).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #800      +/-   ##
==========================================
- Coverage   96.86%   96.45%   -0.41%     
==========================================
  Files         126      126              
  Lines        7739     7739              
==========================================
- Hits         7496     7465      -31     
- Misses        243      274      +31

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull Request Overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

tests/unit/test_common.cpp

tests/utils/timeout_test_environment.h

…to dorer-test-timeout-guard_MOD-7846

Copilot

Pull Request Overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

tests/utils/timeout_guard.h

Copilot

Pull Request Overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-15T12:36:43Z

tests/utils/test_main_with_timeout.cpp

+ * @brief Custom main function that registers global timeout protection for all tests.
+ *
+ * This replaces gtest_main and adds automatic timeout protection to every test.
+ * Tests will automatically timeout after 30 seconds (or customized duration based on test type).


Documentation inaccuracy: The comment states tests timeout after 30 seconds, but the code sets a 100-second timeout on line 23. Update the comment to reflect the actual timeout value.

Suggested change

* Tests will automatically timeout after 30 seconds (or customized duration based on test type).

* Tests will automatically timeout after 100 seconds (or customized duration based on test type).

alonre24

Very nice job!

alonre24 · 2025-10-21T16:03:22Z

tests/utils/timeout_test_environment.h

+        // Get timeout for this specific test (can be customized per test)
+        auto timeout = GetTimeoutForTest(test_info);


Can we add one test for testing this mechanism as well, using a lower timeout?

dor-forer added 4 commits October 12, 2025 15:58

Changes

03a48ea

benchmark

40d1f96

change the documention

c38c41f

Remove md files and _

5d5a1df

dor-forer requested a review from Copilot October 15, 2025 06:42

Copilot AI reviewed Oct 15, 2025

View reviewed changes

tests/utils/timeout_guard.h Show resolved Hide resolved

tests/unit/CMakeLists.txt Show resolved Hide resolved

tests/unit/test_common.cpp Show resolved Hide resolved

dor-forer added 3 commits October 15, 2025 09:49

Added std::exit(testing::Test::HasFailure() ? -1 : 0); // Exit with f…

68061da

…ailure if any test failed again

add timeout guard manually to death tests

c72467a

format

c791530

dor-forer requested a review from Copilot October 15, 2025 11:17

Copilot AI reviewed Oct 15, 2025

View reviewed changes

tests/unit/test_common.cpp Show resolved Hide resolved

tests/utils/timeout_test_environment.h Outdated Show resolved Hide resolved

tests/utils/timeout_test_environment.h Show resolved Hide resolved

Merge branch 'main' of https://github.com/RedisAI/VectorSimilarity in…

decca95

…to dorer-test-timeout-guard_MOD-7846

dor-forer requested a review from Copilot October 15, 2025 11:29

Copilot AI reviewed Oct 15, 2025

View reviewed changes

tests/utils/timeout_guard.h Outdated Show resolved Hide resolved

remove the if notify without lock

1c4eb2c

dor-forer requested a review from Copilot October 15, 2025 12:35

Copilot AI reviewed Oct 15, 2025

View reviewed changes

Change documention

cf512e7

dor-forer requested review from alonre24 and meiravgri and removed request for meiravgri October 16, 2025 07:26

alonre24 reviewed Oct 21, 2025

View reviewed changes

	* Tests will automatically timeout after 30 seconds (or customized duration based on test type).
	* Tests will automatically timeout after 100 seconds (or customized duration based on test type).

		// Get timeout for this specific test (can be customized per test)
		auto timeout = GetTimeoutForTest(test_info);

Add test and benchmarks Timeout Guards [MOD-7846] #800

Are you sure you want to change the base?

Add test and benchmarks Timeout Guards [MOD-7846] #800

Uh oh!

Conversation

dor-forer commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jit-ci bot commented Oct 15, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Key changes:

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

alonre24 left a comment

Choose a reason for hiding this comment

Uh oh!

alonre24 Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dor-forer commented Oct 15, 2025 •

edited

Loading

codecov bot commented Oct 15, 2025 •

edited

Loading