Add a new --fail-on-increase option

As requested from some users, add a new --fail-on-increase option that
makes the test run fail if the memory usage increases from previous runs
in tests marked with "limit_memory".

Signed-off-by: Pablo Galindo <[email protected]>

README.md

@@ -97,6 +97,7 @@ MEMORY PROBLEMS demo/test_ok.py::test_memory_exceed
 - `--stacks=STACKS` - Show the N stack entries when showing tracebacks of memory allocations
 - `--native` - Show native frames when showing tracebacks of memory allocations (will be slower)
 - `--trace-python-allocators` - Record allocations made by the Pymalloc allocator (will be slower)
+- `fail-on-increase` - Fail if the memory usage increases from previous runs in tests marked with "limit_memory"
 
 ## Configuration - INI
 
@@ -105,7 +106,8 @@ MEMORY PROBLEMS demo/test_ok.py::test_memory_exceed
 - `hide_memray_summary(bool)` - hide the memray summary at the end of the execution
 - `stacks(int)` - Show the N stack entries when showing tracebacks of memory allocations
 - `native(bool)`- Show native frames when showing tracebacks of memory allocations (will be slower)
-- `trace_python_allocators` - Record allocations made by the Pymalloc allocator (will be slower)
+- `trace_python_allocators(bool)` - Record allocations made by the Pymalloc allocator (will be slower)
+- `fail-on-increase(bool)` - Fail if the memory usage increases from previous runs in tests marked with "limit_memory"
 
 ## License
 

docs/configuration.rst

@@ -29,6 +29,9 @@ The complete list of command line options is:
 
   ``--trace-python-allocators``
     Record allocations made by the Pymalloc allocator (will be slower)
+
+  ``--fail-on-increase``
+    Fail if the memory usage increases from previous runs in tests marked with "limit_memory"
 
 .. tab:: Config file options
 
@@ -49,3 +52,6 @@ The complete list of command line options is:
 
   ``trace_python_allocators(bool)``
     Record allocations made by the Pymalloc allocator (will be slower)
+
+  ``--fail-on-increase(bool)``
+    Fail if the memory usage increases from previous runs in tests marked with "limit_memory"

docs/news/91.feature.rst

@@ -0,0 +1 @@
+Add a new --fail-on-increase option that makes the test run fail if the memory usage increases from previous runs in tests marked with "limit_memory"

tests/test_pytest_memray.py

@@ -742,3 +742,68 @@ def test_bar():
 
     output = result.stdout.str()
     assert "Only one Memray marker can be applied to each test" in output
+
+
+def test_fail_on_increase(pytester: Pytester):
+    pytester.makepyfile(
+        """
+        import pytest
+        from memray._test import MemoryAllocator
+        allocator = MemoryAllocator()
+
+        @pytest.mark.limit_memory("100MB")
+        def test_memory_alloc_fails():
+            allocator.valloc(1024)
+            allocator.free()
+        """
+    )
+    result = pytester.runpytest("--memray")
+    assert result.ret == ExitCode.OK
+    pytester.makepyfile(
+        """
+        import pytest
+        from memray._test import MemoryAllocator
+        allocator = MemoryAllocator()
+
+        @pytest.mark.limit_memory("100MB")
+        def test_memory_alloc_fails():
+            allocator.valloc(1024 * 10)
+            allocator.free()
+        """
+    )
+    result = pytester.runpytest("--memray", "--fail-on-increase")
+    assert result.ret == ExitCode.TESTS_FAILED
+    output = result.stdout.str()
+    assert "Test uses more memory than previous run" in output
+    assert "Test previously used 1.0KiB but now uses 10.0KiB" in output
+
+
+def test_fail_on_increase_unset(pytester: Pytester):
+    pytester.makepyfile(
+        """
+        import pytest
+        from memray._test import MemoryAllocator
+        allocator = MemoryAllocator()
+
+        @pytest.mark.limit_memory("100MB")
+        def test_memory_alloc_fails():
+            allocator.valloc(1024)
+            allocator.free()
+        """
+    )
+    result = pytester.runpytest("--memray")
+    assert result.ret == ExitCode.OK
+    pytester.makepyfile(
+        """
+        import pytest
+        from memray._test import MemoryAllocator
+        allocator = MemoryAllocator()
+
+        @pytest.mark.limit_memory("100MB")
+        def test_memory_alloc_fails():
+            allocator.valloc(1024 * 10)
+            allocator.free()
+        """
+    )
+    result = pytester.runpytest("--memray")
+    assert result.ret == ExitCode.OK