Extend unit tests #1967

qododavid · 2024-11-26T16:28:48Z

Hi, I used cover-agent to write the tests for this PR. It uses an AI-based workflow to add new test cases that increase code coverage.

All AI-generated tests have met these four essential criteria:

Execute successfully
Pass all assertions
Increase overall code coverage
Test previously uncovered behaviors (as specified in the LLM prompt)

Increased Coverage

Coverage Improvement Report
=========================

File: onnxscript/evaluator.py
Coverage improvement: 73.26% → 80.62% (+7.36%)
Newly covered lines: 19
Remaining uncovered lines: 50


File: onnxscript/onnx_opset/_impl/opset15.py
Coverage improvement: 60.66% → 85.25% (+24.59%)
Newly covered lines: 15
Remaining uncovered lines: 9


File: onnxscript/onnx_types.py
Coverage improvement: 86.67% → 95.00% (+8.33%)
Newly covered lines: 10
Remaining uncovered lines: 6


File: onnxscript/ir/serde.py
Coverage improvement: 54.29% → 57.53% (+3.24%)
Newly covered lines: 25
Remaining uncovered lines: 327


File: onnxscript/_internal/version_utils.py
Coverage improvement: 32.00% → 94.00% (+62.00%)
Newly covered lines: 31
Remaining uncovered lines: 3


File: opgen/pygen.py
Coverage improvement: 60.70% → 64.91% (+4.21%)
Newly covered lines: 43
Remaining uncovered lines: 359


File: onnxscript/values.py
Coverage improvement: 68.79% → 71.02% (+2.23%)
Newly covered lines: 7
Remaining uncovered lines: 91


File: onnxscript/tensor.py
Coverage improvement: 77.78% → 98.52% (+20.74%)
Newly covered lines: 28
Remaining uncovered lines: 2


File: onnxscript/type_annotation.py
Coverage improvement: 63.27% → 82.99% (+19.72%)
Newly covered lines: 29
Remaining uncovered lines: 25


File: onnxscript/_internal/param_manipulation.py
Coverage improvement: 48.33% → 100.00% (+51.67%)
Newly covered lines: 31
Remaining uncovered lines: 0


File: onnxscript/rewriter/broadcast_to_matmul.py
Coverage improvement: 90.70% → 100.00% (+9.30%)
Newly covered lines: 8
Remaining uncovered lines: 0


File: onnxscript/rewriter/llama_rule_sets.py
Coverage improvement: 78.33% → 78.89% (+0.56%)
Newly covered lines: 1
Remaining uncovered lines: 38


File: onnxscript/rewriter/collapse_slices.py
Coverage improvement: 63.38% → 97.18% (+33.80%)
Newly covered lines: 24
Remaining uncovered lines: 2


File: onnxscript/rewriter/generic_pattern.py
Coverage improvement: 77.70% → 79.09% (+1.39%)
Newly covered lines: 4
Remaining uncovered lines: 60


File: onnxscript/rewriter/onnxruntime/fused_matmul_rule_sets.py
Coverage improvement: 96.33% → 98.17% (+1.84%)
Newly covered lines: 2
Remaining uncovered lines: 2


File: onnxscript/rewriter/onnxruntime/softmax.py
Coverage improvement: 92.00% → 100.00% (+8.00%)
Newly covered lines: 2
Remaining uncovered lines: 0


File: onnxscript/rewriter/onnxruntime/bfloat16_utils/bfloat16_converter.py
Coverage improvement: 91.11% → 100.00% (+8.89%)
Newly covered lines: 4
Remaining uncovered lines: 0


File: tests/common/testutils.py
Coverage improvement: 78.69% → 98.36% (+19.67%)
Newly covered lines: 12
Remaining uncovered lines: 1


File: onnxscript/version_converter/_version_converter.py
Coverage improvement: 92.02% → 93.09% (+1.07%)
Newly covered lines: 2
Remaining uncovered lines: 13


File: onnxscript/tools/memory_peak.py
Coverage improvement: 79.69% → 85.94% (+6.25%)
Newly covered lines: 8
Remaining uncovered lines: 18


File: onnxscript/tools/benchmark/benchmark_helpers.py
Coverage improvement: 13.12% → 49.50% (+36.38%)
Newly covered lines: 147
Remaining uncovered lines: 204


File: onnxscript/tools/benchmark/export_model.py
Coverage improvement: 52.70% → 54.05% (+1.35%)
Newly covered lines: 1
Remaining uncovered lines: 34


File: onnxscript/tools/transformers_models/mistral.py
Coverage improvement: 60.00% → 97.14% (+37.14%)
Newly covered lines: 26
Remaining uncovered lines: 2


File: onnxscript/tools/transformers_models/phi.py
Coverage improvement: 57.75% → 94.37% (+36.62%)
Newly covered lines: 26
Remaining uncovered lines: 4


File: onnxscript/tools/transformers_models/phi3.py
Coverage improvement: 58.44% → 93.51% (+35.07%)
Newly covered lines: 27
Remaining uncovered lines: 5


File: onnxscript/optimizer/_inliner.py
Coverage improvement: 79.11% → 81.01% (+1.90%)
Newly covered lines: 3
Remaining uncovered lines: 30


File: onnxscript/optimizer/_legacy/_simple_function_folding.py
Coverage improvement: 96.69% → 97.35% (+0.66%)
Newly covered lines: 1
Remaining uncovered lines: 4


File: onnxscript/_legacy_ir/visitor.py
Coverage improvement: 60.10% → 68.49% (+8.39%)
Newly covered lines: 49
Remaining uncovered lines: 184


File: onnxscript/ir/_core.py
Coverage improvement: 41.19% → 52.61% (+11.42%)
Newly covered lines: 140
Remaining uncovered lines: 581


File: onnxscript/ir/_enums.py
Coverage improvement: 94.20% → 100.00% (+5.80%)
Newly covered lines: 4
Remaining uncovered lines: 0


File: onnxscript/ir/_external_data.py
Coverage improvement: 95.65% → 99.13% (+3.48%)
Newly covered lines: 4
Remaining uncovered lines: 1


File: onnxscript/ir/_type_casting.py
Coverage improvement: 55.56% → 88.89% (+33.33%)
Newly covered lines: 12
Remaining uncovered lines: 4


File: onnxscript/ir/_schemas.py
Coverage improvement: 51.48% → 59.92% (+8.44%)
Newly covered lines: 20
Remaining uncovered lines: 95


File: onnxscript/ir/traversal.py
Coverage improvement: 86.11% → 97.22% (+11.11%)
Newly covered lines: 4
Remaining uncovered lines: 1


File: onnxscript/ir/_linked_list.py
Coverage improvement: 92.59% → 100.00% (+7.41%)
Newly covered lines: 8
Remaining uncovered lines: 0


File: onnxscript/ir/tensor_adapters.py
Coverage improvement: 90.91% → 93.94% (+3.03%)
Newly covered lines: 1
Remaining uncovered lines: 2


File: onnxscript/backend/onnx_backend.py
Coverage improvement: 60.49% → 64.81% (+4.32%)
Newly covered lines: 7
Remaining uncovered lines: 57

Why adding unit tests is important

Writing and maintaining unit tests can be tedious, especially as codebases grow larger
Despite this challenge, unit tests provide key benefits:
- Enable confident code changes through reliable test coverage
- Allow faster development by preventing regressions
- Make refactoring safer with comprehensive test validation
Tests serve as executable documentation by:
- Clearly demonstrating input/output relationships
- Providing quick feedback during development
- Presenting behavior changes in readable format

justinchuby · 2024-11-26T22:42:08Z

This is pretty cool. If you can isolate changes to onnxscript/ir in a separate PR that’d be great

onnxscript/function_libs/tools/torch_lib/deduce_type_constraints_test.py

+        type_constraint = deduce_type_constraints.TypeConstraint("test_constraint", {"tensor(float)", "tensor(int64)"})
+        type_constraint.bind_value(value1)
+        type_constraint.bind_value(value2)
+        expected_repr = "TypeConstraint(name=test_constraint, type_strs={'tensor(float)', 'tensor(int64)'}, values=['value1', 'value2'])"


onnxscript/ir/_display_test.py

+            graph.register_initializer(unnamed_initializer)
+
+        # Attempt to register an initializer that is produced by a node
+        node = ir.Node(domain="", op_type="Add", inputs=[], outputs=[initializer])


onnxscript/ir/_schemas_test.py

+        type_constraints = {}
+        parameter = _schemas._convert_formal_parameter(mock_param, type_constraints)
+        self.assertEqual(parameter.name, "param")
+        self.assertTrue(ir.TensorType(ir.DataType.FLOAT) in parameter.type_constraint.allowed_types)


onnxscript/rewriter/generic_pattern_test.py

+        def match_pattern(op, x):
+            return op.Add(x, x)
+
+        def apply_pattern(op, x, **_):


onnxscript/tools/benchmark/benchmark_helpers_test.py

@@ -4,6 +4,11 @@

 import onnxscript.tools.benchmark.benchmark_helpers as bh

+import torch


onnxscript/tools/benchmark/benchmark_helpers_test.py

@@ -4,6 +4,11 @@

 import onnxscript.tools.benchmark.benchmark_helpers as bh

+import torch
+import onnx


tests/external_tensor_test.py

+    def test_class_getitem_shape_already_specified(self):
+        from onnxscript.onnx_types import FLOAT
+        with self.assertRaises(ValueError):
+            FLOAT[None][None]


codecov · 2024-11-26T22:44:53Z

Codecov Report

Attention: Patch coverage is 98.09069% with 24 lines in your changes missing coverage. Please review.

Project coverage is 77.04%. Comparing base (99cf79f) to head (9a5f69a).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
onnxscript/tools/transformers_models/phi3_test.py	77.08%	10 Missing and 1 partial ⚠️
onnxscript/tools/transformers_models/llama_test.py	82.69%	9 Missing ⚠️
onnxscript/rewriter/generic_pattern_test.py	92.30%	1 Missing ⚠️
...rewriter/onnxruntime/transformers/fastgelu_test.py	93.75%	1 Missing ⚠️
...ewriter/onnxruntime/transformers/layernorm_test.py	83.33%	1 Missing ⚠️
onnxscript/values_test.py	93.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1967      +/-   ##
==========================================
+ Coverage   75.13%   77.04%   +1.90%     
==========================================
  Files         265      265              
  Lines       28907    30164    +1257     
  Branches     3331     3336       +5     
==========================================
+ Hits        21720    23239    +1519     
+ Misses       6158     5903     -255     
+ Partials     1029     1022       -7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-advanced-security

lintrunner found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

add tests

9a5f69a

github-advanced-security bot found potential problems Nov 26, 2024

View reviewed changes

titaiwangms requested review from justinchuby, xadupre, gramalingam and titaiwangms December 3, 2024 17:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend unit tests #1967

Extend unit tests #1967

qododavid commented Nov 26, 2024

justinchuby commented Nov 26, 2024

codecov bot commented Nov 26, 2024 •

edited

Loading

github-advanced-security bot left a comment

		@@ -4,6 +4,11 @@

		import onnxscript.tools.benchmark.benchmark_helpers as bh

		import torch

Extend unit tests #1967

Are you sure you want to change the base?

Extend unit tests #1967

Conversation

qododavid commented Nov 26, 2024

Increased Coverage

Why adding unit tests is important

justinchuby commented Nov 26, 2024

codecov bot commented Nov 26, 2024 • edited Loading

Codecov Report

github-advanced-security bot left a comment

Choose a reason for hiding this comment

codecov bot commented Nov 26, 2024 •

edited

Loading