[Feature Request] PadToSquare: Square Padding to Preserve Aspect Ratios When Resizing Images with Varied Shapes in torchvision.transforms.v2 #8701
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/8701
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
Hi @geezah! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at [email protected]. Thanks! |
There was a problem hiding this comment.
Thanks a lot for the PR @geezah . This looks great overall!
I was about to push some minor changes (and new test) to your PR, but I'm unfortunately not able to do so, as you started the changes from your main branch. Would you mind applying this diff into your PR and push it, so I can land it? Thank you!
diff --git a/test/test_transforms_v2.py b/test/test_transforms_v2.py
index ffd203f846..0731bf21a6 100644
--- a/test/test_transforms_v2.py
+++ b/test/test_transforms_v2.py
@@ -3929,41 +3929,33 @@ class TestPad:
class TestPadToSquare:
- @pytest.mark.parametrize(
- "image",
- [
- (make_image((3, 10), device="cpu", dtype=torch.uint8)),
- (make_image((10, 3), device="cpu", dtype=torch.uint8)),
- (make_image((10, 10), device="cpu", dtype=torch.uint8)),
- ],
- )
- def test__get_params(self, image):
+ # Most checks like the kernels are done in TestPad above
+ @pytest.mark.parametrize("height, width", ((3, 10), (10, 3), (10, 10)))
+ def test__get_params(self, height, width):
transform = transforms.PadToSquare()
- params = transform._get_params([image])
+ params = transform._get_params(make_image((height, width)))
- assert "padding" in params
- padding = params["padding"]
+ assert all(p >= 0 for p in params["padding"])
+ left, top, right, bottom = params["padding"]
- assert len(padding) == 4
- assert all(p >= 0 for p in padding)
+ if height < width:
+ assert width == height + top + bottom
+ elif height > width:
+ assert height == width + left + right
+ else:
+ assert left == top == right == bottom == 0
- height, width = F.get_size(image)
- assert max(height, width) == height + padding[1] + padding[3]
- assert max(height, width) == width + padding[0] + padding[2]
+ @pytest.mark.parametrize("height, width", ((3, 10), (10, 3), (10, 10)))
+ def test_pad_square_correctness(self, height, width):
+ output = transforms.PadToSquare()(make_image((height, width)))
+ assert F.get_size(output) == [10, 10]
@pytest.mark.parametrize(
- "image, expected_output_shape",
- [
- (make_image((3, 10), device="cpu", dtype=torch.uint8), [10, 10]),
- (make_image((10, 3), device="cpu", dtype=torch.uint8), [10, 10]),
- (make_image((10, 10), device="cpu", dtype=torch.uint8), [10, 10]),
- ],
+ "make_input",
+ [make_image_tensor, make_image_pil, make_image, make_bounding_boxes, make_segmentation_mask, make_video],
)
- def test_pad_square_correctness(self, image, expected_output_shape):
- transform = transforms.PadToSquare()
- output = transform(image)
-
- assert F.get_size(output) == expected_output_shape
+ def test_transform(self, make_input):
+ check_transform(transforms.PadToSquare(), make_input())
class TestCenterCrop:
diff --git a/torchvision/transforms/v2/_geometry.py b/torchvision/transforms/v2/_geometry.py
index 744db5a22e..9a5d659f4e 100644
--- a/torchvision/transforms/v2/_geometry.py
+++ b/torchvision/transforms/v2/_geometry.py
@@ -537,28 +537,21 @@ class PadToSquare(Transform):
self.fill = _setup_fill_arg(fill)
def _get_params(self, flat_inputs: List[Any]) -> Dict[str, Any]:
- # Get the original height and width from the inputs
orig_height, orig_width = query_size(flat_inputs)
- # Find the target size (maximum of height and width)
target_size = max(orig_height, orig_width)
if orig_height < target_size:
- # Need to pad height
- pad_height = target_size - orig_height
- pad_top = pad_height // 2
- pad_bottom = pad_height - pad_top
- pad_left = 0
- pad_right = 0
+ amount_to_pad = target_size - orig_height
+ pad_top = amount_to_pad // 2
+ pad_bottom = amount_to_pad - pad_top
+ pad_left = pad_right = 0
else:
- # Need to pad width
- pad_width = target_size - orig_width
- pad_left = pad_width // 2
- pad_right = pad_width - pad_left
- pad_top = 0
- pad_bottom = 0
-
- # The padding needs to be in the format [left, top, right, bottom]
+ amount_to_pad = target_size - orig_width
+ pad_left = amount_to_pad // 2
+ pad_right = amount_to_pad - pad_left
+ pad_top = pad_bottom = 0
+
return dict(padding=[pad_left, pad_top, pad_right, pad_bottom])
def _transform(self, inpt: Any, params: Dict[str, Any]) -> Any:
|
Thanks for your feedback! After experimenting with the PadToSquare transform and inspecting the learned features, I realized that it might introduce spurious correlations. For instance, if images of a particular class (e.g., airplanes) are predominantly in a specific format (e.g., landscape, where width > height), the transformation could reinforce this bias. Based on these outcomes, I would now advise against including this transformation in torchvision. A safer approach is likely to use a batch size of 1 and, optionally if required by the model architecture, resize the image to a multiple of the expected dimensions. If object location annotations are available, images can also be safely cropped using the bounding box coordinates to ensure the object remains within the cropped region. This way, multiple images can be resized together after being cropped and thus be batched at training time. |
Closes #8699.