How to use Custom RGB Image as background during rendering

[shivangi-aneja]

Hi,
This looks like an awesome library. I see that you are building support for background models. I was wondering if it would be possible to use a custom RBB image (3 channels) as background during training instead of using a single background color as done in vanilla 3DGS.
I want to reconstruct a scene with single object with precise boundaries, however the matting that I apply to my multiview images (using off-the-shelf matting methods) is not 100% perfect so I get some artefacts in my scene during rendering. Rendering it with a pre-captured background could solve it, I guess, but I think it's not possible to do this with vanilla 3DGS. Do you plan to provide something like that?

[hugoycj]

Great suggestion! We have integrated the bg_image feature into both the dataset and the renderer. Now, when you train using a dataset that includes masks, we will automatically use the background portion of the original image as the background model. This should be beneficial in your scenario!

[hugoycj]

To test the feature, you can utilize this repository. Refer to the implementation shown in the following code snippets from train.py:

if viewpoint_cam.bg_image is not None:
    bg = viewpoint_cam.bg_image.to("cuda").permute(2, 0, 1)
else:
    if dataset.white_background:
        bg_image_tensor = torch.ones((height, width, 3)).to("cuda")
    else:
        bg_image_tensor = torch.zeros((height, width, 3)).to("cuda")

Additionally, in __init__.py:

rendered_image = rendered_image + (1-rendered_final_opacity).repeat(3, 1, 1) * bg_color

We have now added support to use the preloaded bg_image as the background color.

[hugoycj]

As shown in [colmap.py]https://github.com/GAP-LAB-CUHK-SZ/gaustudio/blob/master/gaustudio/datasets/colmap.py#L94C1-L105C87,

mask = cv2.imread(str(mask_path), cv2.IMREAD_GRAYSCALE)
_, mask = cv2.threshold(mask, 1, 255, cv2.THRESH_BINARY) # Ensure mask is binary so multiply operation works as expected
mask = cv2.resize(mask, (width, height)) # Resize the mask to match the size of the image
bg_mask = cv2.bitwise_not(mask) # Invert the mask to get the background
bg_image = cv2.bitwise_and(_image, _image, mask=bg_mask)
_image_tensor = torch.from_numpy(cv2.cvtColor(_image, cv2.COLOR_BGR2RGB)).float() / 255
bg_image_tensor = torch.from_numpy(cv2.cvtColor(bg_image, cv2.COLOR_BGR2RGB)).float() / 255
_mask_tensor = torch.from_numpy(mask)
_camera = datasets.Camera(R=R, T=T, FoVy=FoVy, FoVx=FoVx, image_name=os.path.basename(extr.name), 
              image_width=width, image_height=height, image=_image_tensor, 
              bg_image=bg_image_tensor, mask=_mask_tensor)

We use the given mask to calculate the background