Implement defocus deblurring. #469

jefequien · 2024-10-28T18:57:33Z

Loosely based on Deblurring 3D Gaussian Splatting, https://arxiv.org/pdf/2401.00834 and BAGS, https://arxiv.org/pdf/2403.04926.

Each step requires rendering both clear and blurry images so this method is >2x slower than normal 3DGS.

defocuscake_15k.mp4

Evaluation

Run bash examples/benchmarks/mcmc_deblur.sh

	psnr	ssim	lpips	num_GS
defocuscake	26.8007	0.891839	0.137933	250000
defocuscaps	24.3061	0.838382	0.201384	250000
defocuscisco	20.4781	0.861181	0.0854522	250000
defocuscoral	19.3734	0.719444	0.285026	250000
defocuscupcake	22.2543	0.82716	0.127639	250000
defocuscups	25.2897	0.875219	0.147801	250000
defocusdaisy	23.6329	0.837565	0.171997	250000
defocussausage	18.4746	0.817682	0.216327	250000
defocusseal	25.757	0.883999	0.121537	250000
defocustools	27.2291	0.933414	0.0903888	250000
AVERAGE	23.3596	0.848589	0.158548	250000

ichsan2895 · 2024-11-01T14:46:56Z

How to use this method? What is the command?

jefequien · 2024-11-01T15:30:22Z

Run bash examples/benchmarks/mcmc_deblur.sh. You may need to install https://github.com/NVlabs/tiny-cuda-nn to use the mlps. I'll post more detailed instructions soon.

ichsan2895 · 2024-11-02T10:38:36Z

I tried with multi GPU support, but it did not work:

>> CUDA_VISIBLE_DEVICES=0,1 python3 examples/simple_trainer.py mcmc --disable_viewer --data_factor $DATA_FACTOR \
        --strategy.cap-max $CAP_MAX \
        --blur_opt \
        --blur_opt_lr 1e-3 \
        --blur_a 0.8 \
        --blur_mask_reg 0.002 \
        --render_traj_path $RENDER_TRAJ_PATH \
        --data_dir $SCENE_DIR/$SCENE/ \
        --result_dir $RESULT_DIR/$SCENE \
        --steps-scaler 0.5

  0%|          | 0/15000 [00:01<?, ?it/s]
  0%|          | 0/15000 [00:01<?, ?it/s]
Exception in thread Thread-2 (_pin_memory_loop):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/pin_memory.py", line 54, in _pin_memory_loop
    do_one_step()
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/pin_memory.py", line 31, in do_one_step
    r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL)
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 122, in get
    return _ForkingPickler.loads(res)
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/reductions.py", line 355, in rebuild_storage_fd
    fd = df.detach()
  File "/usr/lib/python3.10/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/usr/lib/python3.10/multiprocessing/resource_sharer.py", line 86, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 508, in Client
    answer_challenge(c, authkey)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 752, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/workspace/GSPLAT_DEFOCUSJEF/gsplat/examples/simple_trainer.py", line 1193, in <module>
    cli(main, cfg, verbose=True)
  File "/workspace/GSPLAT_DEFOCUSJEF/gsplat/gsplat/distributed.py", line 344, in cli
    process_context.join()
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 163, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 74, in _wrap
    fn(i, *args)
  File "/workspace/GSPLAT_DEFOCUSJEF/gsplat/gsplat/distributed.py", line 295, in _distributed_worker
    fn(local_rank, world_rank, world_size, args)
  File "/workspace/GSPLAT_DEFOCUSJEF/gsplat/examples/simple_trainer.py", line 1138, in main
    runner.train()
  File "/workspace/GSPLAT_DEFOCUSJEF/gsplat/examples/simple_trainer.py", line 659, in train
    blur_mask = self.blur_module.predict_mask(image_ids, depths)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'DistributedDataParallel' object has no attribute 'predict_mask'

It worked well with single GPU setup. I will share the result later. Its training now.

ichsan2895 · 2024-11-02T13:09:46Z

Some of result is coming :)

Top video = Gsplat MCMC with 500k splats & Data-Factor 4.
Bottom video = Gsplat MCMC with 500k splats & Data-Factor 4 with defocusing.

defocuscupcake_comparison_29999.mp4

defocusdaisy_comparison_29999.mp4

defocusseal_comparison_29999.mp4

It worked well with single GPU setup. Hopefully the multiGPU setup will be fixed.

ichsan2895 · 2024-11-02T13:49:23Z

Gsplat MCMC with 500k splats & Data-Factor 4.

NUM	SCENE	PSNR	SSIM	LPIPS	VRAM (GB)	NUM_GS
0	defocuscake	24.716	0.832	0.207	0.714	500000
1	defocuscaps	22.442	0.744	0.285	0.751	500000
2	defocuscisco	20.571	0.848	0.119	0.805	500000
3	defocuscoral	19.649	0.750	0.292	0.741	500000
4	defocuscupcake	21.388	0.762	0.191	0.812	500000
5	defocuscups	24.274	0.827	0.218	0.744	500000
6	defocusdaisy	22.175	0.720	0.285	0.695	500000
7	defocussausage	17.324	0.729	0.302	0.896	500000
8	defocusseal	22.105	0.681	0.286	0.780	500000
9	defocustools	24.886	0.884	0.148	0.808	500000
10	AVERAGE	21.953	0.778	0.233	0.775	500000

Gsplat MCMC with 500k splats & Data-Factor 4 with Defocusing.

NUM	SCENE	PSNR	SSIM	LPIPS	VRAM (GB)	NUM_GS
0	defocuscake	26.439	0.886	0.130	1.814	500000
1	defocuscaps	23.874	0.837	0.183	1.815	500000
2	defocuscisco	20.488	0.853	0.101	1.828	500000
3	defocuscoral	19.419	0.834	0.156	1.838	500000
4	defocuscupcake	22.529	0.850	0.090	1.847	500000
5	defocuscups	24.510	0.851	0.175	1.844	500000
6	defocusdaisy	23.659	0.856	0.123	1.806	500000
7	defocussausage	18.159	0.830	0.181	1.856	500000
8	defocusseal	25.041	0.865	0.129	1.797	500000
9	defocustools	27.869	0.947	0.057	1.841	500000
10	AVERAGE	23.199	0.861	0.133	1.829	500000

Tested with :
Ubuntu 22.04.3 LTS
Python 3.10.12
Torch 2.1.2+cu118
gsplat commit 8f7e92c

jefequien · 2024-11-02T14:18:00Z

Thanks for running this. I can get slightly higher metrics with different hyperparameters but I haven't been able to beat the 23.61 avg reported in the original paper. Also, if anyone has an out of focus real world dataset, I'd be curious to see how this performs.

ichsan2895 · 2024-11-02T14:30:29Z

Thanks for running this. I can get slightly higher metrics with different hyperparameters but I haven't been able to beat the 23.61 avg reported in the original paper. Also, if anyone has an out of focus real world dataset, I'd be curious to see how this performs.

How about using higher num of splats such as 1 million?

jefequien · 2024-11-02T14:33:48Z

I found 250k and 500k get similar metrics. The original paper uses ~300k so it would be good to keep things fair.

ichsan2895 · 2024-11-02T14:34:45Z

BTW SSIM is the way higher than the original paper. (0.861 vs 0.7471) 👍

jefequien · 2024-11-02T14:35:57Z

I looked into it briefly. The SSIM implementation is different so don't pay much attention to that.

ichsan2895 · 2024-11-02T23:47:21Z

Somehow, it do not work good in real_motion_blur dataset.

Gsplat MCMC with 500k splats & Data-Factor 4

NUM	SCENE	PSNR	SSIM	LPIPS	VRAM (GB)	NUM_GS
0	blurball	23.112	0.643	0.322	0.767	500000
1	blurbasket	23.342	0.767	0.260	0.732	500000
2	blurbuick	20.915	0.700	0.263	0.734	500000
3	blurcoffee	25.657	0.857	0.217	0.786	500000
4	blurdecoration	21.951	0.716	0.274	0.768	500000
5	blurgirl	19.622	0.774	0.258	0.744	500000
6	blurheron	20.090	0.558	0.317	0.711	500000
7	blurparterre	12.799	0.299	0.661	0.787	500000
8	blurpuppet	21.687	0.703	0.273	0.751	500000
9	blurstair	22.600	0.613	0.346	0.844	500000
10	AVERAGE	21.178	0.663	0.319	0.762	500000

Gsplat MCMC with 500k splats & Data-Factor 4 with Defocusing.

NUM	SCENE	PSNR	SSIM	LPIPS	VRAM (GB)	NUM_GS
0	blurball	23.211	0.646	0.316	1.836	500000
1	blurbasket	24.216	0.794	0.236	1.828	500000
2	blurbuick	22.053	0.726	0.231	1.830	500000
3	blurcoffee	26.553	0.866	0.198	1.880	500000
4	blurdecoration	23.267	0.760	0.242	1.854	500000
5	blurgirl	19.707	0.788	0.232	1.840	500000
6	blurheron	20.664	0.598	0.294	1.816	500000
7	blurparterre	14.115	0.302	0.631	1.846	500000
8	blurpuppet	22.475	0.733	0.242	1.835	500000
9	blurstair	22.766	0.647	0.311	1.881	500000
10	AVERAGE	21.903	0.686	0.293	1.845	500000

How about other real world dataset?

I tested a dataset called dozer from nerfstudio-download-data. The metrics is low since it has more sharper image than original image!

ichsan2895 · 2024-11-14T15:09:51Z

Seems an interesting update. Hopefully I can benchmark it with previous dataset again.

ichsan2895 · 2024-11-16T13:35:33Z

Here is it my benchmark. Hopefully it will helpful for you @jefequien

Gsplat MCMC with 500k splats & Data-Factor 4 with Defocusing commit `8f7e92c`.

CUDA_VISIBLE_DEVICES=0 python3 examples/simple_trainer.py mcmc \
        --disable_viewer --data_factor $DATA_FACTOR \
        --strategy.cap-max $CAP_MAX \
        --blur_opt \
        --blur_opt_lr 1e-3 \
        --blur_a 0.8 \
        --blur_mask_reg 0.002 \
        --render_traj_path $RENDER_TRAJ_PATH \
        --data_dir $SCENE_DIR/$SCENE/ \
        --result_dir $RESULT_DIR/$SCENE

real deblur dataset

NUM	SCENE	PSNR	SSIM	LPIPS	VRAM (GB)	NUM_GS
0	defocuscake	26.439	0.886	0.130	1.814	500000
1	defocuscaps	23.874	0.837	0.183	1.815	500000
2	defocuscisco	20.488	0.853	0.101	1.828	500000
3	defocuscoral	19.419	0.834	0.156	1.838	500000
4	defocuscupcake	22.529	0.850	0.090	1.847	500000
5	defocuscups	24.510	0.851	0.175	1.844	500000
6	defocusdaisy	23.659	0.856	0.123	1.806	500000
7	defocussausage	18.159	0.830	0.181	1.856	500000
8	defocusseal	25.041	0.865	0.129	1.797	500000
9	defocustools	27.869	0.947	0.057	1.841	500000
10	AVERAGE	23.199	0.861	0.133	1.829	500000

real motion blur dataset

NUM	SCENE	PSNR	SSIM	LPIPS	VRAM (GB)	NUM_GS
0	blurball	23.211	0.646	0.316	1.836	500000
1	blurbasket	24.216	0.794	0.236	1.828	500000
2	blurbuick	22.053	0.726	0.231	1.830	500000
3	blurcoffee	26.553	0.866	0.198	1.880	500000
4	blurdecoration	23.267	0.760	0.242	1.854	500000
5	blurgirl	19.707	0.788	0.232	1.840	500000
6	blurheron	20.664	0.598	0.294	1.816	500000
7	blurparterre	14.115	0.302	0.631	1.846	500000
8	blurpuppet	22.475	0.733	0.242	1.835	500000
9	blurstair	22.766	0.647	0.311	1.881	500000
10	AVERAGE	21.903	0.686	0.293	1.845	500000

Gsplat MCMC with 500k splats & Data-Factor 4 with Defocusing commit `d9db9e0`

CUDA_VISIBLE_DEVICES=0 python3 examples/simple_trainer.py mcmc \
        --disable_viewer --data_factor $DATA_FACTOR \
        --strategy.cap-max $CAP_MAX \
        --blur_opt \
        --render_traj_path $RENDER_TRAJ_PATH \
        --data_dir $SCENE_DIR/$SCENE/ \
        --result_dir $RESULT_DIR/$SCENE

real deblur dataset

NUM	SCENE	PSNR	SSIM	LPIPS	VRAM (GB)	NUM_GS
0	defocuscake	26.138	0.869	0.157	2.491	500000
1	defocuscaps	23.579	0.829	0.194	2.506	500000
2	defocuscisco	20.496	0.860	0.084	2.537	500000
3	defocuscoral	19.670	0.764	0.248	2.509	500000
4	defocuscupcake	22.490	0.832	0.117	2.562	500000
5	defocuscups	25.439	0.876	0.145	2.560	500000
6	defocusdaisy	23.366	0.846	0.138	2.454	500000
7	defocussausage	18.232	0.823	0.192	2.570	500000
8	defocusseal	25.527	0.879	0.122	2.467	500000
9	defocustools	27.717	0.944	0.063	2.548	500000
10	AVERAGE	23.265	0.852	0.146	2.520	500000

real motion blur dataset

NUM	SCENE	PSNR	SSIM	LPIPS	VRAM (GB)	NUM_GS
0	blurball	23.219	0.649	0.316	2.531	500000
1	blurBasket	23.731	0.783	0.236	2.518	500000
2	blurBuick	22.016	0.725	0.238	2.537	500000
3	blurCoffee	26.288	0.862	0.194	2.549	500000
4	blurDecoration	22.716	0.735	0.265	2.577	500000
5	blurGirl	18.615	0.772	0.250	2.540	500000
6	blurHeron	20.751	0.602	0.284	2.500	500000
7	blurParterre	22.894	0.648	0.303	2.483	500000
8	blurPuppet	21.597	0.711	0.265	2.553	500000
9	blurStair	22.791	0.619	0.336	2.646	500000
10	AVERAGE	22.462	0.711	0.269	2.543	500000

ichsan2895 · 2024-11-16T14:03:59Z

Top : commit 8f7e92c
Bottom : commit d9db9e0

defocuscake_defocus1-vs-defocus2_traj_val_29999.mp4

defocuscoral_comparison_deblur1-vs-2_29999.mp4

defocusseal_comparison_deblur1-vs-2_29999.mp4

jefequien added 30 commits October 4, 2024 23:18

gtnet

cf9a806

reduce diff

5088b3b

refactor

214c8f0

exact same gtnet

d00cb1a

wip

5aecb73

cleanup

15f8898

simplify

9767f14

masks works

e168738

mask is working

1e338e3

mlp mask

abad1d1

mlp mask

052d272

no need for quantile

3d4b5c0

single mlp

649acd0

cleanup

fad9b21

cleanup

2ae57b9

focal embedding

904979e

init blur

c367c0a

reg to prevent collapse

ba514ed

mlp

6da4119

deltas mlp

5714315

cleanup

291a74c

tcnn works with log_transform

ad9c5cd

cleanup

fa5957a

send

2659a9d

log depth

464249d

send

af99af2

init focal

e58b2da

init strat

e70f474

log 10 center init around 0.2

beac915

slower lr

56f7937

jefequien added 2 commits October 23, 2024 22:54

less freqs

ed830d7

latest

d874ef1

jefequien marked this pull request as draft October 28, 2024 18:58

jefequien added 10 commits October 28, 2024 16:53

latest run avg psnr 23.50

8a02e74

cleanup

c8fd74d

cleanup

834e4e8

docstring

cb826c6

docstring

ac0cef5

rescale

a3d7d15

minor

10bac1d

warmup 3

68289ae

delayed start instead of warmup

d95b929

wip

8f7e92c

jefequien added 3 commits November 13, 2024 11:57

bounded l1 losos

f355877

mlp folder

a14bcb7

Merge branch 'main' into jeff/defocus

d9db9e0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement defocus deblurring. #469

Implement defocus deblurring. #469

jefequien commented Oct 28, 2024 •

edited

Loading

ichsan2895 commented Nov 1, 2024

jefequien commented Nov 1, 2024

ichsan2895 commented Nov 2, 2024

ichsan2895 commented Nov 2, 2024 •

edited

Loading

ichsan2895 commented Nov 2, 2024

jefequien commented Nov 2, 2024 •

edited

Loading

ichsan2895 commented Nov 2, 2024

jefequien commented Nov 2, 2024 •

edited

Loading

ichsan2895 commented Nov 2, 2024 •

edited

Loading

jefequien commented Nov 2, 2024 •

edited

Loading

ichsan2895 commented Nov 2, 2024

ichsan2895 commented Nov 14, 2024

ichsan2895 commented Nov 16, 2024

ichsan2895 commented Nov 16, 2024

Implement defocus deblurring. #469

Are you sure you want to change the base?

Implement defocus deblurring. #469

Conversation

jefequien commented Oct 28, 2024 • edited Loading

Evaluation

ichsan2895 commented Nov 1, 2024

jefequien commented Nov 1, 2024

ichsan2895 commented Nov 2, 2024

ichsan2895 commented Nov 2, 2024 • edited Loading

ichsan2895 commented Nov 2, 2024

Gsplat MCMC with 500k splats & Data-Factor 4.

Gsplat MCMC with 500k splats & Data-Factor 4 with Defocusing.

jefequien commented Nov 2, 2024 • edited Loading

ichsan2895 commented Nov 2, 2024

jefequien commented Nov 2, 2024 • edited Loading

ichsan2895 commented Nov 2, 2024 • edited Loading

jefequien commented Nov 2, 2024 • edited Loading

ichsan2895 commented Nov 2, 2024

Gsplat MCMC with 500k splats & Data-Factor 4

Gsplat MCMC with 500k splats & Data-Factor 4 with Defocusing.

ichsan2895 commented Nov 14, 2024

ichsan2895 commented Nov 16, 2024

Gsplat MCMC with 500k splats & Data-Factor 4 with Defocusing commit 8f7e92c.

Gsplat MCMC with 500k splats & Data-Factor 4 with Defocusing commit d9db9e0

ichsan2895 commented Nov 16, 2024

jefequien commented Oct 28, 2024 •

edited

Loading

ichsan2895 commented Nov 2, 2024 •

edited

Loading

jefequien commented Nov 2, 2024 •

edited

Loading

jefequien commented Nov 2, 2024 •

edited

Loading

ichsan2895 commented Nov 2, 2024 •

edited

Loading

jefequien commented Nov 2, 2024 •

edited

Loading

Gsplat MCMC with 500k splats & Data-Factor 4 with Defocusing commit `8f7e92c`.

Gsplat MCMC with 500k splats & Data-Factor 4 with Defocusing commit `d9db9e0`