You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
I've noticed that, when trying to use a dataset that I've aligned using GPS reference in Metashape, it will not load, or it will load but produce no results. This has been an issue since nerfstudio implemented loading in point clouds (.ply) for splat seeding.
I thought that @simonbethke might have reported this problem when it first was being tested in pull #3122. When @jb-ye asked for sample data, I assumed that they had shared that somewhere, but I couldn't find any issue related to this, so I'm making one here.
On older (month ago) nerfstudio and gsplat versions, I was getting error: Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
On the ODM_mygla dataset on current versions, I'm getting this error after ~1000its: CONSOLE.log(f"Splitting {split_mask.sum().item()/self.num_points} gaussians: {n_splits}/{self.num_points}") ZeroDivisionError: division by zero
If I do process the data without the gps reference, it does load correctly.
I'm able to use this data when exporting using the gaussian splatting metashape script on inria's and postshot implementation of gaussian splatting. The data is exported using colmap format.
Why use GPS reference
I've found that using GPS reference in metashape helps with alignment (speed and accuracy), and usually results in a consistent scale, orientation, and ground plane.
I'm not asking to somehow implement GPS data in splats/nerfstudio-data, but to simply accept data created with it.
To Reproduce
My workflow usually consists of:
importing dataset(s) into metashape in different chunks. GPS data is auto selected in the reference panel from exif.
Align images, usually with high/highest details, and high number of keypoints and unlimited tiepoints.
Export camera positions and sparse/tie pointcloud using a batch job (Also tried manually)
Process data into nerfstudio using the following command: ns-process-data metashape --data "e:/3D Datasets/[Dataset Name]" --xml "e:/3D Datasets/[Dataset Name]/db.xml" --ply "e:/3D Datasets/[Dataset Name]/PointCloud.ply" --output-dir e:/NerfStudio/data/[Dataset Name]/out
Run Splatfacto with ns-train splatfacto --output-dir ./outputs/ODMlogs nerfstudio-data --data ./data/ODMlogs/out
After undistorting it either loads incorrectly or give an error
I've tried exporting camera positions/point cloud with local coordinates and wgs84 as the coordinate system, both give the same error.
Expected behavior
To be able to load the data that has been georeferenced.
Alternative solution
Using the colmap export script data to create the nerfstudio data, but this will only be an option with people who have Metashape pro, and in my opinion a workaround (maybe still useful for using data used in other programs).
I tried to simply copy and paste this data into the colmap structure in nerfstudio, but I failed. I'm sure that there is a command/script to convert this into something that nerfstudio can run, but I wasn't able to get something that works.
Here is how the folder is formatted in case it helps:
Screenshots
ODM_mygla processed using splatfacto georeferenced before it crashes:
The only notable detail is a small white dot at the bottom of the scene at possibly infinite distance. ODM_helenenschacht also displays similar results, but doesn't crash. Camera sometimes are obscured by the scene, so I need to disable composite depth to use the camera positions as reference.
ODM_mygla processed using nerfacto on same dataset:
Additional context
Machines specs and info:
Pc 1- R9 5900X, 128gb ram, 3060 12gb, data on SSD. Windows 10, anaconda, Nerfstudio 1.1.2, Gsplat 1.0.0
Pc 2- i7 7700HQ, 16gb ram, 1060 6gb, data on SSD. Windows 10, anaconda, Nerfstudio 1.1.0, Gsplat 0.1.12
All latest nvidia drivers, also tested with drivers from 4 months ago.
(I'm aware that Pc 2 won't be able to run any future gsplats, still included this info since the old version was giving different errors that might help narrowing down the problem)
Data processed using Metashape 2.x
I capture my own data using a Mavic Air 2s, it often aligns a few meters below ground, but even when adjusting for that, there are errors. I'm not sure how to share my own dataset, so here are some datasets that I've tested and display the same errors:
Datasets for reference: ODM_mygla 41 images ~5mb each. Captured on DJI phantom 3
ODM_helenenschacht 176 images ~12mb each. Captured on Autel Evo II Pro RTK
I've also tested, but pointcloud is too big. Here is the camera positions: db.xml.txt
This is my first issue submitted, so apologies for any missing info, or bad etiquette.
Dump of Logs
Console error from trying to run splatfacto with ODM_mygla dataset on nerfstudio 1.1.2 with PC 1
890 (2.97%) 8.359 ms 4 m, 3 s 78.09 M
---------------------------------------------------------------------------------------------------- splatfacto.py:
Viewer running locally at: http://localhost:7007 (listening on 0.0.0.0)
Printing profiling stats, from longest to shortest duration in seconds
Trainer.train_iteration: 0.0155
VanillaPipeline.get_train_loss_dict: 0.0118
Traceback (most recent call last):
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\[User]\.conda\envs\nerfstudio\Scripts\ns-train.exe\__main__.py", line 7, in <module>
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\scripts\train.py", line 262, in entrypoint
main(
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\scripts\train.py", line 247, in main
launch(
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\scripts\train.py", line 189, in launch
main_func(local_rank=0, world_size=world_size, config=config)
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\scripts\train.py", line 100, in train_loop
trainer.train()
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\engine\trainer.py", line 265, in train
callback.run_callback_at_location(
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\engine\callbacks.py", line 115, in run_callback_at_location
self.run_callback(step=step)
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\engine\callbacks.py", line 100, in run_callback
self.func(*self.args, **self.kwargs, step=step)
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\models\splatfacto.py", line 456, in refinement_after
split_params = self.split_gaussians(splits, nsamps)
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\models\splatfacto.py", line 543, in split_gaussians
CONSOLE.log(f"Splitting {split_mask.sum().item()/self.num_points} gaussians: {n_splits}/{self.num_points}")
ZeroDivisionError: division by zero
Console error from trying to run splatfacto with nerfstudio 1.1.0 with PC 2
[17:24:14] Caching / undistorting train images full_images_datamanager.py:183
Printing profiling stats, from longest to shortest duration in seconds
Trainer.train_iteration: 8.2725
VanillaPipeline.get_train_loss_dict: 8.2715
Traceback (most recent call last):
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\[User]\.conda\envs\nerfstudio\Scripts\ns-train.exe\__main__.py", line 7, in <module>
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\scripts\train.py", line 262, in entrypoint
main(
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\scripts\train.py", line 247, in main
launch(
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\scripts\train.py", line 189, in launch
main_func(local_rank=0, world_size=world_size, config=config)
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\scripts\train.py", line 100, in train_loop
trainer.train()
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\engine\trainer.py", line 261, in train
loss, loss_dict, metrics_dict = self.train_iteration(step)
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\utils\profiler.py", line 112, in inner
out = func(*args, **kwargs)
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\nerfstudio\engine\trainer.py", line 498, in train_iteration
self.grad_scaler.scale(loss).backward() # type: ignore
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\torch\_tensor.py", line 492, in backward
torch.autograd.backward(
File "C:\Users\[User]\.conda\envs\nerfstudio\lib\site-packages\torch\autograd\__init__.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
The text was updated successfully, but these errors were encountered:
Describe the bug
I've noticed that, when trying to use a dataset that I've aligned using GPS reference in Metashape, it will not load, or it will load but produce no results. This has been an issue since nerfstudio implemented loading in point clouds (.ply) for splat seeding.
I thought that @simonbethke might have reported this problem when it first was being tested in pull #3122. When @jb-ye asked for sample data, I assumed that they had shared that somewhere, but I couldn't find any issue related to this, so I'm making one here.
On older (month ago) nerfstudio and gsplat versions, I was getting error:
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
On the ODM_mygla dataset on current versions, I'm getting this error after ~1000its:
CONSOLE.log(f"Splitting {split_mask.sum().item()/self.num_points} gaussians: {n_splits}/{self.num_points}") ZeroDivisionError: division by zero
If I do process the data without the gps reference, it does load correctly.
I'm able to use this data when exporting using the gaussian splatting metashape script on inria's and postshot implementation of gaussian splatting. The data is exported using colmap format.
Why use GPS reference
I've found that using GPS reference in metashape helps with alignment (speed and accuracy), and usually results in a consistent scale, orientation, and ground plane.
I'm not asking to somehow implement GPS data in splats/nerfstudio-data, but to simply accept data created with it.
To Reproduce
My workflow usually consists of:
ns-process-data metashape --data "e:/3D Datasets/[Dataset Name]" --xml "e:/3D Datasets/[Dataset Name]/db.xml" --ply "e:/3D Datasets/[Dataset Name]/PointCloud.ply" --output-dir e:/NerfStudio/data/[Dataset Name]/out
ns-train splatfacto --output-dir ./outputs/ODMlogs nerfstudio-data --data ./data/ODMlogs/out
I've tried exporting camera positions/point cloud with
local coordinates
andwgs84
as the coordinate system, both give the same error.Expected behavior
To be able to load the data that has been georeferenced.
Alternative solution
Using the colmap export script data to create the nerfstudio data, but this will only be an option with people who have Metashape pro, and in my opinion a workaround (maybe still useful for using data used in other programs).
I tried to simply copy and paste this data into the colmap structure in nerfstudio, but I failed. I'm sure that there is a command/script to convert this into something that nerfstudio can run, but I wasn't able to get something that works.
Here is how the folder is formatted in case it helps:
[Dataset]/
├─ node_modules/
├─ images/
│ ├─ Image1.jpg
│ ├─ Image2.jpg
│ ├─ Image3.jpg
├─ sparse/
│ ├─ 0/
│ │ ├─ cameras.bin
│ │ ├─ images.bin
│ │ ├─ points3D.bin
Screenshots
![image](https://private-user-images.githubusercontent.com/25045316/342574579-764de58a-001c-4911-8339-0500d4169044.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3NDczNzYsIm5iZiI6MTcxOTc0NzA3NiwicGF0aCI6Ii8yNTA0NTMxNi8zNDI1NzQ1NzktNzY0ZGU1OGEtMDAxYy00OTExLTgzMzktMDUwMGQ0MTY5MDQ0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MzAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjMwVDExMzExNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWU3M2Y1NjViYmJjYTk3OGFiY2Y4ZjljMzIwYTM1YTA2MzJjY2M2NGFhMjRlNTU3N2ZkOWI0MWJkYTQzZTM5ZGUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.0I0BugFBmcPf0HewHH66VEssnFtpR13VRwkwSwC1gmM)
ODM_mygla processed using splatfacto georeferenced before it crashes:
The only notable detail is a small white dot at the bottom of the scene at possibly infinite distance. ODM_helenenschacht also displays similar results, but doesn't crash. Camera sometimes are obscured by the scene, so I need to disable
composite depth
to use the camera positions as reference.ODM_mygla processed using nerfacto on same dataset:
![image](https://private-user-images.githubusercontent.com/25045316/342573958-c6b67775-23b1-4e1d-8c3a-cb48d8186536.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3NDczNzYsIm5iZiI6MTcxOTc0NzA3NiwicGF0aCI6Ii8yNTA0NTMxNi8zNDI1NzM5NTgtYzZiNjc3NzUtMjNiMS00ZTFkLThjM2EtY2I0OGQ4MTg2NTM2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MzAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjMwVDExMzExNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTM4ZGQzZTRhYTZkYWQzMGQ2NDA1ZDg3ODRhZjNmMTQyOWNhMzIwNWNjNTJlOGMzZDUwZDNhMWU3ZmRmODk2YzUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.UKqFQ6H8iSXENXfW1RHMqVL6QmwLKrHlHGWc4njtbp0)
Additional context
Machines specs and info:
Pc 1- R9 5900X, 128gb ram, 3060 12gb, data on SSD. Windows 10, anaconda, Nerfstudio 1.1.2, Gsplat 1.0.0
Pc 2- i7 7700HQ, 16gb ram, 1060 6gb, data on SSD. Windows 10, anaconda, Nerfstudio 1.1.0, Gsplat 0.1.12
All latest nvidia drivers, also tested with drivers from 4 months ago.
(I'm aware that Pc 2 won't be able to run any future gsplats, still included this info since the old version was giving different errors that might help narrowing down the problem)
Data processed using Metashape 2.x
I capture my own data using a Mavic Air 2s, it often aligns a few meters below ground, but even when adjusting for that, there are errors. I'm not sure how to share my own dataset, so here are some datasets that I've tested and display the same errors:
Datasets for reference:
ODM_mygla 41 images ~5mb each. Captured on DJI phantom 3
Here are the export files as .txt, you'll need to change the extension:
db.xml.txt
PointCloud.ply.txt
ODM_helenenschacht 176 images ~12mb each. Captured on Autel Evo II Pro RTK
I've also tested, but pointcloud is too big. Here is the camera positions:
db.xml.txt
This is my first issue submitted, so apologies for any missing info, or bad etiquette.
Dump of Logs
Console error from trying to run
splatfacto
with ODM_mygla dataset on nerfstudio 1.1.2 with PC 1Console error from trying to run
splatfacto
with nerfstudio 1.1.0 with PC 2The text was updated successfully, but these errors were encountered: