PointCloud from drone aerial images

[bgramaje]

Hello everyone,

First of all, thanks for this awesome project, has helped me a lot in diverse ways.

I need help in the reconstrucion of a point cloud given a set of images taken by a drone. The images are mainly from electrical poles where the done goes and takes a set of pictures. I've also prepared and AI that identifies the tower in the images, and generates a mask per images. Find an example below this paragraph.

My question is which parameters should I use when using DensifiyPointCloud.cpp, cause I have got some results like this

How should I refine the parameters, and also if there is anyway to identify which best parameters could be used depending on the scenario given by the dataset, just for automatically do this process if there is a way.

Thank you so much!

[cdcseacave]

The problem is that the masks you generate for the towers are not very accurate (they do have very few false positives, but a lot of false negatives) so they mise a lot of the real tower structure and you can not use them directly. Therefor these masks should be used in my opinion only for view neighbor selection during depth-map estimation, but otherwise the depth-map estimation itself to be done on the entire image. These would be the best approach here, but would require some code customization.

An other solution, sub-optimal but could work well enough, is to run in tower mode: --tower-mode 4

[bgramaje]

Could you please further detail why my masks arent accurate. They only need to be accurate on the objective I want to represent right? How could I specify to only use masks then for view neighbour selection? Sorry for this basic questions, but this is my first time using photogramettry and still processing all this stuff

Also I've tried with tower-mode set to 4, but the result wanst really nice, I got some better results forcing it to be tower-mode with -1 option

Thanks for the answer

[bgramaje]

This is with the following parameters:

 DensifyPointCloud scene.mvs --dense-config-file Densify.ini --resolution-level 0 --number-views 4 -w "/home/balbert/europe/r2d2/3d-reconstruction-towers/demo/towerCrevillente8_out3/mvs" --mask-path "/home/balbert/europe/r2d2/3d-reconstruction-towers/demo/towerCrevillente8/masks" --cuda-device 0 --tower-mode -1 --filter-point-cloud 1 --ignore-mask-label 0 --estimate-roi 0 --crop-to-roi false --estimate-colors 1 --number-views-fuse 2

But still is too noisy, so I dont know how can I further refine this process. Kind of stuck right now

[cdcseacave]

the masks are missing a lot of the structure of the tower

pls share the dataset undistorted images and scene.mvs

[bgramaje]

I've also come to the approach of giving the mask to the openMVG rather than the openMVS. The good thing is that its way much faster giving the mask to the openMVG rather than giving exclusively to the openMVS.

As openMVG states (https://openmvg.readthedocs.io/en/latest/software/SfM/ComputeFeatures/):

For this kind of needs you can use a mask. A mask is simply a binary image having the same size (width and height) than the target image. The black areas on a mask denote the “bad parts”, i.e. the areas to be masked and for which descriptors are not computed. A point is kept if the mask value at the point position is different than 0. In openMVG_main_ComputeFeatures, the association of a mask and an image is implicit. It uses the following conventions:

It tries to load a global mask.png file from where the images are stored.
It tries to load an individual <image_name>_mask.png from directory where the current image is stored.

Since it says that:

A point is kept if the mask value at the point position is different than 0. 

I am now also currently passing a mask for each image, but not a mask being the black part the thing to dismiss and the white part the thing to keep. I am cropping the image with a YOLO model, and placing it the cropped part on a black background image.

The dataset structure is the one as follows.

• Under the directory images there are both the raw images and the ${raw}_mask.png mask images. The raw images have been resized to FULLHD.
• Under the directory masks is the same masks but rather than being the cropped image, I use a neural network model in charge of removing the background of it and building a mask of it.

Here you have the dataset, I've also shared the last scene.mvs I got.
https://drive.google.com/drive/folders/1b2p9Kq46jmrdfiSmmo4WfZ9itrzDdYkl?usp=sharing

This is the final result I got:

PS. Also using the MvgMvsPipeline.py python script just in case.

And thanks for the help in this one. Highly appreciated.

[cdcseacave]

There are missing images (used in scene.mvs but missing from the images folder), and you should not add the mask png images from images folder to scene.mvs as separate images.

Also, I suggest you use the same black/white mask for OpenMVG, and not a crop of the RGB image. The idea is to use feautres only from the tower structure, and not from the background, so that the tower pointcloud is optimized.

[bgramaje]

Maybe because on the same images folder there is the masks as well for the openMVG?
About this you should not add the mask png images from images folder to scene.mvs as separate images. I used to use the masks from the mask folder during the pointcloud, and the cropped ones during the openMVG, I can use the black/white ones during the both processes. But which masks should I use? Cause when doing the pipeline, in the outputh path, i got under the mvs directory, a subdirectort called images where the original ones + a mask is placed.

Should I use this mask for the DensifyPointCloud? I am having trouble understanding the actual behaviour of the masks

Thanks for the support

[bgramaje]

Also, how can I check this There are missing images (used in scene.mvs but missing from the images folder), how can I see which images are not under the images folder and had been used? Thanks

[bgramaje]

Hello @cdcseacave ,

I've uploaded a new set of images and new-computed masks to the Google Drive, could you please have a look at it and tell me if they are good enough for the point cloud reconstruction?

Also should I use those masks in the OpenMVG as well, or just in the DensifyPointCloud process.

https://drive.google.com/drive/folders/1b2p9Kq46jmrdfiSmmo4WfZ9itrzDdYkl?usp=sharing

Thanks for the effort.

[bgramaje]

Also with the current dataset, I have changed the pipeline. Rather than injecting the mask into the images folder to be used by the OpenMVG, thus giving non-accurate matching points, I am sending the raw image completely into the OpenMVG. At the DesnsifyPointCloud process, I give the param of the mask-path into the CLI.

Nevertheless I am not achieving to make understand that the scene is a tower. All time says the following.

13:59:30 [App     ] error: does not seem to be a tower: X(43.22), Y(68.26), Z(104.87)

Given the following piece of code in Scene.cpp:

if (quality.y / quality.z > 0.6f || quality.x / quality.y < 0.8f) {
		// does not seem to be a line
		if (towerMode > 0) {
			DEBUG_ULTIMATE("error: does not seem to be a tower: X(%.2f), Y(%.2f), Z(%.2f)", quality.x, quality.y, quality.z);
			return false;
		}
	}

I see that the division of 68.26f / 104.87f is 0.65f which is greater than the 0.6f thereshold that idetifies the scene as a tower. Why is that thresehold there as well, how is it computed or decided to use that one?

Also, thanks for disturbing you, and thanks as well for this repository. Highly appreciated.

[bgramaje]

Hey @cdcseacave, sorry for bothering you,

I have been using the Line3Dpp to see if the features are extracted smoothly, and I got the following results:

It seemts that the COLMAP process has identifed the features and the points there, since the Line3Dpp has been able to identify those for the outline model, but it seems that the DensifyPointCloud process did not.

Attached you can see the output of the DensifyPointCloud process
DensifyPointCloud-2410021052138B5A05.log

Thanks for this, hoping to hear from you back on this matter,

Sorry for all the inconveniences

Best
@