Could I use this for creating art?

[m3504]

Hello Mr. Park Cham Chi

I am a student seeking for creating this effect in my VR work. When I spot this, I am very amazed by your work!
I just started to learn everything relates to VR so I think generally the instruction is not sufficient for beginners. So I would be very appreciated if you could kindly provide me some answers here.
The questions I have now is if it's possible to create mesh right from the video clip? Just like the one we preview in the build. Because I have this green screen clips I would like to transfer into VR and make work with them.
Thank you!!!

[parkchamchi]

Hi!
By the mesh do you mean the Unity mesh to import in the other project? (If not, you can use the mp4 export) There is a console command to save the current mesh and the texture (sa) but it only works in editor and saves only single frame - & the parameters (DepthMult, ...) can't be changed. The method I'd use is to save depth maps only - which is generated when you toggle Save the output toggle or run the python script, which makes a zip file with depth maps per frame. But this approach requires the target project to implement the depthfile decoding and control mesh like this program. Also the multimedia control is needed.
More context for your project will be helpful. If you have any other questions feel free to ask me. Thx

[m3504]

Hello,

Thank you for reaching out! Woo it seems more complicated than I thought but thank you for answering!
And also thank you for asking, more context about my work is I am trying to put stopmotion footages in VR. The footages would be flat 2D film(,mp4) with green screen, and I would like to create depth for them and make fake 3D(mesh) in VR. Afterwards using Unity to remove the green part. Ideally, I wish to get some generated film clip in VR.
:) Thank you !

[parkchamchi]

1. Would the Unity program you mentioned be an extension of this repo or another standalone project?
2. Also, what method are you thinking to use for removing the green part? I can think of: (i) set the threshold for the depth, since the green part will mostly have the "far" depth (ii) or, detect the green part and cut it off, using OpenCV & Python
3. Would the final product be a Unity program that has interactivity such as parameter manipulation etc, or would it be a 3D 360 mp4 file? Or a intermediate file that contains depth maps to be used by another software?

Ty

[m3504]

1, It will be an independent standalone project, more or less like an interactive VR experience. The effect that depth viewer presents would serve as the visual assistance in the project. Overall I mainly wish to make 2D film in VR looks like 3D. and use all these films to recreate a surrounding.

2, I was using video player and Transparent Material with ChromaKeyShader to mask the color. But it was heavily legged. Using(i) can be ideal but since you mention it is only possible to get one frame of mesh in editor mode right?
3, It would be interactive experience. But mainly some simple interaction. So with the depth map which making the film runs real time in VR will be sufficient already.

Thank you!! I don't know if I make anything clear, since I haven't gained enough knowledge for programming. But I'm super grateful that you are willing to provide insights and assistance.

[m3504]

movie_002.mp4

As you can see, they are super flat without depth map :,(

[parkchamchi]

Okay, one more: does the image/video input determined on runtime, so that the depth has to be inferred on program? Or is it fixed so the pre-generated depth map can be used?

[m3504]

Sorry, I have to admit I do not fully understand what runtime is, but I think it should be applicable with the pre-generated depth maps.
They are loop films so plainly playing the experience is like playing those video in VR, which is composed by certain orders of pictures. Even if I want to interact with them (for example lower the playback speed of the film in realtime), it should be the same order of pics. So with the same order of depth maps, should work properly with the viedo(mesh) ?

Thank you !!

[parkchamchi]

OK, there is a feature in this program for saving depth maps in the video. When you toggle Save the mesh (which you have) a collection of depth maps per frame will be generated - in the folder that opens when you click the Open button. Now this is a renamed .zip file that has depth maps as .pgm file (which can be previewed with photo viewer program such as HoneyView). When you load videos in Unity I assume that you use VideoPlayer, which can retrieve its frame number - and the corresponding depth map is in [framenumber].pgm. What we do is read this depth map, which is a sequence of float or Depth object. Now that we have the depth map, we should use this to actually print on screen. There is a MeshBehavior code that creates a mesh and accepts texture (from VideoPlayer) and the corresponding depth map. When we call this a mesh with the video frame's texture and depth map will be shown.

Now that was a wall of text, I can draw you a simple diagram soon. If you reuse this repo's code (it becomes fairly simpler if you only implement the features above) it can be a trivial job.

[parkchamchi]

Pardon my childish drawing.

1. Consider there exists a video file "video.mp4" We play this with `vp.play("video.mp4").
2. We can retrieve the framenumber of the current frame, which can be 1.
3. We also can retrieve the texture for this frame, texture.
4. There also exists a pre-generated depth maps, as [framenumber].pgm. In this case: 1.pgm. We read this depth map and save it to depth.
5. Finally, we set the mesh with texture and depth. Now this will be shown to the screen.