How to set several cameras as active in a declarative way

[ekaradon]

Hello,

So I want to try this: https://doc.babylonjs.com/divingDeeper/cameras/multiViewsPart2
But with react-babylonjs, I don't succeed to find a way to do it.
I can mix both of the approach, for example this works:

function InternalScene() {
  const scene = useScene()
  const canvas = useCanvas()

  if (scene) {
    const camera1 = new BABYLON.ArcRotateCamera(
      'camera1',
      (3 * Math.PI) / 8,
      (3 * Math.PI) / 8,
      15,
      new BABYLON.Vector3(0, 2, 0),
      scene
    )
    camera1.attachControl(canvas, true)

    //camera 2
    const camera2 = new BABYLON.ArcRotateCamera(
      'camera1',
      (5 * Math.PI) / 8,
      (5 * Math.PI) / 8,
      30,
      new BABYLON.Vector3(0, 2, 0),
      scene
    )
    camera2.attachControl(canvas, true)

    // Two Viewports
    camera1.viewport = new BABYLON.Viewport(0, 0.5, 1, 1)
    camera2.viewport = new BABYLON.Viewport(0, 0, 1, 0.5)

    if (scene.activeCameras) {
      scene.activeCameras.push(camera1)
      scene.activeCameras.push(camera2)
    }
  }
  return null
}

function App() {
  return (
    <Engine adaptToDeviceRatio antialias>
      <Scene>
        <InternalScene />
        <hemisphericLight name='light1' intensity={0.7} direction={Vector3.Up()} />
        <SpinningBox name='left' position={Vector3.Zero()} />
        <ground name='ground' position={Vector3.Zero()} width={10} height={10} />
      </Scene>
    </Engine>
  )
}

But it seems, I can't do it with the declarative way?

function InternalScene() {
  const scene = useScene()

  return (
    <>
      <arcRotateCamera
        onCreated={(gizmoCamera) => {
          if (scene) {
            scene.activeCameras = [...(scene.activeCameras || []), gizmoCamera]
            console.log(scene.activeCameras)
          }
        }}
        name='GizmoCamera'
        target={Vector3.Zero()}
        alpha={Math.PI / 2}
        beta={Math.PI / 2}
        upperBetaLimit={Math.PI / 2}
        lowerBetaLimit={Math.PI / 4}
        radius={8}
        inertialRadiusOffset={0.1}
        allowUpsideDown={false}
        lowerRadiusLimit={5}
        upperRadiusLimit={20}
        wheelPrecision={100}
        viewport={new Viewport(0.5, 0, 0.5, 1.0)}
      />
      )
      <arcRotateCamera
        onCreated={(mainCamera) => {
          if (scene) {
            scene.activeCameras = [...(scene.activeCameras || []), mainCamera]
            console.log(scene.activeCameras)
          }
        }}
        name='arcRotateCamera'
        target={Vector3.Zero()}
        alpha={Math.PI / 2}
        beta={Math.PI / 2}
        upperBetaLimit={Math.PI / 2}
        lowerBetaLimit={Math.PI / 4}
        radius={8}
        inertialRadiusOffset={0.1}
        allowUpsideDown={false}
        lowerRadiusLimit={5}
        upperRadiusLimit={20}
        wheelPrecision={100}
        viewport={new Viewport(0, 0, 0.5, 1.0)}
      />
    </>
  )
}

function App() {
  return (
    <Engine adaptToDeviceRatio antialias>
      <Scene>
        <InternalScene />
        <hemisphericLight name='light1' intensity={0.7} direction={Vector3.Up()} />
        <SpinningBox name='left' position={Vector3.Zero()} />
        <ground name='ground' position={Vector3.Zero()} width={10} height={10} />
      </Scene>
    </Engine>
  )
}

Is it possible on a way I didn't see? If it's not, could we add something in order to support it? I'm willing to help if I can but I would need directions first?

[brianzinn]

hi @ekaradon That's a good question. I'm happy to add anything that would be useful or if you could provide a syntax. One thing I have tried to do is stick as close to the API as possible, but in some cases like here it can be useful to add new functionality. It's a bit of work, because I really need to move the specific properties to specific objects whereas many are now global (ie: if new props are added they should be for cameras only).

Here is non-declarative sample that may give you some ideas:
https://github.com/kencyke/react-babylon-multiview/blob/master/src/App.tsx

I feel like we would need multiple properties on cameras to enforce not only active, but add to active cameras (as you have done). Everytime a camera is added it goes through this onMount:
https://github.com/brianzinn/react-babylonjs/blob/master/src/customHosts/CameraLifecycleListener.ts

The other thing that you noticed is that 'Viewport' isn't assigned by the renderer - I can add that to the code generation (there are currently around 200 properties not assigned due to their type). That's because you can dynamically declare how the reconciler can handle some argument types (like Vector3 as any array, for example) - that can only be done if the property type is known and is a cool feature, but comes with a bit of baggage and props that don't assign.

This could be added as an issue, but good idea to discuss the syntax here.

[ekaradon]

Hi @brianzinn,

So I lack of context and knowledge on babylonJS and OpenGL/3D more broadly, but from my point of view, here is what I have imagined:
First, you would have a prop active which could be either a boolean or undefined.
When the property is not assigned, it would behave like now (the last one declared is going to be used?).
When the property is assigned, it would be forcibly removed or to the contrary used?
Then, when you have several cameras declared as active, a check could be made to assess that every one of them use a different viewport and throw an error otherwise? Because, what would even be the meaning to have several active cameras using the same viewport?

So I don't know about multiple properties for activating the camera. What could be the different scenarios where we would need to several properties instead of just one?

For the viewport, we could just declare a prop viewport taking a Viewport Class or undefined like in the example?

[brianzinn]

My reason to say that active may not be enough is because cameras can be added dynamically in the future and 'onMount' only happens once, so maybe you just want the new one to be active and that is already default behaviour. So, there is a difference between active and multiple-active. So far any custom properties have only been needed at time of object creation, but this would maybe require these properties to be known post-creation via likely a camera added observable and I would like to avoid needing anything post-creation.

[ekaradon]

I had a naive approach on this. I was thinking about creating something similar to this:

function Camera(props: CameraProps) {
    const scene = useScene();
    const camera = new BABYLON.Camera()
    const { active } = props;
    useEffect(() => {
        if (active) {
            scene.activeCameras = [...scene.activeCameras, camera]
            return () => (scene.activeCameras = scene.activeCameras.filter(c => c !==camera)
        }
    })
    ...
}

But it does not seem compatible with the approach here and I'm not familiar with it. Is it possible to specify wrappers for some kind of components? Or through the lifecycle class to get access to the props update?

What would you suggest?

[brianzinn]

That looks nearly like a hook - the main issue is that new BABYLON.Camera() will be called on every render and you end up with a bunch of cameras. You can get around that with a function, but it would need access to scene probably via closure:

function useActiveCamera(createCameraFn: () => Camera, active: boolean) {
    const scene = useScene();
    const [camera] = useState<Camera>(createCameraFn);
    useEffect(() => {
        if (active) {
            scene.activeCameras = [...scene.activeCameras, camera]
            return () => {
              scene.activeCameras = scene.activeCameras.filter(c => c !==camera);
              camera.dispose();
            }
        }
    }, [])
    ...
}

The downside of that is that you lose all declarative access to a camera. With something like this you can maybe go declarative (I'm not 100% positive the useEffect will trigger here).

const arcCameraRef = useRef<ArcRotateCamera | null>(null);
const cameraActivated = useRef<boolean>(false);
const scene = useScene();
useEffect(() => {
   if (arcCameraRef.current && !cameraActivated.current) {
     cameraActivated.current = true;
     scene.activeCameras.push(arcCameraRef.current);
   }
}, [arcCameraRef]);

return (
  <arcRotateCamera ref={arcCameraRef} />
 );

I should comment as well that the declarative example in your first post should work, if you set the viewport in the "onCreated" and that is a secondary fix that is needed.

I think that could be handled in the framework with a new prop for Cameras only and it would be useful for others. I don't think your approach is naive - maybe I am overthinking it. If we had a property alwaysAddToActiveCameras then it would probably accomplish what you are after and solve for most cases. I would like to add at the same time a property skipAttachControl to skip attaching the camera to the canvas on creation. I would need to look at the babylon source and write some unit tests, but I would hope that disposing a camera (which is done automatically) would remove it from the active cameras, but otherwise it's easy to add here like you did or I can make a PR upstream.

Also, the reason that I prefer longer descriptive names (like alwaysAddToActiveCameras) is that I try to choose names that won't collide in the future with real properties.

If that looks like it would work then I can add it. What do you think?

[ekaradon]

Well, in the first hook, I think active needs to be added to the dependencies, so if the hook is called with a changed boolean, it actually remove the camera from the actives ones.

The second example won't work as a ref is a constant reference to an object. And if you got the eslint react-hooks/exhaustive-deps, I think it will warn you.

What we could do is return the camera from the hook so we can still use it after that?

function useCamera(createCameraFn: () => Camera, options: { active: boolean }) {
    const scene = useScene();
    const [camera] = useState<Camera>(createCameraFn());
    useEffect(() => {
        if (options.active) {
            scene.activeCameras = [...scene.activeCameras, camera]
            return () => {
              scene.activeCameras = scene.activeCameras.filter(c => c !==camera);
              camera.dispose();
            }
        }
    }, [options.active])
    ...
    return camera;
}

Typescript may not be happy about expanding scene.activeCameras with itself as it starts with a null or undefined property it seems? So we may have to write something like:
scene.activeCameras = [...scene.activeCameras??[], camera]

So the idea would be that if the user wants to custom the setup of its camera (for unsupported API), he would then have to use this hook? And we could add more and more options as we see fit?

skipAttachControl sounds like a good name for the prop in my opinion. I like it.
Not so sure with alwaysAddToActiveCameras. In my opinion, cameras can be deactivated and reactivated at will (for example displaying a map on a game, another view in a 3D editor, a preview, ...), so this kind of prop really look like something that need to be listened during the component lifetime.
For example, with an editor similar to unreal-engine, we like to be able to switch from top, bottom, left, right, front and back view. So it's more like a state than an initial property for the construction process?

[brianzinn]

I had another idea, which was to update the LifecycleListener to handle custom property updates. It would have off to there instead of using the other prop handlers.

[brianzinn]

I have put more thought into the declarative syntax following a recent issue created for multiple scene and canvas support.

What about a syntax like this:

<Scene ...>
  <arcRotateCamera ..>
    <viewport x={0.5} y={0} height={0.5} width={1} />
  </arcRotateCamera>
  <arcRotateCamera ..>
    <viewport x={0} y={0} height={0.5} width={1} />
  </arcRotateCamera>  
</Scene>

If that was connected automatically then what is remaining is how to set the activeCameras declaratively.

[ekaradon]

Sounds good. Just asking, why the camera is the parent and not the viewport?
It would not have been my first intuition but after thinking about it, it seems to be the right choice. For example this kind of thing would be easy to set with this approach: BabylonJS/Babylon.js#10119
And if the developer wants to have several camera for the same viewport, he could just declare several camera sharing the same viewport. 👍

[brianzinn]

@ekaradon Although I have not addressed your question on how to set several cameras as active - did manage to make partial progress by adding <viewport />. You can now do something like this:

const size = 2;
const shade = 0;

const BoxWithArrows = ({ position }) => {
  return (<transformNode position={position} name="transform-node">
    <lines
      name="red-line"
      points={
        [
          new Vector3.Zero(),
          new Vector3(size, 0, 0),
          new Vector3(size * 0.95, 0.05 * size, 0),
          new Vector3(size, 0, 0),
          new Vector3(size * 0.95, -0.05 * size, 0)
        ]
      }
      color={new Color3(1, shade, shade)}
    />
    <lines
      name="green-line"
      points={
        [
          new Vector3.Zero(),
          new Vector3(0, size, 0),
          new Vector3(-0.05 * size, size * 0.95, 0),
          new Vector3(0, size, 0),
          new Vector3(0.05 * size, size * 0.95, 0)
        ]
      }
      color={new Color3(shade, 1, shade)}
    />
    <lines
      name="blue-line"
      points={
        [
          new Vector3.Zero(),
          new Vector3(0, 0, size),
          new Vector3(0, -0.05 * size, size * 0.95),
          new Vector3(0, 0, size),
          new Vector3(0, 0.05 * size, size * 0.95)
        ]
      }
      color={new Color3(shade, shade, 1)}
    />
    <box
      name="color-box"
      faceColors={[
        Color3.Blue(),
        Color3.Red(),
        Color3.Green(),
        Color3.White(),
        Color3.Yellow(),
        Color3.Black()
      ]}
    ></box>
  </transformNode>)
}

const MultiViewport = () => {
  const scene = useScene();

  const alwaysActive = (camera) => {
    scene.activeCameras = [...(scene.activeCameras || []), camera];
  }
  return (
    <>
      <BoxWithArrows position={Vector3.Zero()} />

      <hemisphericLight name='light1' intensity={0.7} direction={Vector3.Up()} />

      <arcRotateCamera name='camera1' alpha={3 * Math.PI / 8} beta={3 * Math.PI / 8} radius={15} target={new Vector3(0, 2, 0)} onCreated={alwaysActive}>
        <viewport x={0} y={0.5} height={1} width={1} />
      </arcRotateCamera>

      <arcRotateCamera name='camera2' alpha={5 * Math.PI / 8} beta={5 * Math.PI / 8} radius={30} target={new Vector3(0, 2, 0)} onCreated={alwaysActive}>
        <viewport x={0} y={0} height={1} width={0.5} />
      </arcRotateCamera>
    </>
  )
}

It looks to have all of the functionality of the multiview playground from docs:
https://doc.babylonjs.com/divingDeeper/cameras/multiViewsPart2

One thing also of interest is your useEffect to remove camera from activeCameras on unmount. I think I can handle that in the "unmount" of the lifecycle listener always using a filter like you did.

Then to get back to your original question we could add an active prop (or something more descriptive like addToActiveCameras that also avoids future conflicts) to cameras that would always push that camera to activeCameras of the scene. You can give <viewport /> a try with v3.0.17 that was just published.