The 4th Secondary Server Always Crash, version 0.9.15

[lygbuaa]

I am working with carla-0.9.15 on a 4*RTX3090 workstation, after the document https://carla.readthedocs.io/en/latest/adv_multigpu
Here is my steps:

1. start primary server: bash CarlaUE4.sh -quality-level=low -fps=30 -windowed -Resx=600 -Resy=480 -nullrhi
2. start the 1st secondary server: bash CarlaUE4.sh -quality-level=low -fps=30 -windowed -Resx=600 -Resy=480 -RenderOffscreen -carla-rpc-port=13000 -carla-primary-host=127.0.0.1 -carla-primary-port=2002 -ini:[/Script/Engine.RendererSettings]:r.GraphicsAdapter=0
3. start the 2nd secondary server: bash CarlaUE4.sh -quality-level=low -fps=30 -windowed -RenderOffscreen -RenderOffscreen -carla-rpc-port=14000 -carla-primary-host=127.0.0.1 -carla-primary-port=2002 -ini:[/Script/Engine.RendererSettings]:r.GraphicsAdapter=1
4. start the 3rd secondary server: bash CarlaUE4.sh -quality-level=low -fps=30 -windowed -Resx=600 -Resy=480 -RenderOffscreen -carla-rpc-port=15000 -carla-primary-host=127.0.0.1 -carla-primary-port=2002 -ini:[/Script/Engine.RendererSettings]:r.GraphicsAdapter=2
5. start the 4th secondary server: bash CarlaUE4.sh -quality-level=low -fps=30 -windowed -Resx=600 -Resy=480 -RenderOffscreen -carla-rpc-port=16000 -carla-primary-host=127.0.0.1 -carla-primary-port=2002 -ini:[/Script/Engine.RendererSettings]:r.GraphicsAdapter=3

The primary server and 1~3 secondary servers are working fine. But the 4th secondary server always crash down, the logs are below:

=2002 -ini:[/Script/Engine.RendererSettings]:r.GraphicsAdapter=3
4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
CommonUnixCrashHandler: Signal=11
Malloc Size=131160 LargeMemoryPoolOffset=196744 
Malloc Size=131160 LargeMemoryPoolOffset=327928 
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
CarlaUE4.sh: line 5: 232377 Segmentation fault      (core dumped) "$UE4_PROJECT_ROOT/CarlaUE4/Binaries/Linux/CarlaUE4-Linux-Shipping" CarlaUE4 "$@"

My host machine is ubuntu 22.04.1, nvidia-dkms-535， cuda 12.2, Vulkan Instance Version: 1.3.204， Vulkan apiVersion 1.3.242.
Any suggestions?

[Blyron]

Hi!

Could you send the information of nvidia-smi command?

[PatrickPromitzer]

5. start the 4th secondary server: `bash CarlaUE4.sh  -quality-level=low -fps=30 -windowed -Resx=600 -Resy=480 -RenderOffscreen  -carla-rpc-port=16000 -carla-primary-host=127.0.0.1 -carla-primary-port=2002 -ini:[/Script/Engine.RendererSettings]:r.GraphicsAdapter=3`

From the link you posted:

-ini:[/Script/Engine.RendererSettings]:r.GraphicsAdapter: tells which GPU device to use for this secondary server

./CarlaUE4.sh -carla-rpc-port=4000 -carla-primary-host=127.0.0.1 -carla-primary-port=2002 -ini:[/Script/Engine.RendererSettings]:r.GraphicsAdapter=1

This secondary server will use port 4000 as the RPC server to avoid conflicts with other ports and 
will connect to the primary server located at IP 127.0.0.1 in the port 2002, 
and also this server will use the GPU device 1.

It looks like, you only have 3 GPU devices or the 4th can't be used with Carla for some reasons.