CI testing ZCU102 SMP

[axel-h]

Test with: seL4/seL4_projects_libs#124, seL4/camkes-vm#54

actual PR is seL4/seL4_projects_libs#84, but we need some debug infos

testing seL4/ci-actions#327

[chrisguikema]

Can you add this change to really test the multicore changes?

diff --git a/apps/Arm/vm_minimal/zcu102/2022_1/devices.camkes b/apps/Arm/vm_minimal/zcu102/2022_1/devices.camkes
index 5450e81..7d64d8e 100644
--- a/apps/Arm/vm_minimal/zcu102/2022_1/devices.camkes
+++ b/apps/Arm/vm_minimal/zcu102/2022_1/devices.camkes
@@ -27,7 +27,8 @@ assembly {
             "initrd_addr" : VAR_STRINGIZE(VM_INITRD_ADDR),
             "kernel_entry_addr" : VAR_STRINGIZE(VM_ENTRY_ADDR),
         };
-        vm0.num_vcpus = 4;
+        vm0.num_vcpus = 2;
+        vm0.pcpus = [2, 3];

[chrisguikema]

Sorry, I had a typo in the original diff. It needs to be pcpus. I edited it, but you must have caught it before the edit.

[axel-h]

Sorry, I had a typo in the original diff. It needs to be pcpus. I edited it, but you must have caught it before the edit.

Thanks, I should have spotted this instead of blindly copying the block.

[chrisguikema]

  [    0.056212] psci: failed to boot CPU1 (-22)
  [    0.060054] CPU1: failed to boot: -22

Looks like its failing?

This is what I get locally:

[    0.051504] smp: Bringing up secondary CPUs ...
[    0.056266] Detected VIPT I-cache on CPU1
[    0.056597] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.056738] smp: Brought up 1 node, 2 CPUs
[    0.070429] SMP: Total of 2 processors activated.

[axel-h]

Can you add some more debug output, so we can see what happens?

[chrisguikema]

Here's a seL4_projects_libs PR that adds some debug statements: seL4/seL4_projects_libs#124

And here is what I get with those debug prints:

[    0.051503] smp: Bringing up secondary CPUs ...
handle_psci@psci.c:50 target_cpu: 1
handle_psci@psci.c:53 Null VCPU
handle_psci@psci.c:55 Found free unassigned VCPU
handle_psci@psci.c:57 Started found VCPU
[    0.071825] Detected VIPT I-cache on CPU1
[    0.075670] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.075809] smp: Brought up 1 node, 2 CPUs

[axel-h]

This is what CI shows. Needs more debug output to see what fails ;)

  [    0.051503] smp: Bringing up secondary CPUs ...
  handle_psci@psci.c:50 target_cpu: 1
  handle_psci@psci.c:53 Null VCPU
  handle_psci@psci.c:55 Found free unassigned VCPU
  handle_psci@psci.c:60 Failed to start found VCPU
  [    0.076080] psci: failed to boot CPU1 (-22)
  [    0.076120] CPU1: failed to boot: -22
  [    0.078426] smp: Brought up 1 node, 1 CPU

[chrisguikema]

My guess is that the target_vcpu is returning NULL. All of the function calls in start_vcpu print errors if they return failures. I pushed up another commit to the seL4_projects_libs draft PR.

[    0.051504] smp: Bringing up secondary CPUs ...
handle_psci@psci.c:53 target_cpu: 1
handle_psci@psci.c:56 Null VCPU
handle_psci@psci.c:58 Found free unassigned VCPU: 0x857a60
vcpu_start@vm.c:192 Calling TBC Resume
handle_psci@psci.c:60 Started found VCPU
[    0.076150] Detected VIPT I-cache on CPU1
[    0.079997] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.080138] smp: Brought up 1 node, 2 CPUs

[chrisguikema]

So guess who forgot to change the typo in his own config....

I've replicated the error. Checking into it now.

[chrisguikema]

Okay, I see what's happening now:

The PCPU list is 2 and 3. PSCI is trying to enable (what it thinks it) CPU 1, so we search the PCPU list for 1. When that's not found, we error out.

handle_psci@psci.c:53 target_cpu: 1
vm_vcpu_for_target_cpu@guest_vm_util.h:51 Finding vcpu for target 1
vm_vcpu_for_target_cpu@guest_vm_util.h:53 vcpu 0: target cpu: 2
vm_vcpu_for_target_cpu@guest_vm_util.h:53 vcpu 1: target cpu: 3
handle_psci@psci.c:56 Null VCPU
handle_psci@psci.c:58 Found free unassigned VCPU: 0
handle_psci@psci.c:63 Failed to start found VCPU

[chrisguikema]

@apave24 did make this PR which I think handles this exact situation: seL4/seL4_projects_libs@c95179f

[axel-h]

This seems the PR: seL4/seL4_projects_libs#85