dune: exit due to EPT violation

[steven-chien]

Hello, I am trying to get ZygOS up and running, using exactly the environment specified in the paper (Ubuntu 16.04.7 Xeniel, Kernel 4.11). We made the following changes:

• to read the syscall symbole table address from /proc/kallsyms when compiling dune because of Linux address randomization since 4.8.
• Added PCI device code to the i40e driver DPDK so that our NIC is supported (XXV710)

We are able to set up the dependencies and load the kernel modules, as well as bind the NIC to DPDK. However, when we run the example echoserver like in the IX README, we get

...
CPU 00| 18:14:53 <4>: eth:	link up - speed 100 Mbps, half-duplex

CPU 00| 18:14:53 <4>: init: barrier after al CPU initialization
CPU 00| 18:14:53 <4>: init: module hw         SUCESS
CPU 00| 18:14:53 <4>: load_elf(/lib64/ld-linux-x86-64.so.2)
CPU 00| 18:14:53 <4>: load_elf(): /lib64/ld-linux-x86-64.so.2
CPU 00| 18:14:53 <4>: dune_elf_iter_ph(): /lib64/ld-linux-x86-64.so.2
CPU 00| 18:14:53 <2>: sandbox: loading segment - va 0x000000000, len 253b0
CPU 00| 18:14:53 <2>: sandbox: loading segment - va 0x000225bc0, len 15a8
dune: exit due to EPT violation

It appears the violation/segfault is coming from setup_arguments in the sandbox:

https://github.com/ix-project/zygos/blob/master/dp/sandbox/main.c#L260

Do you have any ideas about this? Also, the AUX_NUM is hardcoded to 14, which is different from the auxvec.h in Xeinal, is that intentional?

[prekageo]

How large is your environment? wc -c /proc/self/environ

What happens if you execute IX with env -i ...?

[steven-chien]

Thanks for the quick response!

How large is your environment? wc -c /proc/self/environ

1927 /proc/self/environ

What happens if you execute IX with env -i ...?

root@n05:~/zygos# env -i ./dp/ix -- ./apps/echoserver 4
13:57:49 <4>: init: starting IX
13:57:49 <4>: init: cpu phase
13:57:49 <4>: init: module Dune       SUCESS
13:57:49 <4>: cpu: detected 4 cores
mem_alloc_pages(1, 2097152, NULL, 1)=-2097152
mempool_datastore: runners         pages:   1 elem_len:  24 nostraddle:0 chunk_size:128 num_chunks:4682 [0x3fffe00000-0x4000000000]
13:57:49 <4>: init: module CPU        SUCESS
13:57:50 <4>: timer: detected 3092 ticks per US
13:57:50 <4>: init: module timer      SUCESS
mem_alloc_pages(1, 2097152, NULL, 1)=-4194304
mempool_datastore: pending_pkt     pages:   1 elem_len:  56 nostraddle:0 chunk_size:128 num_chunks:4292 [0x3fffc00000-0x3fffe00000]
mem_alloc_pages(3, 2097152, NULL, 1)=-10485760
mempool_datastore: arp             pages:   3 elem_len:  80 nostraddle:0 chunk_size:128 num_chunks:4614 [0x3fff600000-0x3fffc00000]
13:57:50 <4>: init: module net        SUCESS
13:57:50 <4>: using config :'./ix.conf'
13:57:50 <4>: no static arp entries defined in config
13:57:50 <4>: net: using the following configuration:
13:57:50 <4>: 	host IP:	192.168.11.164
13:57:50 <4>: 	broadcast IP:	192.168.11.255
13:57:50 <4>: 	gateway IP:	192.168.11.1
13:57:50 <4>: 	subnet mask:	255.255.255.0
13:57:50 <4>: init: module cfg        SUCESS
13:57:50 <4>: init: module cp         SUCESS
13:57:50 <4>: pci: created device 0000:01:00.0, NUMA node -1
13:57:50 <4>: pci:	IOMEM - base a2000000, len 1000000
13:57:50 <4>: pci:	IOMEM - base a3008000, len 8000
13:57:50 <4>: init: module pci        SUCESS
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 4 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x50c00000 bytes
EAL: Virtual area found at 0x7fa31c400000 (size = 0x50c00000)
EAL: Ask a virtual area of 0xa6200000 bytes
EAL: Virtual area found at 0x7fa276000000 (size = 0xa6200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fa275c00000 (size = 0x200000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7fa275600000 (size = 0x400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fa275200000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fa274e00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fa274a00000 (size = 0x200000)
EAL: Ask a virtual area of 0x1400000 bytes
EAL: Virtual area found at 0x7fa273400000 (size = 0x1400000)
EAL: Ask a virtual area of 0x600000 bytes
EAL: Virtual area found at 0x7fa272c00000 (size = 0x600000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7fa272600000 (size = 0x400000)
EAL: Ask a virtual area of 0x1a00000 bytes
EAL: Virtual area found at 0x7fa270a00000 (size = 0x1a00000)
EAL: Ask a virtual area of 0x1800000 bytes
EAL: Virtual area found at 0x7fa26f000000 (size = 0x1800000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fa26ec00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fa26e800000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fa26e400000 (size = 0x200000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7fa26de00000 (size = 0x400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fa26da00000 (size = 0x200000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7fa26d400000 (size = 0x400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fa26d000000 (size = 0x200000)
EAL: Ask a virtual area of 0xa00000 bytes
EAL: Virtual area found at 0x7fa26c400000 (size = 0xa00000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7fa26be00000 (size = 0x400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fa26ba00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fa26b600000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fa26b200000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fa26ae00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fa26aa00000 (size = 0x200000)
EAL: Requesting 74 pages of size 2MB from socket 0
EAL: TSC frequency is ~3092855 KHz
EAL: Master lcore 0 is ready (tid=7075f8c0;cpuset=[0])
EAL: PCI device 0000:01:00.0 on NUMA socket -1
EAL:   probe driver: 8086:158b rte_i40e_pmd
EAL:   PCI memory mapped at 0x7fa325800000
EAL:   PCI memory mapped at 0x7fa326800000
PMD: eth_i40e_dev_init(): FW 6.81 API 1.7 NVM 06.08.00 eetrack 80003d17
PMD: i40e_dcb_init_configure(): Failed to stop lldp
EAL: PCI device 0000:01:00.1 on NUMA socket -1
EAL:   probe driver: 8086:158b rte_i40e_pmd
EAL:   PCI memory mapped at 0x7fa326808000
EAL:   PCI memory mapped at 0x7fa327808000
PMD: eth_i40e_dev_init(): FW 6.81 API 1.7 NVM 06.08.00 eetrack 80003d17
PMD: i40e_dcb_init_configure(): Failed to stop lldp
13:57:50 <4>: init: module dpdk       SUCESS
13:57:50 <4>: init: module ethdev     SUCESS
CPU 00| 13:57:50 <4>: cpu: started core 0, numa node 0, apicid 0
CPU 00| 13:57:50 <4>: init: module firstcpu   SUCESS
mempool_datastore: mbuf            pages: 793 elem_len:2112 nostraddle:1 chunk_size:128 num_chunks:46145 [0x4000000000-0x4063200000]
CPU 00| 13:57:50 <4>: init: module mbuf       SUCESS
mem_alloc_pages(21, 2097152, NULL, 1)=-56623104
mempool_datastore: pbuf            pages:  21 elem_len:  56 nostraddle:0 chunk_size:128 num_chunks:46144 [0x3ffca00000-0x3fff400000]
mem_alloc_pages(512, 2097152, NULL, 1)=-1130364928
mempool_datastore: pbuf_payload    pages: 512 elem_len:4096 nostraddle:0 chunk_size:128 num_chunks:42048 [0x3fbca00000-0x3ffca00000]
mem_alloc_pages(34, 2097152, NULL, 1)=-1201668096
mempool_datastore: tcp_pcb         pages:  34 elem_len: 272 nostraddle:0 chunk_size:128 num_chunks:42048 [0x3fb8600000-0x3fbca00000]
mem_alloc_pages(4, 2097152, NULL, 1)=-1210056704
mempool_datastore: tcp_seg         pages:   4 elem_len:  32 nostraddle:0 chunk_size:128 num_chunks:42048 [0x3fb7e00000-0x3fb8600000]
CPU 00| 13:57:50 <4>: init: module memp       SUCESS
mem_alloc_pages(30, 2097152, NULL, 1)=-1272971264
mempool_datastore: pcb             pages:  30 elem_len: 120 nostraddle:0 chunk_size:128 num_chunks:44096 [0x3fb4200000-0x3fb7e00000]
mempool_datastore: ip              pages:   4 elem_len:  16 nostraddle:1 chunk_size:128 num_chunks:44096 [0x4063200000-0x4063a00000]
CPU 00| 13:57:50 <4>: init: module tcpapi     SUCESS
CPU 00| 13:57:50 <4>: init: percpu phase 0
CPU 00| 13:57:50 <4>: init: module timer      on 0: SUCESS 
CPU 00| 13:57:50 <4>: init: module mbuf       on 0: SUCESS 
CPU 00| 13:57:50 <4>: init: module memp       on 0: SUCESS 
CPU 00| 13:57:50 <4>: init: module tcpapi     on 0: SUCESS 
CPU 00| 13:57:50 <4>: init: module migration  on 0: SUCESS 
CPU 00| 13:57:50 <4>: init: module syscall    on 0: SUCESS 
PMD: i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
CPU 00| 13:57:50 <2>: queue_setup : phys 0x366000100
PMD: i40e_set_tx_function_flag(): Vector tx can be enabled on this txq.
CPU 00| 13:57:50 <4>: init: module init-net   on 0: SUCESS 
CPU 00| 13:57:50 <4>: init: CPU 0 ready
init:CPU 0 ready
PMD: i40e_set_tx_function(): Vector tx finally be used.
PMD: i40e_pf_config_rss(): Max of contiguous 1 PF queues are configured
PMD: i40e_set_rx_function(): Port[0] doesn't meet Vector Rx preconditions
PMD: i40e_set_rx_function(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0.
CPU 00| 13:57:50 <4>: eth: started an ethernet device
CPU 00| 13:57:50 <4>: eth:	MAC address: 3C:FD:FE:E5:A4:D0
CPU 00| 13:57:51 <4>: eth:	link up - speed 100 Mbps, half-duplex

CPU 00| 13:57:51 <4>: init: barrier after al CPU initialization
CPU 00| 13:57:51 <4>: init: module hw         SUCESS
CPU 00| 13:57:51 <4>: load_elf(/lib64/ld-linux-x86-64.so.2)
CPU 00| 13:57:51 <4>: load_elf(): /lib64/ld-linux-x86-64.so.2
CPU 00| 13:57:51 <4>: dune_elf_iter_ph(): /lib64/ld-linux-x86-64.so.2
CPU 00| 13:57:51 <2>: sandbox: loading segment - va 0x000000000, len 253b0
CPU 00| 13:57:51 <2>: sandbox: loading segment - va 0x000225bc0, len 15a8
CPU 00| 13:57:51 <4>: dune_elf_iter_ph(): done
CPU 00| 13:57:51 <4>: dune_elf_close(): /lib64/ld-linux-x86-64.so.2
CPU 00| 13:57:51 <4>: load_elf() ret=0
CPU 00| 13:57:51 <4>: setup_arguments() loader path = /lib64/ld-linux-x86-64.so.2
CPU 00| 13:57:51 <4>: setup_arguments() argv=0x7ffd09cc5888, argc=2, tmp=20, len=20
CPU 00| 13:57:51 <4>: setup_arguments() envv=0x7ffd09cc58a0, envc=0, tmp=0, len=20
CPU 00| 13:57:51 <4>: setup_arguments() strlen(/lib64/ld-linux-x86-64.so.2)+1=28, tmp=0, len=48
CPU 00| 13:57:51 <4>: setup_arguments() sp=545460846592 len=48 data_ptr=0x7effffffd0
CPU 00| 13:57:51 <4>: setup_arguments() aux_ptr=0x7ffd09cc58a8
CPU 00| 13:57:51 <4>: setup_arguments() aux_ptr=0x7ffd09cc58a8, len=320
CPU 00| 13:57:51 <4>: setup_arguments() arg_ptr=0x7efffffec0, len=320
dune: exit due to EPT violation
root@n05:~/zygos# 

[prekageo]

Can you copy paste the last few lines from dmesg as well?

[steven-chien]

[  844.679255] vmx: created VCPU (VPID 1)
[  844.984182] ept: failed to get user page 7f543ffffec0
[  844.989882] vmx: page fault failure GPA: 0x7efffffec0, GVA: 0x7efffffec0
[  844.997407] vmx: --- Begin VCPU Dump ---
[  844.997407] vmx: CPU 0 VPID 1
[  844.997408] vmx: RIP 0x000000007002222b RFLAGS 0x00010202
[  844.997408] vmx: RAX 0x0000000000000003 RCX 0x000000007fffffb9
[  844.997409] vmx: RDX 0x00007f54f38de780 RBX 0x0000000000000002
[  844.997409] vmx: RSP 0x00007ffe39e3ed70 RBP 0x00007ffe39e3efc8
[  844.997410] vmx: RSI 0x00000000700b2591 RDI 0x0000000000000004
[  844.997410] vmx: R8  0x0000000000000000 R9  0x0000000000000046
[  844.997410] vmx: R10 0x0000000000000046 R11 0x0000000000000000
[  844.997411] vmx: R12 0x00007ffe39e3efe0 R13 0x0000007efffffed0
[  844.997411] vmx: R14 0x0000000000000000 R15 0x00007ffe39e3efe8
[  844.997412] vmx: Dumping Stack Contents...
[  844.997412] vmx: RSP+0  0x0007000000000000
[  844.997413] vmx: RSP+8  0x0000007efffffec0
[  844.997413] vmx: RSP+16 0x000000006f000040
[  844.997413] vmx: RSP+24 0x000000006f000c30
[  844.997414] vmx: RSP+32 0x0000000000000008
[  844.997414] vmx: RSP+40 0x0000000000000003
[  844.997414] vmx: RSP+48 0x0000007effffffd0
[  844.997415] vmx: RSP+56 0x000000000000002f
[  844.997415] vmx: RSP+64 0x00007f54f38dc0a4
[  844.997415] vmx: RSP+72 0x0000007f00000000
[  844.997416] vmx: RSP+80 0x0000000000000010
[  844.997416] vmx: RSP+88 0x0000000000000000
[  844.997416] vmx: --- End VCPU Dump ---
[  844.997417] vmx: stopping VCPU (VPID 1)
[  845.002865] vmx: destroying VCPU (VPID 1)

[steven-chien]

@prekageo I did a few more tests and it seems the issue is coming that the stack space from mmap cannot be accessed.

I added the following to dp/sandbox/main.c just to test if I can deference addresses on the stack.

...
        ret = umm_alloc_stack(&sp);
        if (ret) {
                log_err("sandbox: failed to alloc stack\n");
                return ret;
        }
uintptr_t *ptr = (uintptr_t*)(sp - 8384512);
log_info("umm_alloc_stack stack start: %p\n", ptr);
log_info("umm_alloc_stack: *ptr=%x\n", *ptr);

        sp = setup_arguments(sp, CFG.loader_path, &argv[0], environ, data);
...

Result:

...
ummap_mmap_anom_flags: mem=0x7eff801000 , addr=0x7eff801000 , len=8384512
CPU 00| 17:51:03 <4>: umm_alloc_stack stack start: 0x7eff801000
dune: exit due to EPT violation

If I immediately dereference the address returned by mmap in ummap_mmap_anom_flags, before sending the mapping to Dune, it will also fail immediately as well.

...
        mem = mmap(addr, len, prot, flags, -1, 0);
        if (mem != addr)
                return -errno;
char *ptr =  (char*)mem;
printf("ummap_mmap_anom_flags: mem=%p , addr=%p , len=%ld\n", mem, addr, len);
printf("ummap_mmap_anom_flags: *ptr=%x\n", *ptr);


        ret = dune_vm_map_phys(pgroot, addr, len,
                               (void *) dune_va_to_pa(addr),
                               perm);
        if (ret) {
                munmap(addr, len);
                return ret;
        }
...

ummap_mmap_anom_flags: mem=0x7eff801000 , addr=0x7eff801000 , len=8384512
dune: exit due to EPT violation

[prekageo]

I agree but I am not sure why. I would start my changing the pr_debug to pr_err in https://github.com/ix-project/dune/blob/e0b4d9aab29788b26a46f9f2461588eea231938c/kern/ept.c#L553 and then adding some debugging pr_err in ept_set_epte to understand where the failure happens.