Integer overflow in `mmap` when using high addresses

[nuczyc]

Describe the bug

A kernel panic occurs in RuxOS when the mmap system call is invoked with a memory address near the end of the virtual address space. The panic is caused by an unhandled integer overflow in the internal utility function find_free_region.

ruxos/api/ruxos_posix_api/src/imp/mmap/utils.rs

Line 150 in 32a8fe8

let end = start + len;

To Reproduce

1. Compile the program and run.

#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <unistd.h>
#include <stdint.h>

/*
 * PoC for RuxOS kernel panic in find_free_region function
 * 
 * The crash occurs in ruxos_posix_api/src/imp/mmap/utils.rs:111
 * when calling unwrap_or() on the result of peek_next() from vma_map.
 * 
 * This function is part of the mmap implementation and handles finding
 * free virtual memory regions for memory mapping operations.
 * 
 * The panic likely occurs when:
 * 1. A mmap call is made with a specific address near VMA_END
 * 2. The vma_map.upper_bound() call returns an iterator
 * 3. peek_next() on that iterator returns None
 * 4. The unwrap_or() call panics (despite having a default value)
 * 
 * This PoC attempts to trigger this by:
 * 1. Getting the page size
 * 2. Attempting to mmap at a very high address (near memory limit)
 * 3. Using MAP_FIXED to force the address
 * 4. This should exercise the find_free_region code path
 */

int main() {
    size_t page_size = sysconf(_SC_PAGESIZE);
    if (page_size == (size_t)-1) {
        perror("sysconf");
        return 1;
    }

    printf("Page size: %zu\n", page_size);
    
    // Try to map at a very high address to trigger the edge case
    // This should exercise the find_free_region function with addr=None path
    // and potentially hit the unwrap_or() panic when peek_next() returns None
    
    // Use a high address that might be near VMA_END
    // On 64-bit systems, we'll use something in the upper half
    void* hint = (void*)(0x7fffff000000UL);
    size_t length = page_size;
    
    printf("Attempting mmap at address %p with length %zu\n", hint, length);
    
    // First try without MAP_FIXED to let the kernel choose
    void* ptr1 = mmap(hint, length, PROT_READ | PROT_WRITE, 
                      MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    
    if (ptr1 == MAP_FAILED) {
        perror("mmap (without MAP_FIXED) failed");
    } else {
        printf("mmap (without MAP_FIXED) succeeded at %p\n", ptr1);
        munmap(ptr1, length);
    }
    
    // Now try with MAP_FIXED to force the address
    // This should trigger the find_free_region function with the specific addr path
    printf("Attempting mmap with MAP_FIXED at address %p\n", hint);
    
    void* ptr2 = mmap(hint, length, PROT_READ | PROT_WRITE, 
                      MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0);
    
    if (ptr2 == MAP_FAILED) {
        perror("mmap (with MAP_FIXED) failed");
    } else {
        printf("mmap (with MAP_FIXED) succeeded at %p\n", ptr2);
        munmap(ptr2, length);
    }
    
    // Try multiple mappings to potentially fill up the VMA map
    // and create conditions where peek_next() might return None
    for (int i = 0; i < 10; i++) {
        void* addr = (void*)(0x7fffff000000UL + (i * page_size * 16));
        void* ptr = mmap(addr, page_size, PROT_READ | PROT_WRITE,
                         MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0);
        if (ptr != MAP_FAILED) {
            printf("Mapped at %p\n", ptr);
        }
    }
    
    // Try one more at the edge to potentially trigger the panic
    void* edge_addr = (void*)(0xfffffffffffff000UL);
    void* ptr3 = mmap(edge_addr, page_size, PROT_READ | PROT_WRITE,
                      MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0);
    
    if (ptr3 == MAP_FAILED) {
        perror("mmap at edge address failed");
    } else {
        printf("mmap at edge address succeeded at %p\n", ptr3);
        munmap(ptr3, page_size);
    }
    
    printf("PoC completed. If the kernel is vulnerable, it should have panicked.\n");
    return 0;
}

2.features.txt

alloc
paging
net
multitask
irq
fs

Environment

• RuxOS version:[Merge pull request #219 from syswonder/dev · syswonder/ruxos@32a8fe8](32a8fe8)

Logs

SeaBIOS (version 1.16.3-debian-1.16.3-2)


iPXE (https://ipxe.org) 00:03.0 CA00 PCI2.10 PnP PMM+7EFCAA40+7EF0AA40 CA00
                                                                               


Booting from ROM..
Initialize IDT & GDT...

8888888b.                     .d88888b.   .d8888b.
888   Y88b                   d88P" "Y88b d88P  Y88b
888    888                   888     888 Y88b.
888   d88P 888  888 888  888 888     888  "Y888b.
8888888P"  888  888 `Y8bd8P' 888     888     "Y88b.
888 T88b   888  888   X88K   888     888       "888
888  T88b  Y88b 888 .d8""8b. Y88b. .d88P Y88b  d88P
888   T88b  "Y88888 888  888  "Y88888P"   "Y8888P"

arch = x86_64
platform = x86_64-qemu-q35
target = x86_64-unknown-none
smp = 1
build_mode = debug
log_level = warn

[  0.186234 0 axfs_ramfs::dir:68] AlreadyExists sys
Page size: 4096
Attempting mmap at address 0x7fffff000000 with length 4096
mmap (without MAP_FIXED) succeeded at 0xffff800000000000
Attempting mmap with MAP_FIXED at address 0x7fffff000000
mmap (with MAP_FIXED) failed: Out of memory
[  0.188017 0:1 ruxruntime::lang_items:14] panicked at api/ruxos_posix_api/src/imp/mmap/utils.rs:150:15:
attempt to add with overflow