Data (part 2)

0. Pitch

• demo:

razvan@himinbjorg:~$ pmap $(pgrep -f chrom | tail -1)
 total       1283528500K

• 1.2 TB of memory (the system only has 64GB RAM and 80 GB of swap)

• demonstrate the larger size of virtual memory and how it works

• demo: allocate 10 TB of memory with malloc

• it's all virtual memory

table/diagram: virtual memory vs physical memory

• physical memory belongs to the system
• physical memory is predefined
• physical memory cannot be accessed directly
• virtual memory belongs to each process - the process virtual address space
• there is a cap of virtual memory per process - but it's a lot
• a new process adds new virtual memory

Why virtual memory?

• each process view the entire memory of its own - simple to use memory: everything starts from zero
• each process is isolated in its own view - security
• can allocate more virtual memory than system memory
• can share memory

1. Recap / Summary of Memory Operations

• diagram: allocation as new area or extending existing area
  • deallocation as shrinking existing area or removing area
• demo: allocation and deallocation: see update of process map: malloc() and mmap()
• demo: malloc size; based on brk and / or mmap syscalls
• diagram: allocator in libc, makes brk and / or mmap syscalls
• introduce process (virtual) address space
  • show by pmap, and by /proc/<PID>/maps

2. Process Virtual Address Space

• diagram with typical zones

• (P)VAS: a list of zones:

  struct zone *pvas_head;
  struct zone {
        unsigned long start;
        unsigned long size;
        unsigned int permissions;
        struct zone *prev, *next;
  };

• allocation: update size of existing zone element or add new zone element

3. Physical Memory and Virtual Memory

• virtual memory vs physical memory for a process
  • demo: ps -o pid,cmd,vsz,rss $$
  • vsz: virtual size
  • rss: resident set size (working set)
• what do we allocate: physical memory, or virtual memory?
  • demo: show step-by-step reservation and virtual and physical size updates

4. Memory Operations

• we call virtual memory allocation: reservation
• we call physical memory allocation: allocation
• diagram: steps are: reserve, allocate, map (associate)
• we can do reserve without allocation

Why is there more virtual memory than physical memory?

• some virtual memory is mapped
• some is shared
• some is only reserved, not mapped - demand paging
• some is swapped

When does reservation happen?

• malloc()
• mmap()

When does allocation happen?

• at access - on demand

• on demand paging (mapping at request)

• at access, page is initialized (with zero) or brought from disk (swap, or file - executable)

• demo: reservation and then allocation

Where is the mapping kept?

• we need a map structure

5. The Page Table

• diagram: virtual memory - page table - physical memory

• an array:

  struct page page_table[N];
  struct page {
      bool valid;
      unsigned int frame_index;
      unsigned int permissions;
      unsigned int flags;
  }

• a page table per process

• N is the index of the virtual page

Why pages?

• easy management: allocate, share, swap
• no fragmentation

Conclusion and Takeaways

• virtual memory is at the core of modern operating systems and applications
• virtual memory is presented as a process virtual address space (per process): a list of zones
• all operations deal with virtual memory, all addresses are virtual addresses
• memory allocation reserves virtual memory
• memory access allocates physical memory and maps it to virtual memory
• virtual memory (and its mapping to physical is managed by a page table: a page table per address space: an array of pages, indexed by the virtual page address