RFC: Migrate to Hyper-V Nested VM Architecture for Fast Boot & Cost Optimization

[louis030195]

Summary

Migrate from individual Azure VMs to a Hyper-V nested VM architecture where larger Azure VMs ("hosts") run multiple nested VMs. This enables:

• 10x faster boot: 2-5 seconds (resume from saved state) vs 30-60 seconds (cold boot)
• ~60% cost reduction: Pack 8-10 VMs on one host vs paying per individual VM
• Better isolation: Reset to clean checkpoint between runs
• Instant availability: Pre-warmed VM pools ready to acquire

---

Current Architecture

Vercel (mediar-web-app)
  /api/admin/machines/provision -> Azure SDK
  /api/machines/{id}/start|stop -> Azure SDK
  /api/remote-workflows/{id}/execute -> MCP endpoint
                    |
                    v
    +-----------+ +-----------+ +-----------+
    | Azure VM  | | Azure VM  | | Azure VM  |
    | D4s_v3    | | D4s_v3    | | D4s_v3    |
    | $140/mo   | | $140/mo   | | $140/mo   |
    | Terminator| | Terminator| | Terminator|
    +-----------+ +-----------+ +-----------+
          30-60s cold boot each

Problems:

• Slow boot (30-60s) hurts UX for on-demand executions
• Expensive at scale ($140/mo per idle VM)
• Dirty state between runs (no clean reset)
• No pooling/pre-warming capability

---

Proposed Architecture

Vercel (mediar-web-app)
  /api/hosts -> manage Hyper-V host VMs
  /api/pools -> manage VM pools
  /api/vms/acquire|release -> fast VM checkout
                    |
                    v
+-----------------------------------------------------+
| Azure VM "Host" (D8s_v5 - $280/mo)                  |
| +----------+ +----------+ +----------+ +----------+ |
| |Nested VM | |Nested VM | |Nested VM | |Nested VM | |
| | (saved)  | | (saved)  | | (running)| | (saved)  | |
| | 2GB RAM  | | 2GB RAM  | | 2GB RAM  | | 2GB RAM  | |
| +----------+ +----------+ +----------+ +----------+ |
|                                                     |
|   Hyper-V + Host Agent (orchestrator)               |
|   Base VHDX template with Terminator pre-installed  |
+-----------------------------------------------------+
        2-5s resume | $280/mo / 8 VMs = $35/VM

---

Detailed Implementation Plan

Phase 1: Infrastructure & Packer Images

1.1 Create Hyper-V Host Image

Update Packer config to build a host image with:

# packer/azure-hyperv-host.pkr.hcl

source "azure-arm" "hyperv-host" {
  vm_size = "Standard_D8s_v5"  # 8 vCPU, 32GB RAM, nested virt support
}

build {
  provisioner "powershell" {
    scripts = [
      "scripts/enable-hyperv.ps1",
      "scripts/install-host-agent.ps1",
      "scripts/configure-networking.ps1"
    ]
  }
}

scripts/enable-hyperv.ps1:

# Enable Hyper-V and management tools
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All -NoRestart
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-Management-PowerShell -All -NoRestart

# Create internal switch for nested VMs
New-VMSwitch -Name "NestedSwitch" -SwitchType Internal
# Configure NAT for internet access
New-NetNat -Name "NestedNAT" -InternalIPInterfaceAddressPrefix "192.168.100.0/24"

1.2 Create Nested VM Template Image

Separate Packer build for the nested VM template (outputs VHDX).

1.3 Host Agent (Orchestrator)

Minimal HTTP agent running on each host:

// host-agent/index.ts (Bun)
import { $ } from "bun";

Bun.serve({
  port: 8081,
  async fetch(req) {
    const op = await req.json();

    switch (op.action) {
      case "list":
        const vms = await $`powershell -Command "Get-VM | ConvertTo-Json"`.json();
        return Response.json({ vms });
      case "create":
        await $`powershell -Command "New-VHD -ParentPath '${op.templatePath}' -Path '${op.vhdxPath}' -Differencing"`;
        await $`powershell -Command "New-VM -Name '${op.vmName}' -MemoryStartupBytes 2GB -VHDPath '${op.vhdxPath}'"`;
        return Response.json({ success: true });
      case "start":
        await $`powershell -Command "Start-VM -Name '${op.vmName}'"`;
        return Response.json({ success: true });
      case "save":
        await $`powershell -Command "Save-VM -Name '${op.vmName}'"`;
        return Response.json({ success: true });
      case "reset":
        await $`powershell -Command "Restore-VMCheckpoint -VMName '${op.vmName}' -Name 'clean' -Confirm:$false"`;
        return Response.json({ success: true });
      // ... etc
    }
  },
});

---

Phase 2: Database Schema Changes

2.1 New Tables

-- Hyper-V host VMs (the Azure VMs running Hyper-V)
CREATE TABLE hyperv_hosts (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  name TEXT NOT NULL,
  azure_resource_id TEXT UNIQUE NOT NULL,
  public_ip TEXT,
  agent_endpoint TEXT,  -- http://{ip}:8081
  total_memory_mb INTEGER NOT NULL DEFAULT 32768,
  total_cpus INTEGER NOT NULL DEFAULT 8,
  max_nested_vms INTEGER NOT NULL DEFAULT 10,
  status TEXT NOT NULL DEFAULT 'provisioning'
    CHECK (status IN ('provisioning', 'active', 'draining', 'offline', 'failed')),
  power_state TEXT,
  region TEXT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- VM templates (golden images)
CREATE TABLE vm_templates (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  name TEXT NOT NULL UNIQUE,
  vhdx_path TEXT NOT NULL,
  memory_mb INTEGER NOT NULL DEFAULT 2048,
  cpu_count INTEGER NOT NULL DEFAULT 2,
  gpu_enabled BOOLEAN NOT NULL DEFAULT false,
  description TEXT,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- VM pools (groups of pre-warmed VMs)
CREATE TABLE vm_pools (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  name TEXT NOT NULL UNIQUE,
  template_id UUID NOT NULL REFERENCES vm_templates(id),
  desired_count INTEGER NOT NULL DEFAULT 5,
  min_ready INTEGER NOT NULL DEFAULT 2,
  max_count INTEGER NOT NULL DEFAULT 20,
  organization_id TEXT,  -- NULL = global pool
  status TEXT NOT NULL DEFAULT 'active'
    CHECK (status IN ('active', 'paused', 'draining')),
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Individual nested VMs
CREATE TABLE nested_vms (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  name TEXT NOT NULL,
  host_id UUID NOT NULL REFERENCES hyperv_hosts(id) ON DELETE CASCADE,
  pool_id UUID NOT NULL REFERENCES vm_pools(id),
  template_id UUID NOT NULL REFERENCES vm_templates(id),
  state TEXT NOT NULL DEFAULT 'creating'
    CHECK (state IN ('creating', 'off', 'saved', 'running', 'acquired', 'error')),
  internal_ip TEXT,
  mcp_port INTEGER DEFAULT 8080,
  acquired_by TEXT,
  acquired_at TIMESTAMPTZ,
  boot_count INTEGER NOT NULL DEFAULT 0,
  last_resumed_at TIMESTAMPTZ,
  total_runtime_seconds INTEGER NOT NULL DEFAULT 0,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  UNIQUE(host_id, name)
);

CREATE INDEX idx_nested_vms_pool_state ON nested_vms(pool_id, state);
CREATE INDEX idx_nested_vms_host ON nested_vms(host_id);
CREATE INDEX idx_hyperv_hosts_status ON hyperv_hosts(status);

2.2 RPC Functions

-- Find and acquire available VM from a pool
CREATE OR REPLACE FUNCTION acquire_nested_vm(
  p_pool_id UUID,
  p_acquired_by TEXT
) RETURNS TABLE(vm_id UUID, host_ip TEXT, internal_ip TEXT, mcp_endpoint TEXT) AS $$
DECLARE
  v_vm nested_vms%ROWTYPE;
  v_host hyperv_hosts%ROWTYPE;
BEGIN
  SELECT * INTO v_vm FROM nested_vms
  WHERE pool_id = p_pool_id AND state = 'saved'
  ORDER BY last_resumed_at NULLS FIRST
  LIMIT 1 FOR UPDATE SKIP LOCKED;

  IF v_vm IS NULL THEN
    RAISE EXCEPTION 'No available VMs in pool %', p_pool_id;
  END IF;

  SELECT * INTO v_host FROM hyperv_hosts WHERE id = v_vm.host_id;

  UPDATE nested_vms SET
    state = 'acquired',
    acquired_by = p_acquired_by,
    acquired_at = now(),
    boot_count = boot_count + 1,
    last_resumed_at = now()
  WHERE id = v_vm.id;

  RETURN QUERY SELECT v_vm.id, v_host.public_ip, v_vm.internal_ip,
    format('http://%s:%s', v_vm.internal_ip, v_vm.mcp_port);
END;
$$ LANGUAGE plpgsql;

-- Release VM back to pool
CREATE OR REPLACE FUNCTION release_nested_vm(
  p_vm_id UUID,
  p_reset BOOLEAN DEFAULT false
) RETURNS void AS $$
BEGIN
  UPDATE nested_vms SET
    state = CASE WHEN p_reset THEN 'off' ELSE 'saved' END,
    acquired_by = NULL,
    acquired_at = NULL
  WHERE id = p_vm_id;
END;
$$ LANGUAGE plpgsql;

---

Phase 3: API Endpoints

3.1 Host Management

// src/app/api/admin/hosts/route.ts
GET  /api/admin/hosts           - List all hosts with VM counts
POST /api/admin/hosts           - Provision new Hyper-V host
GET  /api/admin/hosts/{id}      - Get host details
DELETE /api/admin/hosts/{id}    - Drain and delete host

3.2 Pool Management

// src/app/api/pools/route.ts
GET  /api/pools                     - List pools (filtered by org)
POST /api/pools                     - Create new pool
GET  /api/pools/{id}                - Get pool status
POST /api/pools/{id}/provision      - Provision N VMs to pool
POST /api/pools/{id}/scale          - Scale pool up/down

3.3 VM Acquire/Release

// src/app/api/vms/route.ts
POST /api/vms/acquire               - Acquire VM from pool (2-5s)
POST /api/vms/{id}/release          - Release VM back to pool
GET  /api/vms/{id}                  - Get VM status

---

Phase 4: Networking

4.1 Host NAT Configuration

Each nested VM gets internal IP (192.168.100.x). Port forwarding on host:

# VM 1: external 8080 -> 192.168.100.10:8080
# VM 2: external 8081 -> 192.168.100.11:8080
netsh interface portproxy add v4tov4 listenport=8080 connectaddress=192.168.100.10 connectport=8080

4.2 Dynamic Port Allocation

Assign unique port offset per VM. Host proxies external port to internal MCP.

---

Phase 5: Pool Scaling & Health

5.1 Pool Reconciliation Job

Runs every 60s:

• Check each pool's saved VM count vs min_ready
• Provision more if deficit
• Recover error VMs

5.2 Host Health Monitoring

• Ping host agent every 30s
• Sync VM states from Hyper-V to Supabase
• Mark failed hosts, trigger VM redistribution

---

Phase 6: Integration with Workflow Execution

Update selectMachineForExecution():

1. Check workflow -> pool assignment
2. Fall back to org's default pool
3. Fall back to global pool
4. Call /api/vms/acquire
5. Return MCP endpoint

---

Migration Strategy

1. Parallel Operation: Keep existing Azure VMs, deploy hosts alongside, feature flag routes % to nested
2. Gradual Migration: Start with non-critical workflows, monitor, increase traffic
3. Decommission: Once stable, deallocate individual Azure VMs

---

Cost Comparison

Metric	Current (10 VMs)	Nested (2 hosts x 10 VMs)
Compute	10 x D4s_v3 = $1,400/mo	2 x D8s_v5 = $560/mo
Boot time	30-60s	2-5s
Clean state	No	Yes (checkpoint reset)
Pre-warming	No	Yes (saved VMs ready)

Savings: ~60% ($840/mo for same capacity)

---

Open Questions

1. GPU support: GPU-PV for nested VMs? May need dedicated GPU hosts.
2. Networking: Port forwarding vs host proxy vs VPN tunnel?
3. Template distribution: How to sync VHDX across hosts?
4. Failure domains: Host dies -> VM redistribution strategy?

---

Acceptance Criteria

• Hyper-V host Packer image builds successfully
• Nested VM template with Terminator pre-installed
• Host agent can create/start/save/reset VMs
• Database schema deployed and RPC functions working
• API endpoints for host/pool/VM management
• Acquire VM completes in <5 seconds
• Release VM with checkpoint reset works
• Pool auto-scaling maintains min_ready VMs
• Existing workflow execution uses new pool system
• Monitoring dashboard for hosts/pools/VMs

---

References

• hyperv-kube prototype
• Azure nested virtualization docs
• Hyper-V save/resume