Support for Virtualization.framework #923

balajiv113 · 2022-06-21T06:34:42Z

balajiv113
Jun 21, 2022
Maintainer

This discussion is to come up with a plan for implementing drivers in lima and also provide support for Virtualization.framework in macOS.

Requirements

Support for Virtualization must be optional
Only available on macOS
Same model should be extensible for other drivers in future

Support for different drivers

The base idea is to support different drivers implementations and pick a driver accordingly based on the VM configuration.

Idea - Driver code will be part of lima

The idea is to have different drivers as part of lima itself (No separate binaries for drivers)

qemu - Will be the default driver
vz - available only for macOS

Driver - qemu

All the current features/codeflows will stay the same, we will only refactor code to support multiple driver model.

Note - There should be no change in configuration file as well

Driver - vz

The main complication here will be Virtualization.framework is available only after macOS 11. Also features we will use (EFILoader) are available only as part of macOS 13.
But we should support older versions of macOS as well.

To overcome this complications, I have came up with 2 integration options

Integration - Directly use vz go-binding in lima

In this model, lima will directly depend on code-hex/vz.git go binding for mac virtualization API. And the vz driver will create VM using those go-binding.

Pros

Better/Deep integration with virtualization API
We can make use of VZ features like vsock, vmStateChange etc

Cons

We should plan for not building this driver/use no-op if lower version of macOS is used. One way to do is maybe to use custom build tag and set it during build based on current building os version

Integration - Cmd line utility for vz

In this model, We will develop a cmd line tool like https://github.com/code-ready/vfkit and the driver will simply spawn a child process like in qemu driver

Pros

Easily ignore driver at runtime base on current running version. The cli is already built, and we will simply download when needed.

Cons

Hard to provide a deep integration / more work will be needed on creating the cli tool

Configuration changes

I tried to come up with a basic configuration model for driver based solution.
The goal I had in mind was to accommodate our current configuration for new driver as well.
This way, we can reuse configuration for different drivers as well.

driver: "vz" //We should maybe allow this to be given during initial start command

images:
  - location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
    arch: "x86_64"
  - location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"
    arch: "aarch64"
  - location: "x86_64 iso file"
    arch: "x86_64"
    driver: "vz"
  - location: "aarch64 iso file"
    arch: "aarch64"
    driver: "vz"

mounts:
  - location: "~"
    mountPoint: null
    writable: null
    sshfs:
      cache: null
      followSymlinks: null
      sftpDriver: null
  - location: "/tmp/lima"
    writable: true

#Default - virtio-fs, alternatively we can support reverse-sshfs as well.
mountType: "virtio-fs"

video:
  #Only display vz will be support for driver type vz
  display: vz
networks:
#Need to think more on this on how to provide host to guest communication, vz support NAT and Bridge networks models
#We should implement slirp network for the guest to host communication

With this above configurations, We must need to do some driver specific yaml validations as well.

This will be the first steps. Based on the integration model we choose I can come up with a set of checklists on what needs to be done.

AkihiroSuda · 2022-06-30T02:12:54Z

AkihiroSuda
Jun 30, 2022
Maintainer

To overcome this complications, I have came up with 2 integration options

Thanks, the former one seems better but the latter one is also acceptable if it is easier

1 reply

balajiv113 Jun 30, 2022
Maintainer Author

Sure, i will do a small poc on the former one to validate my assumptions around building.

The later is easier for basic setup but will get more complicated as we look for more advanced stuffs.

jandubois · 2022-06-30T18:46:32Z

jandubois
Jun 30, 2022
Maintainer

Thanks, the former one seems better but the latter one is also acceptable if it is easier

I agree that the former sounds better for deeper integration, but I'm not so keen on having different binaries for each macOS version. We would need at least 6: macOS 10, 11, and 13, for both aarch64 and x86_64.

But even with the latter approach we would probably need separate binaries of the CLI for macOS 11 and 13, so that wouldn't really help either.

I don't know if it is reasonably feasible to use a mechanism like dlopen/dlsym on Linux to dynamically link the framework at runtime based on OS version.

2 replies

balajiv113 Jun 30, 2022
Maintainer Author

I have some ideas to resolve this concern. Objective c has @available which can be used to provide condition logic at runtime based on os version.

Example - https://github.com/gyf304/vmcli/blob/main/vmcli/Sources/vmcli/main.swift#L213

Will give a shot with these as well and see if it resolves our concern

afbjorklund Jul 1, 2022
Maintainer

I seem to recall using function pointers to support different versions of OpenGL, so it is certainly doable - not sure about feasible

Code-Hex · 2022-09-04T08:17:46Z

Code-Hex
Sep 4, 2022

@AkihiroSuda @balajiv113 Hi 👋
I'm author of vz.

lima VM is great!! Thank you.

Btw, I released v2 version of the library.

https://github.com/Code-Hex/vz

I think I can help if you can report bugs etc.

Thank you!

0 replies

balajiv113 · 2022-10-11T17:54:20Z

balajiv113
Oct 11, 2022
Maintainer Author

Apologies for the quiet period on this thread. Have been trying to get hands-on to macOS 13 to try out new API's :)

The following are the updates,

vz now supports building in older version that has support for Virtualization.framework (Greater than 11.x).

Some questions now will be,

What will be the minimum macOS supported version for lima ?? (With release of macOS 13 i believe macOS catalina will be reaching EOL and also QEMU might drop support)
I assume, we will target Virtualization.Framework support only from macOS 13. The reason being, only from macOS 13 we have support for EFILoader.

2 replies

AkihiroSuda Oct 12, 2022
Maintainer

What will be the minimum macOS supported version for lima ?? (With release of macOS 13 i believe macOS catalina will be reaching EOL and also QEMU might drop support)

macOS 11, currently.
10.15 seems to have reached EOL.

I assume, we will target Virtualization.Framework support only from macOS 13. The reason being, only from macOS 13 we have support for EFILoader.

Yes 👍

jandubois Oct 12, 2022
Maintainer

What will be the minimum macOS supported version for lima ?? (With release of macOS 13 i believe macOS catalina will be reaching EOL and also QEMU might drop support)

macOS 11, currently.

I'm not aware of any feature in Lima 0.12 that is not supported on 10.15. So we many not support it, but it doesn't mean it doesn't work. 😄

FWIW, Rancher Desktop does still support macOS 10.15 (using Lima 0.12).

10.15 seems to have reached EOL.

It still had a security update in July and a new Safari release in August, but I agree, once Ventura is released I would not expect any further updates to Catalina.

All of this is kind of irrelevant though; I agree that macOS 13 is a reasonable minimum requirement for supporting the Virtualization framework in Lima.

balajiv113 · 2022-10-13T18:24:02Z

balajiv113
Oct 13, 2022
Maintainer Author

I have started working on some code changes on this.
As initial step i have done the following,

create driver API
migrate existing code into qemu driver
create new driver for vz (for now i created using macOS 12.0 API, we can change this easily once 13.0 API are supported from vz)

The initial changes are under,
master...balajiv113:lima:driver
Feel free to provide feedback on Driver Interface or other parts as well.

The next important step would be networking (SLIRP). For this, i was able to successfully use gvisor-tap-vsock with vz.
Is it okay to depend on gvisor-tap-vsock ? Is there any other options / ideas around this area to consider ??

7 replies

AkihiroSuda Oct 14, 2022
Maintainer

@abiosoft Could you share your thought on pros and cons of gvisor-tap-vsock over libslirp?

Wondering we should replace libslirp with gvisor-tap-vsock for QEMU too, as I don't want to maintain two slirp drivers.

abiosoft Oct 14, 2022

FYI, Colima does not replace slirp entirely, it simply adds gvisor-tap-vsock as an additional network and makes it the default (when enabled).

This is to avoid the possibility of breaking Lima's network assumptions and to retain Lima's port-forwarding feature.

The following is what I could think of (right now) if Lima were to adopt gvisor-tap-vsock.

	slirp	gvisor-tap-vsock
Setup	Tied to the QEMU process, i.e. no setup required	Daemon/running process required (can be tied to host-agent)
Dynamic Port Forwarding	Handled by Lima (SSH forwarding)	Supported via the REST API
Network Speed	Fast, near the speed of the host machine	Fast, but slower than slirp
DNS - custom hosts	Handled by Lima	Supported
VPN connections	Handled by Lima	Supported (requires CGO_ENABLED=1)
Inter-VM network	Handled by Lima (via vmnet)	Supported (VMs must be connected to same instance)

I think @afbjorklund might have more experience with it, as gvisor-tap-vsock is more utilised on podman-machine than Colima.

AkihiroSuda Oct 14, 2022
Maintainer

Thanks,

Fast, but slower than slirp

Do you have benchmark?

balajiv113 Oct 14, 2022
Maintainer Author

@AkihiroSuda
These are the benchmarks i did recently with gvisor-tap-vsock after dependency updates

containers/gvisor-tap-vsock#135 (comment)

afbjorklund Oct 14, 2022
Maintainer

I think @afbjorklund might have more experience with it, as gvisor-tap-vsock is more utilised on podman-machine than Colima.

The old podman-machine still used slirp (for qemu), before the Red Hat rewrite with CoreOS. So don't know much about it.

balajiv113 · 2022-10-29T08:51:03Z

balajiv113
Oct 29, 2022
Maintainer Author

Next updates,
code-hex/vz now supports macOS 13 API 🎉

I have integrated the same in this branch, also integrated gvisor for slirp network.
master...balajiv113:lima:driver

I was able to run existing docker.yml template successfully 🎉

Next & Final Steps

Cleanup code (need to do proper error handlings, make gvisor more generic to be used by qemu, configurable driver)
Host to guest communication (Should be able to use NAT support from vz (sudo not needed) or use file attachment to support existing vmnet)
Support iso based startup for alpine
VM Display support

5 replies

shyim Oct 30, 2022

I tested your branch and I failed with this. IDK If I failed something. Using macOS 13.1 beta and XCode 14.1 RC 2

INFO[0000] [hostagent] 2022-10-30 21:17:24.480 limactl[8645:167866] *** Terminating app due to uncaught exception 'UnhandledException', reason: 'bug'
INFO[0000] [hostagent] *** First throw call stack:
INFO[0000] [hostagent] (
INFO[0000] [hostagent] 	0   CoreFoundation                      0x00000001ad572418 __exceptionPreprocess + 176
INFO[0000] [hostagent] 	1   libobjc.A.dylib                     0x00000001ad0beea8 objc_exception_throw + 60
INFO[0000] [hostagent] 	2   CoreFoundation                      0x00000001ad59b830 _CFBundleGetValueForInfoKey + 0
INFO[0000] [hostagent] 	3   limactl                             0x00000001032a61b4 newVZGenericMachineIdentifier + 44
INFO[0000] [hostagent] 	4   limactl                             0x00000001032a3f30 _cgo_dbeaa51307a6_Cfunc_newVZGenericMachineIdentifier + 32
INFO[0000] [hostagent] 	5   limactl                             0x0000000102bcce5c runtime.asmcgocall.abi0 + 124
INFO[0000] [hostagent] )
INFO[0000] [hostagent] libc++abi: terminating with uncaught exception of type NSException
INFO[0000] [hostagent] SIGABRT: abort
INFO[0000] [hostagent] PC=0x1ad3e11c4 m=5 sigcode=0
INFO[0000] [hostagent] signal arrived during cgo execution
INFO[0000] [hostagent] goroutine 1 [syscall]:
INFO[0000] [hostagent] runtime.cgocall(0x1032a3f10, 0x1400038f598)
INFO[0000] [hostagent] 	/nix/store/84fhzs9abhyaa8pzhc1vg4j3w2mf5bn2-go-1.19.2/share/go/src/runtime/cgocall.go:158 +0x54 fp=0x1400038f560 sp=0x1400038f520
INFO[0000] [hostagent] 0x102b66524
INFO[0000] [hostagent] github.com/Code-Hex/vz/v2._Cfunc_newVZGenericMachineIdentifier()
INFO[0000] [hostagent] 	_cgo_gotypes.go:873 +0x38
INFO[0000] [hostagent]  sp=0x1400038f560 pc=0x103212cd8
INFO[0000] [hostagent] github.com/Code-Hex/vz/v2.NewGenericMachineIdentifier()
INFO[0000] [hostagent] 	/Users/shyim/go/pkg/mod/github.com/!code-!hex/vz/[email protected]/platform.go:119 +0x74 fp=0x1400038f5b0 sp=0x1400038f590 pc=0x103218674
INFO[0000] [hostagent] github.com/lima-vm/lima/pkg/vz.CreateVM({0x14000100000
INFO[0000] [hostagent] 0x140001fb200)
INFO[0000] [hostagent] 	/Users/shyim/Downloads/lima/pkg/vz/vz.go:
INFO[0000] [hostagent]  +0x1a0 fp=0x1400038f8d0 sp=0x1400038f5b0 pc=0x10324eb40
INFO[0000] [hostagent] github.com/lima-vm/lima/pkg/vz.(*LimaVzDriver).Start(

balajiv113 Oct 31, 2022
Maintainer Author

I believe you are using M1. Am not getting this in my intel. But getting this error in M1.
I will check it out and update once fixed

balajiv113 Oct 31, 2022
Maintainer Author

The same error happened to me initially when tried with M1. But after using proper command line tools (14.1RC) and cleaning of go modules. It started to work fine for me.

During compilation please check if you are getting any error like "macOS 13 API's are disabled". If this error is coming, then it will fail during execution.

If you are still facing error, then please post down the following details,

 - Xcode version: [`xcodebuild -version | pbcopy`]
 - macOS Version: [`sw_vers | pbcopy`]
 - mac architecture: [`uname -m | pbcopy`]
 - Go Version: [`go version | pbcopy`]

shyim Oct 31, 2022

After fixing APIs I get now:

INFO[0000] [hostagent] 52:55:55:3e:35:2f
INFO[0000] [hostagent] After disk
FATA[0000] exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/shyim/.lima/default/ha.stderr.log")

❯ cat /Users/shyim/.lima/default/ha.stderr.log
{"level":"debug","msg":"Creating iso file /Users/shyim/.lima/default/cidata.iso","time":"2022-10-31T09:00:00+01:00"}
{"level":"debug","msg":"Using /var/folders/g7/5l87gt3960b0ww74f0t4v42m0000gn/T/diskfs_iso3306954927 as workspace","time":"2022-10-31T09:00:00+01:00"}
{"level":"debug","msg":"OpenSSH version 9.0.1 detected","time":"2022-10-31T09:00:01+01:00"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing [email protected] and [email protected]","time":"2022-10-31T09:00:01+01:00"}
{"level":"debug","msg":"Start udp server listening on: 127.0.0.1:64871","time":"2022-10-31T09:00:01+01:00"}
{"level":"debug","msg":"Start tcp server listening on: 127.0.0.1:54319","time":"2022-10-31T09:00:01+01:00"}
{"level":"info","msg":"52:55:55:3e:35:2f","time":"2022-10-31T09:00:01+01:00"}
{"level":"info","msg":"After disk","time":"2022-10-31T09:00:01+01:00"}
{"level":"fatal","msg":"stat ~: no such file or directory","time":"2022-10-31T09:00:01+01:00"}

balajiv113 Oct 31, 2022
Maintainer Author

Please fix the mount directory in docker.yml. its hard-coded to my folder (/Users/balaji).
~ is not supported as of now. I eill add support for this in next set of changes

Code-Hex · 2022-11-02T16:27:41Z

Code-Hex
Nov 2, 2022

Hi 👋

I have released the vz v3 version which has supported Apple Virtualization.framework for macOS 13 API.

Linux GUI Support
Running Intel Binaries in Linux VMs with Rosetta
EFI Boot loader

https://pkg.go.dev/github.com/Code-Hex/vz/v3

I have also started compiling the Linux information needed to use each of the features provided by the Virtualisation.framework here. Anyone is free to edit it.

https://github.com/Code-Hex/vz/wiki

It was very hard, but I'm very excited to use lima with support for the Apple Virtualization.framework. Cheers!

0 replies

balajiv113 · 2022-11-07T06:35:19Z

balajiv113
Nov 7, 2022
Maintainer Author

Updates
Done the final set of changes 🎉

Driver cleanup
Host network via NAT
ISO based alpine support 🎉

Have opened the PR as well #1147
Do try it out and provide your feedbacks

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Virtualization.framework #923

{{title}}

Replies: 8 comments 17 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Support for Virtualization.framework #923

balajiv113 Jun 21, 2022 Maintainer

Requirements

Support for different drivers

Idea - Driver code will be part of lima

Driver - qemu

Driver - vz

Integration - Directly use vz go-binding in lima

Integration - Cmd line utility for vz

Configuration changes

Replies: 8 comments · 17 replies

AkihiroSuda Jun 30, 2022 Maintainer

balajiv113 Jun 30, 2022 Maintainer Author

jandubois Jun 30, 2022 Maintainer

balajiv113 Jun 30, 2022 Maintainer Author

afbjorklund Jul 1, 2022 Maintainer

Code-Hex Sep 4, 2022

balajiv113 Oct 11, 2022 Maintainer Author

AkihiroSuda Oct 12, 2022 Maintainer

jandubois Oct 12, 2022 Maintainer

balajiv113 Oct 13, 2022 Maintainer Author

AkihiroSuda Oct 14, 2022 Maintainer

abiosoft Oct 14, 2022

AkihiroSuda Oct 14, 2022 Maintainer

balajiv113 Oct 14, 2022 Maintainer Author

afbjorklund Oct 14, 2022 Maintainer

balajiv113 Oct 29, 2022 Maintainer Author

shyim Oct 30, 2022

balajiv113 Oct 31, 2022 Maintainer Author

balajiv113 Oct 31, 2022 Maintainer Author

shyim Oct 31, 2022

balajiv113 Oct 31, 2022 Maintainer Author

Code-Hex Nov 2, 2022

balajiv113 Nov 7, 2022 Maintainer Author

balajiv113
Jun 21, 2022
Maintainer

Replies: 8 comments 17 replies

AkihiroSuda
Jun 30, 2022
Maintainer

balajiv113 Jun 30, 2022
Maintainer Author

jandubois
Jun 30, 2022
Maintainer

balajiv113 Jun 30, 2022
Maintainer Author

afbjorklund Jul 1, 2022
Maintainer

Code-Hex
Sep 4, 2022

balajiv113
Oct 11, 2022
Maintainer Author

AkihiroSuda Oct 12, 2022
Maintainer

jandubois Oct 12, 2022
Maintainer

balajiv113
Oct 13, 2022
Maintainer Author

AkihiroSuda Oct 14, 2022
Maintainer

AkihiroSuda Oct 14, 2022
Maintainer

balajiv113 Oct 14, 2022
Maintainer Author

afbjorklund Oct 14, 2022
Maintainer

balajiv113
Oct 29, 2022
Maintainer Author

balajiv113 Oct 31, 2022
Maintainer Author

balajiv113 Oct 31, 2022
Maintainer Author

balajiv113 Oct 31, 2022
Maintainer Author

Code-Hex
Nov 2, 2022

balajiv113
Nov 7, 2022
Maintainer Author