Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User space wireguard not working? #2799

Open
Silex opened this issue Oct 28, 2024 · 11 comments
Open

User space wireguard not working? #2799

Silex opened this issue Oct 28, 2024 · 11 comments

Comments

@Silex
Copy link

Silex commented Oct 28, 2024

First of all, thanks a lot for this awesome software! We are in the process of migrating everything to netbird and so far the experience is great.

I already succeed in running netbird on Teltonika routers, and now I'm trying to run netbird 0.28.4 on AXIS cameras. The goal is to be able to connect to the cameras (HTTP/RTSP) from other peers.

The problem is that the user running the application (ACAP) is very limited, but someone managed to run tailscale on them (https://github.com/Mo3he/Axis_Cam_Tailscale), so I'm pretty sure it'd be feasible to run netbird too.

Apparently they do it with --tun=userspace-networking when using tailscale (https://github.com/Mo3he/Axis_Cam_Tailscale/blob/main/aarch64/app/Tailscale_VPN).

So, I made a script like this:

#!/bin/sh

PACKAGE_DIR='/usr/local/packages/netbird'
TMP_DIR="$PACKAGE_DIR/tmp"

ARGS="--config $TMP_DIR/config.json --log-file $TMP_DIR/client.log --daemon-addr unix://$TMP_DIR/netbird.sock"
NETBIRD="$PACKAGE_DIR/lib/netbird $ARGS"

export NB_WG_KERNEL_DISABLED=true
# export NB_USE_NETSTACK_MODE=true
# export NB_SOCKS5_LISTENER_PORT=30000

echo '----------------------------------------'
echo 'Starting netbird'
mkdir -p $TMP_DIR
$NETBIRD service run &

echo 'Waiting 5 seconds'
sleep 5

echo 'Joining network'
$NETBIRD up --management-url https://netbird.foo.com --setup-key 123123123

echo 'Logs'
tail -f $TMP_DIR/client.log

This script runs fine as root (I can access my cameras), but when I run it as the application user (which will is the non-negociable default in latest firmares) then it outputs this:

2024-10-28T15:59:04.396+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14426]: Starting netbird
2024-10-28T15:59:04.398+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14426]: Waiting 5 seconds
2024-10-28T15:59:09.397+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14426]: Joining network
2024-10-28T15:59:12.398+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14493]: Connected
2024-10-28T15:59:12.406+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14426]: Waiting 5 seconds
2024-10-28T15:59:17.476+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14863]: Error: status failed: create wg interface: operation not permitted

And while the peer shows up in the netbird ui, of course I cannot connect to it. So next if I uncomment the line with NB_USE_NETSTACK_MODE, then it connects fine and I can even ping the camera, but I cannot remotely access it.

With NB_USE_NETSTACK_MODE here is what it status prints:

stvs@axis-b8a44fb94617:/tmp/netbird/lib$ ./netbird $ARGS status
OS: linux/arm64
Daemon version: 0.28.4
CLI version: 0.28.4
Management: Connected
Signal: Connected
Relays: 2/2 Available
Nameservers: 0/0 Available
FQDN: (retracted)
NetBird IP: 100.82.41.54/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 5/9 Connected

Here's what the client.log show:

2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/cmd/service_controller.go:64: started daemon server: /usr/local/packages/netbird/tmp/netbird.sock
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:257: generated new SSH key
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:273: using default Wireguard port 51820
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:284: using default Wireguard interface wt0
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:337: filling in interface blacklist with defaults: [ wt0 wt utun tun0 zt ZeroTier wg ts Tailscale tailscale docker veth br- lo ]
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:383: using default DNS route interval 1m0s
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:02+01:00 INFO client/internal/config.go:216: new Management URL provided, updated to "https://netbird.foo.com" (old value "https://api.netbird.io:443")
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:03+01:00 WARN client/server/server.go:259: failed login: rpc error: code = InvalidArgument desc = invalid setup-key or no sso information provided, err: invalid UUID length: 0
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:05+01:00 INFO client/internal/connect.go:119: starting NetBird client version 0.28.4 on linux/arm
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:05+01:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 ERRO iface/uapi.go:15: failed to open uapi socket: remove /var/run/wireguard/wt0.sock: no such file or directory
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 ERRO iface/wg_configurer_usp.go:191: failed to open uapi listener: remove /var/run/wireguard/wt0.sock: no such file or directory
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/firewall/create_linux.go:58: no firewall manager found, trying to use userspace packet filtering firewall
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/internal/connect.go:267: Netbird engine started, the IP is: 100.82.116.219/16
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/internal/dns/host_unix.go:68: System DNS manager discovered: file
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/internal/engine.go:1479: Network monitor is disabled, not starting
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO iface/tun_netstack.go:46: create netstack tun interface
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO signal/client/grpc.go:158: connected to the Signal Service stream
2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:07+01:00 ERRO client/internal/dns/server.go:322: unable to configure DNS for this peer using file manager without a nameserver group with all domains configured
2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:07+01:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 208.831µs, total rules count: 2
2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:07+01:00 INFO management/client/grpc.go:164: connected to the Management Service stream
2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:07+01:00 WARN client/internal/engine.go:587: running SSH server is not permitted

Can you clarify:

  1. Wether NB_USE_NETSTACK_MODE is actually a dead end and won't help me connect from outside to my camera.
  2. What rights "userspace wireguard" requires so I can ask AXIS about what I should do.
@Silex Silex changed the title User space mode not working User space mode privileges? Oct 28, 2024
@Silex Silex changed the title User space mode privileges? User space wireguard privileges? Oct 28, 2024
@Silex Silex changed the title User space wireguard privileges? User space wireguard not working? Oct 28, 2024
@lixmal
Copy link
Contributor

lixmal commented Oct 28, 2024

We currently haven't implemented any forwarding from the netstack NIC to elsewhere (host or routes), so unfortunately, this access won't work in netstack mode.

@Silex
Copy link
Author

Silex commented Oct 28, 2024

Thanks for the answer. Can you maybe clarify what netstack mode is for? I think it basically means "create a socks server that allows other applications to reach other peers through it", but that's "outgoing" only correct?

@lixmal
Copy link
Contributor

lixmal commented Oct 28, 2024

The intended use is in serverless environments: https://docs.netbird.io/how-to/netbird-on-faas

I think it basically means "create a socks server that allows other applications to reach other peers through it", but that's "outgoing" only correct?

correct

@Silex
Copy link
Author

Silex commented Oct 28, 2024

Thanks. Is it planned to allow for some --tun=userspace-networking like tailscale does?

I asked AXIS about what I can do with this limited user, we'll see what they say. Maybe I could create the wg interface as root and then netbird could work under the ACAP user.

@Silex
Copy link
Author

Silex commented Oct 29, 2024

@lixmal I'm a developer. How easy would it be to provide --tun=userspace-networking like tailscale does? I can maybe make a PR.

It's a shame because I find netbird much better than tailscale in pretty much every aspects, except I need a way to access remote cameras 😢

@lixmal
Copy link
Contributor

lixmal commented Oct 29, 2024

Thanks. Is it planned to allow for some --tun=userspace-networking like tailscale does?

We currently don't have any plans for that, there's been no request so far.

@lixmal I'm a developer. How easy would it be to provide --tun=userspace-networking like tailscale does? I can maybe make a PR.

It's not trivial. You'd have to implement a forwarder for tcp and udp (and possibly others like icmp, sctp, ...) packets that arrive on the wireguard netstack interface and decide where to forward them (= dial the destination address and copy over the payload).

Starting point would be somewhere here

tunIface, err := t.nsTun.Create()

That's when we get ahold of the netstack interface.

@Silex
Copy link
Author

Silex commented Oct 29, 2024

Thanks!

I noticed the AXIS camera sort the logs so they appear out of order. Here are the correct logs:

2024-10-29T16:34:48+01:00 INFO client/cmd/service_controller.go:24: starting Netbird service
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:147: generating new config /usr/local/packages/netbird/tmp/config.json
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:209: using default Management URL https://api.netbird.io:443
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:233: using default Admin URL https://api.netbird.io:443
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:251: generated new Wireguard key
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:257: generated new SSH key
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:273: using default Wireguard port 51820
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:284: using default Wireguard interface wt0
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:337: filling in interface blacklist with defaults: [ wt0 wt utun tun0 zt ZeroTier wg ts Tailscale tailscale docker veth br- lo ]
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:383: using default DNS route interval 1m0s
2024-10-29T16:34:48+01:00 INFO client/cmd/service_controller.go:64: started daemon server: /usr/local/packages/netbird/tmp/netbird.sock
2024-10-29T16:34:53+01:00 INFO client/internal/config.go:216: new Management URL provided, updated to "https://netbird.foo.com" (old value "https://api.netbird.io:443")
2024-10-29T16:34:54+01:00 WARN client/server/server.go:259: failed login: rpc error: code = InvalidArgument desc = invalid setup-key or no sso information provided, err: invalid UUID length: 0
2024-10-29T16:34:55+01:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service
2024-10-29T16:34:55+01:00 INFO client/internal/connect.go:119: starting NetBird client version 0.28.4 on linux/arm64
2024-10-29T16:34:56+01:00 INFO iface/tun_usp_unix.go:33: using userspace bind mode
2024-10-29T16:34:56+01:00 WARN client/internal/routemanager/manager.go:122: Failed cleaning up routing: 4 errors occurred:
        * rule with suppress prefixlen v4: remove routing rule: operation not permitted
        * rule with suppress prefixlen v6: remove routing rule: operation not permitted
        * rule v4 netbird: remove routing rule: operation not permitted
        * rule v6 netbird: remove routing rule: operation not permitted
2024-10-29T16:34:56+01:00 ERRO client/internal/routemanager/systemops/systemops_linux.go:100: Error setting up sysctl: 1 errors occurred:
        * write sysctl net.ipv4.conf.all.src_valid_mark: open /proc/sys/net/ipv4/conf/all/src_valid_mark: permission denied
2024-10-29T16:34:56+01:00 ERRO client/internal/routemanager/systemops/systemops_linux.go:108: Error cleaning up routing: 4 errors occurred:
        * rule with suppress prefixlen v4: remove routing rule: operation not permitted
        * rule with suppress prefixlen v6: remove routing rule: operation not permitted
        * rule v4 netbird: remove routing rule: operation not permitted
        * rule v6 netbird: remove routing rule: operation not permitted
2024-10-29T16:34:56+01:00 ERRO client/internal/engine.go:322: Failed to initialize route manager: setup routing: rule with suppress prefixlen v4: add routing rule: operation not permitted
2024-10-29T16:34:56+01:00 INFO iface/tun_usp_unix.go:48: create tun interface
2024-10-29T16:34:56+01:00 ERRO client/internal/engine.go:332: failed creating tunnel interface wt0: [operation not permitted]
2024-10-29T16:34:56+01:00 ERRO client/internal/routemanager/manager.go:166: Error cleaning up routing: 4 errors occurred:
        * rule with suppress prefixlen v4: remove routing rule: operation not permitted
        * rule with suppress prefixlen v6: remove routing rule: operation not permitted
        * rule v4 netbird: remove routing rule: operation not permitted
        * rule v6 netbird: remove routing rule: operation not permitted
2024-10-29T16:34:56+01:00 ERRO client/internal/connect.go:263: error while starting Netbird Connection Engine: create wg interface: operation not permitted

It looks like the Management Service connects just fine, but then the tun interface creation fails. I forwarded this to AXIS in the hope they can losen up restrictions using some ACAP flags.

@Silex
Copy link
Author

Silex commented Nov 4, 2024

@lixmal just found out that tailscale works because it uses https://github.com/WireGuard/wireguard-go

How feasible would it be for netbird to fallback to wireguard-go as a backup strategy? I assume it'd be a lot of dev.

@lixmal
Copy link
Contributor

lixmal commented Nov 4, 2024

Interface type: Userspace

Already uses wireguard-go

@Silex
Copy link
Author

Silex commented Nov 4, 2024

Then I'm confused about why tailscale works and netbird does not (without NB_USE_NETSTACK_MODE of course).

Will investigate more and report.

@Silex
Copy link
Author

Silex commented Nov 5, 2024

Ok, so the crux is not wireguard-go, tailscale uses a Userspace Network Stack which allows it to bypass the need of CAP_NET_ADMIN.

Implementing the same in netbird is likely to be a lot of work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants