Openwrt support

[wen-long]

Thanks for your work! it's wonderful
Do you hava any plan to make this work on openwrt?

[hack3ric]

Hi, I'm definitely interested in creating OpenWrt packages for Mimic!

However, Mimic currently requires at least Linux 6.1 (due to usage of ringbuf dynptr), while OpenWrt 23.05 uses 5.15. It may be feasible to retrofit bpf_perf_event_output or add an option to complete disable its usage, since it is only used to cache packets before handshake and resend them afterwards.

Even this is resolved, availability on MIPS and 32-bit Arm still needs to be tested. I don't have any spare router to test now but it's definitely worth trying.

[wen-long]

how about drop packets before handshake?
a small and fast working demo can tell the performance on normal openwrt hareware
after then maybe more developer will come to help

I have a router(GL-AXT1800) to try this, but dont have any idea to start a kernel module dev
do you like to give some suggest?

~# cat /proc/cpuinfo
processor	: 0
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 72.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x51
CPU architecture: 7
CPU variant	: 0xa
CPU part	: 0x801
CPU revision	: 4

processor	: 1
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 72.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x51
CPU architecture: 7
CPU variant	: 0xa
CPU part	: 0x801
CPU revision	: 4

processor	: 2
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 72.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x51
CPU architecture: 7
CPU variant	: 0xa
CPU part	: 0x801
CPU revision	: 4

processor	: 3
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 72.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x51
CPU architecture: 7
CPU variant	: 0xa
CPU part	: 0x801
CPU revision	: 4

Hardware	: Qualcomm Technologies, Inc IPQ6000
Revision	: 0000
Serial		: 0000000000000000
~# cat /proc/version
Linux version 4.4.60 (glinet@glinet) (gcc version 5.5.0 (OpenWrt GCC 5.5.0 r16399+165-c67509efd7) ) #0 SMP PREEMPT Tue Jan 23 12:33:08 2024

[hack3ric]

how about drop packets before handshake?
a small and fast working demo can tell the performance on normal openwrt hareware
after then maybe more developer will come to help

Sure, I'll add build flags to disable that part first.

I have a router(GL-AXT1800) to try this, but dont have any idea to start a kernel module dev
do you like to give some suggest?

I'm trying OpenWrt SDK today and I plan to post build scripts shortly after. You can then add the package feed to OpenWrt SDK and install the produced .ipk file directly to test it. Thanks in advance!

[hack3ric]

Just noticed CONFIG_DEBUG_INFO_BTF isn't enabled by default on OpenWrt, so one may need to rebuild the base image in order to get Mimic running on wireless routers...

[hack3ric]

Seems BTF kfuncs support is only added in around 5.18~6.0, so we're out of luck with OpenWrt 23.05's 5.15 kernel. Building on main should be fine, but we need to wait for their new major release with 6.1/6.6 kernel to be stable.

[wen-long]

GL-AXT1800 is not officaly supported by openwrt, and no IPQ6000 under make menuconfig -> Target System
GLinet stop open their repo publicly, I've send mail to ask for permission as the thread mentioned https://forum.gl-inet.com/t/32901/2
or maybe start from openwrt x86 is also a chooice?

[hack3ric]

Built Mimic against snapshot OpenWrt armsr/armv7 with BTF on, sadly 32-bit Arm's BPF isn't capable of calling kernel function even on Linux 6.6.

root@OpenWrt:/# modprobe mimic
[ 2865.893256] mimic: loading out-of-tree module taints kernel.
root@OpenWrt:/# mimic run lo
 Warn libbpf: prog 'egress_handler': BPF program load failed: No error information
 Warn libbpf: prog 'egress_handler': -- BEGIN PROG LOAD LOG --
JIT does not support calling kernel function
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
-- END PROG LOAD LOG --
 Warn libbpf: prog 'egress_handler': failed to load: -524
 Warn libbpf: failed to load object 'mimic_bpf'
 Warn libbpf: failed to load BPF skeleton 'mimic_bpf': -524
Error failed to load BPF program: No error information
 Info cleaning up
root@OpenWrt:/# uname -a
Linux OpenWrt 6.6.41 #0 SMP Fri Jul 19 14:53:24 2024 armv7l GNU/Linux
root@OpenWrt:/# 

or maybe start from openwrt x86 is also a chooice?

Mimic should run fine on x86_64 and aarch64. Will push build script and try OpenWrt on more platforms using QEMU later.

[hack3ric]

Encountered some issues in building kmod-sched-bpf in OpenWrt, which is required by Mimic for its TC egress path. Reported to openwrt/openwrt#16071.

Other than this, Mimic should be able to run on OpenWrt main branch.

[hack3ric]

I've create a OpenWrt package feed for Mimic: https://github.com/hack3ric/mimic/tree/openwrt

After some hacking, kernel BTF and dynptr dependency is disabled in OpenWrt package script. However, on 23.05 it still fails with "Invalid argument" and no reason. I guess it is from the verifier.

It works on snapshot build and should run fine on all (little-endian) platforms, including 32-bit ARM, though.

[wen-long]

make[4]: Entering directory '/home/xx/openwrt/build_dir/target-x86_64_musl/linux-x86_64/bpf-headers/linux-5.15.161'
/bin/sh: 0: Illegal option -W
: unknown compiler
: unknown assembler invoked
scripts/Kconfig.include:50: Sorry, this assembler is not supported.
make[5]: *** [scripts/kconfig/Makefile:77: oldconfig] Error 1
make[4]: *** [Makefile:628: oldconfig] Error 2

not lucky for me, waste too much time on these build script shit, give up now.
hope you keep work on this, maybe one can enjoy mimic outofbox someday