ERROR: Error: Initialization issues during scap_init
#3323
Comments
|
Hi! Thanks for reporting this issue; i don't have an answer, this seems really weird; since at every restart Falco is using the same driver (ie: modern ebpf one in this case), perhaps it is a timing issue with something else on the system? /milestone 0.40.0 |
|
@FedeDP FYI I removed the |
|
IMO we should enable a more verbose log |
|
I encountered the same issue in a similar environment, and switching to eBPF mode instead of modern_eBPF was the only solution that helped. I tried enabling debug logs, but they didn’t provide any insight. Additionally, it’s worth noting that in an EKS cluster with 4 nodes, only 1 node failed to start Falco in modern_eBPF mode (although the kernel version is the same on all nodes) |
|
Seeing a very similar behavior here: This is a very vanilla helm installation with the following values: customRules:
rules-override.yaml: |-
- macro: user_known_contact_k8s_api_server_activities
condition: |-
container.image.repository = registry.k8s.io/node-problem-detector/node-problem-detector
or
proc.name startswith node-problem-de
or
container.image.repository = ghcr.io/roobre/ktemplate
or
container.image.repository = ghcr.io/k8up-io/k8up
or
container.name startswith k8up
override:
condition: replace
- macro: user_known_stand_streams_redirect_activities
condition: |-
container.image.repository = ghcr.io/fluxcd/kustomize-controller
or
(container.name startswith crocochrome and proc.name = chromium)
override:
condition: replace
- macro: known_drop_and_execute_activities
condition: |-
(container.image.repository = ghcr.io/flaresolverr/flaresolverr and proc.name = chromedriver)
override:
condition: replace
- macro: user_read_sensitive_file_containers
condition: |-
container.id = host
override:
condition: replace
- list: user_known_packet_socket_binaries
items:
- speaker # metallb
- bfdd # also metallb
override:
items: append
resources:
requests:
cpu: 50m
memory: 128Mi
limits:
cpu: null
memory: 512Mi
falcosidekick:
enabled: true
replicaCount: 1
resources:
requests:
cpu: 10m
memory: 64Mi
limits:
memory: 64Mi
config:
existingSecret: credsUsing the default image shipped in the chart dependencies:
- name: falco
repository: https://falcosecurity.github.io/charts
version: 4.11.1Also attaching |
|
Hello, The falco pods running in my GKE cluster fail with the same error Falco output: Thu Oct 31 08:50:11 2024: The --cri option is deprecated and will be removed in Falco 0.40.0. Use -o container_engines.cri.sockets[]=<socket_path> instead.
Thu Oct 31 08:50:11 2024: Falco version: 0.39.1 (x86_64)
Thu Oct 31 08:50:11 2024: Falco initialized with configuration files:
Thu Oct 31 08:50:11 2024: /etc/falco/falco.yaml | schema validation: ok
Thu Oct 31 08:50:11 2024: System info: Linux version 6.6.44+ (builder@5b283881ec70) (Chromium OS 17.0_pre498229-r33 clang version 17.0.0 (/var/cache/chromeos-cache/distfiles/egit-src/external/github.com/llvm/llvm-project 14f0776550b5a49e1c42f49a00213f7f3fa047bf), LLD 17.0.0) #1 SMP PREEMPT_DYNAMIC Sat Sep 28 09:09:42 UTC 2024
Thu Oct 31 08:50:11 2024: Loading rules from:
Thu Oct 31 08:50:11 2024: /etc/falco/falco_rules.yaml | schema validation: ok
Thu Oct 31 08:50:11 2024: Hostname value has been overridden via environment variable to: gke-***
Thu Oct 31 08:50:11 2024: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Thu Oct 31 08:50:11 2024: Starting health webserver with threadiness 4, listening on 0.0.0.0:8765
Thu Oct 31 08:50:11 2024: Loaded event sources: syscall
Thu Oct 31 08:50:11 2024: Enabled event sources: syscall
Thu Oct 31 08:50:11 2024: Opening 'syscall' source with modern BPF probe.
Thu Oct 31 08:50:11 2024: One ring buffer every '2' CPUs.
Thu Oct 31 08:50:11 2024: An error occurred in an event source, forcing termination...
Error: Initialization issues during scap_init
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:Environment
|
|
+1, same behaviour as @PierreBart |
|
Which version of k8s are you seeing this behavior on? I'm just starting to troubleshoot the same problem so I haven't had a chance to isolate it yet, but I'm only seeing it in k8s v1.31 so far. |
|
@shane-lawrence 1.29 |
|
Thanks @OneideLuizSchneider, sounds like the k8s version was a red herring and it must be some other difference that's triggering this issue. I will provide more context if I find something. |
|
@shane-lawrence I am running 1.31.1, same as you. I have clusters in 1.30.5, and it runs just fine. |
|
I don't think it has something to do with the k8s version, because it's random on some Nodes, not on every Node. |
|
I'm experiencing the issues on Fedora 40 and Fedora 41 hosts without Kubernetes. Sometimes falco works and other times it crashloop with the same config on different hosts. Could it have something to do with having too few resources available to falco at startup or it conflicting with other processes like auditd? |
|
@shane-lawrence we're observing it on GKE 1.31.1 right now |
|
Hi folks! could you try to enable the sudo ./usr/bin/falco -c ./etc/falco/falco.yaml -r ./etc/falco/falco_rules.yaml -o libs_logger.enabled=true -o libs_logger.severity=debug |
|
There it is: Defaulted container "falco" out of: falco, falcoctl-artifact-follow, falcoctl-artifact-install (init)
Mon Nov 4 09:58:02 2024: The --cri option is deprecated and will be removed in Falco 0.40.0. Use -o container_engines.cri.sockets[]=<socket_path> instead.
Mon Nov 4 09:58:02 2024: Falco version: 0.39.1 (x86_64)
Mon Nov 4 09:58:02 2024: Falco initialized with configuration files:
Mon Nov 4 09:58:02 2024: /etc/falco/falco.yaml | schema validation: ok
Mon Nov 4 09:58:02 2024: System info: Linux version 6.6.44+ (builder@5b283881ec70) (Chromium OS 17.0_pre498229-r33 clang version 17.0.0 (/var/cache/chromeos-cache/distfiles/egit-src/external/github.com/llvm/llvm-project 14f0776550b5a49e1c42f49a00213f7f3fa047bf), LLD 17.0.0) #1 SMP PREEMPT_DYNAMIC Sat Sep 28 09:09:42 UTC 2024
Mon Nov 4 09:58:02 2024: Loading rules from:
Mon Nov 4 09:58:02 2024: /etc/falco/falco_rules.yaml | schema validation: ok
Mon Nov 4 09:58:02 2024: Hostname value has been overridden via environment variable to: gke-***
Mon Nov 4 09:58:02 2024: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Mon Nov 4 09:58:02 2024: Starting health webserver with threadiness 4, listening on 0.0.0.0:8765
Mon Nov 4 09:58:02 2024: Loaded event sources: syscall
Mon Nov 4 09:58:02 2024: Enabled event sources: syscall
Mon Nov 4 09:58:02 2024: Opening 'syscall' source with modern BPF probe.
Mon Nov 4 09:58:02 2024: One ring buffer every '2' CPUs.
Mon Nov 4 09:58:02 2024: [libs]: libbpf: prog 'capset_x': BPF program load failed: Permission denied
Mon Nov 4 09:58:02 2024: [libs]: libbpf: prog 'capset_x': -- BEGIN PROG LOAD LOG --
reg type unsupported for arg#0 function capset_x#984
0: R1=ctx(off=0,imm=0) R10=fp0
; int BPF_PROG(capset_x, struct pt_regs *regs, long ret) {
0: (bf) r7 = r1 ; R1=ctx(off=0,imm=0) R7_w=ctx(off=0,imm=0)
; int BPF_PROG(capset_x, struct pt_regs *regs, long ret) {
1: (79) r9 = *(u64 *)(r7 +8) ; R7_w=ctx(off=0,imm=0) R9_w=scalar()
; uint32_t cpu_id = (uint32_t)bpf_get_smp_processor_id();
2: (85) call bpf_get_smp_processor_id#8 ; R0_w=scalar(umax=3,var_off=(0x0; 0x3))
; uint32_t cpu_id = (uint32_t)bpf_get_smp_processor_id();
3: (63) *(u32 *)(r10 -8) = r0 ; R0_w=scalar(umax=3,var_off=(0x0; 0x3)) R10=fp0 fp-8=
4: (bf) r2 = r10 ; R2_w=fp0 R10=fp0
;
5: (07) r2 += -8 ; R2_w=fp-8
; return (struct ringbuf_map *)bpf_map_lookup_elem(&ringbuf_maps, &cpu_id);
6: (18) r1 = 0xffff8880490d1c00 ; R1_w=map_ptr(off=0,ks=4,vs=4,imm=0)
8: (85) call bpf_map_lookup_elem#1 ; R0=map_value_or_null(id=1,off=0,ks=4,vs=4,imm=0)
9: (bf) r6 = r0 ; R0=map_value_or_null(id=1,off=0,ks=4,vs=4,imm=0) R6_w=map_value_or_null(id=1,off=0,ks=4,vs=4,imm=0)
; if(!rb) {
10: (55) if r6 != 0x0 goto pc+6 17: R0=map_ptr(off=0,ks=0,vs=0,imm=0) R6=map_ptr(off=0,ks=0,vs=0,imm=0) R7=ctx(off=0,imm=0) R9=scalar() R10=fp0 fp-8=????mmmm
; uint32_t cpu_id = (uint32_t)bpf_get_smp_processor_id();
17: (85) call bpf_get_smp_processor_id#8 ; R0_w=scalar(umax=3,var_off=(0x0; 0x3))
; uint32_t cpu_id = (uint32_t)bpf_get_smp_processor_id();
18: (63) *(u32 *)(r10 -8) = r0 ; R0_w=scalar(umax=3,var_off=(0x0; 0x3)) R10=fp0 fp-8=
19: (bf) r2 = r10 ; R2_w=fp0 R10=fp0
;
20: (07) r2 += -8 ; R2_w=fp-8
; return (struct counter_map *)bpf_map_lookup_elem(&counter_maps, &cpu_id);
21: (18) r1 = 0xffff888097e63c00 ; R1_w=map_ptr(off=0,ks=4,vs=136,imm=0)
23: (85) call bpf_map_lookup_elem#1 ; R0_w=map_value_or_null(id=2,off=0,ks=4,vs=136,imm=0)
24: (bf) r7 = r0 ; R0_w=map_value_or_null(id=2,off=0,ks=4,vs=136,imm=0) R7_w=map_value_or_null(id=2,off=0,ks=4,vs=136,imm=0)
; if(!counter) {
25: (15) if r7 == 0x0 goto pc+372 ; R7_w=map_value(off=0,ks=4,vs=136,imm=0)
; counter->n_evts++;
26: (79) r1 = *(u64 *)(r7 +0) ; R1_w=scalar() R7_w=map_value(off=0,ks=4,vs=136,imm=0)
27: (07) r1 += 1 ; R1_w=scalar()
28: (7b) *(u64 *)(r7 +0) = r1 ; R1_w=scalar() R7_w=map_value(off=0,ks=4,vs=136,imm=0)
; uint8_t *space = bpf_ringbuf_reserve(rb, event_size, 0);
29: (bf) r1 = r6 ; R1_w=map_ptr(off=0,ks=0,vs=0,imm=0) R6=map_ptr(off=0,ks=0,vs=0,imm=0)
30: (b7) r2 = 66 ; R2_w=66
31: (b7) r3 = 0 ; R3_w=0
32: (85) call bpf_ringbuf_reserve#131 ; R0=ringbuf_mem_or_null(id=4,ref_obj_id=4,off=0,imm=0) refs=4
33: (bf) r6 = r0 ; R0=ringbuf_mem_or_null(id=4,ref_obj_id=4,off=0,imm=0) R6_w=ringbuf_mem_or_null(id=4,ref_obj_id=4,off=0,imm=0) refs=4
; if(!space) {
34: (55) if r6 != 0x0 goto pc+7 42: R0=ringbuf_mem(ref_obj_id=4,off=0,imm=0) R6_w=ringbuf_mem(ref_obj_id=4,off=0,imm=0) R7=map_value(off=0,ks=4,vs=136,imm=0) R9=scalar() R10=fp0 fp-8=????mmmm refs=4
; return g_event_params_table[event_id];
42: (18) r1 = 0xffffc900015ba010 ; R1_w=map_value(off=16,ks=4,vs=248766,imm=0) refs=4
44: (71) r2 = *(u8 *)(r1 +353) ; R1_w=map_value(off=16,ks=4,vs=248766,imm=0) R2_w=4 refs=4
; ringbuf->payload_pos = sizeof(struct ppm_evt_hdr) + nparams * sizeof(uint16_t);
45: (bf) r7 = r2 ; R2_w=4 R7_w=4 refs=4
46: (67) r7 <<= 1 ; R7_w=8 refs=4
47: (7b) *(u64 *)(r10 -32) = r7 ; R7_w=8 R10=fp0 fp-32_w=8 refs=4
; ringbuf->payload_pos = sizeof(struct ppm_evt_hdr) + nparams * sizeof(uint16_t);
48: (07) r7 += 26 ; R7_w=34 refs=4
49: (b7) r1 = 20 ; R1_w=20 refs=4
50: (7b) *(u64 *)(r10 -24) = r2 ; R2_w=4 R10=fp0 fp-24_w=4 refs=4
; PUSH_FIXED_SIZE_TO_RINGBUF(ringbuf, param, sizeof(int64_t));
51: (2d) if r1 > r2 goto pc+1 ; R1_w=20 R2_w=4 refs=4
; return g_settings.boot_time;
53: (18) r1 = 0xffffc90001c8adb8 ; R1_w=map_value(off=3512,ks=4,vs=600281,imm=0) refs=4
55: (79) r8 = *(u64 *)(r1 +0) ; R1_w=map_value(off=3512,ks=4,vs=600281,imm=0) R8_w=scalar() refs=4
; hdr->ts = maps__get_boot_time() + bpf_ktime_get_boot_ns();
56: (85) call bpf_ktime_get_boot_ns#125 ; R0_w=scalar() refs=4
; hdr->ts = maps__get_boot_time() + bpf_ktime_get_boot_ns();
57: (0f) r0 += r8 ; R0_w=scalar() R8_w=scalar() refs=4
; hdr->ts = maps__get_boot_time() + bpf_ktime_get_boot_ns();
58: (bf) r1 = r0 ; R0_w=scalar(id=5) R1_w=scalar(id=5) refs=4
59: (77) r1 >>= 56 ; R1_w=scalar(umax=255,var_off=(0x0; 0xff)) refs=4
60: (73) *(u8 *)(r6 +7) = r1 ; R1_w=scalar(umax=255,var_off=(0x0; 0xff)) R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
61: (bf) r1 = r0 ; R0_w=scalar(id=5) R1_w=scalar(id=5) refs=4
62: (77) r1 >>= 48 ; R1_w=scalar(umax=65535,var_off=(0x0; 0xffff)) refs=4
63: (73) *(u8 *)(r6 +6) = r1 ; R1_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
64: (bf) r1 = r0 ; R0_w=scalar(id=5) R1_w=scalar(id=5) refs=4
65: (77) r1 >>= 40 ; R1_w=scalar(umax=16777215,var_off=(0x0; 0xffffff)) refs=4
66: (73) *(u8 *)(r6 +5) = r1 ; R1_w=scalar(umax=16777215,var_off=(0x0; 0xffffff)) R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
67: (bf) r1 = r0 ; R0_w=scalar(id=5) R1_w=scalar(id=5) refs=4
68: (77) r1 >>= 32 ; R1_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) refs=4
69: (73) *(u8 *)(r6 +4) = r1 ; R1_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
70: (bf) r1 = r0 ; R0_w=scalar(id=5) R1_w=scalar(id=5) refs=4
71: (77) r1 >>= 24 ; R1_w=scalar(umax=1099511627775,var_off=(0x0; 0xffffffffff)) refs=4
72: (73) *(u8 *)(r6 +3) = r1 ; R1_w=scalar(umax=1099511627775,var_off=(0x0; 0xffffffffff)) R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
73: (bf) r1 = r0 ; R0_w=scalar(id=5) R1_w=scalar(id=5) refs=4
74: (77) r1 >>= 16 ; R1_w=scalar(umax=281474976710655,var_off=(0x0; 0xffffffffffff)) refs=4
75: (73) *(u8 *)(r6 +2) = r1 ; R1_w=scalar(umax=281474976710655,var_off=(0x0; 0xffffffffffff)) R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
76: (73) *(u8 *)(r6 +0) = r0 ; R0_w=scalar(id=5) R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
77: (77) r0 >>= 8 ; R0_w=scalar(umax=72057594037927935,var_off=(0x0; 0xffffffffffffff)) refs=4
78: (73) *(u8 *)(r6 +1) = r0 ; R0_w=scalar(umax=72057594037927935,var_off=(0x0; 0xffffffffffffff)) R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
; hdr->tid = bpf_get_current_pid_tgid() & 0xffffffff;
79: (85) call bpf_get_current_pid_tgid#14 ; R0=scalar() refs=4
80: (b7) r1 = 1 ; R1_w=1 refs=4
; hdr->type = ringbuf->event_type;
81: (73) *(u8 *)(r6 +21) = r1 ; R1_w=1 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
82: (b7) r1 = 97 ; R1_w=97 refs=4
83: (73) *(u8 *)(r6 +20) = r1 ; R1_w=97 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
84: (b7) r1 = 0 ; R1_w=0 refs=4
; hdr->nparams = nparams;
85: (73) *(u8 *)(r6 +25) = r1 ; R1_w=0 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
86: (73) *(u8 *)(r6 +24) = r1 ; R1_w=0 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
87: (73) *(u8 *)(r6 +23) = r1 ; R1_w=0 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
; hdr->tid = bpf_get_current_pid_tgid() & 0xffffffff;
88: (73) *(u8 *)(r6 +15) = r1 ; R1_w=0 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
89: (73) *(u8 *)(r6 +14) = r1 ; R1_w=0 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
90: (73) *(u8 *)(r6 +13) = r1 ; R1_w=0 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
91: (73) *(u8 *)(r6 +12) = r1 ; R1_w=0 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
; hdr->len = ringbuf->reserved_event_size;
92: (73) *(u8 *)(r6 +19) = r1 ; R1_w=0 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
93: (73) *(u8 *)(r6 +18) = r1 ; R1_w=0 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
94: (73) *(u8 *)(r6 +17) = r1 ; R1_w=0 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
95: (b7) r1 = 66 ; R1_w=66 refs=4
96: (73) *(u8 *)(r6 +16) = r1 ; R1_w=66 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
; hdr->tid = bpf_get_current_pid_tgid() & 0xffffffff;
97: (bf) r1 = r0 ; R0=scalar(id=6) R1_w=scalar(id=6) refs=4
98: (77) r1 >>= 24 ; R1_w=scalar(umax=1099511627775,var_off=(0x0; 0xffffffffff)) refs=4
99: (73) *(u8 *)(r6 +11) = r1 ; R1_w=scalar(umax=1099511627775,var_off=(0x0; 0xffffffffff)) R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
100: (bf) r1 = r0 ; R0=scalar(id=6) R1_w=scalar(id=6) refs=4
101: (77) r1 >>= 16 ; R1_w=scalar(umax=281474976710655,var_off=(0x0; 0xffffffffffff)) refs=4
102: (73) *(u8 *)(r6 +10) = r1 ; R1_w=scalar(umax=281474976710655,var_off=(0x0; 0xffffffffffff)) R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
103: (73) *(u8 *)(r6 +8) = r0 ; R0=scalar(id=6) R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
104: (77) r0 >>= 8 ; R0_w=scalar(umax=72057594037927935,var_off=(0x0; 0xffffffffffffff)) refs=4
105: (73) *(u8 *)(r6 +9) = r0 ; R0_w=scalar(umax=72057594037927935,var_off=(0x0; 0xffffffffffffff)) R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
; hdr->nparams = nparams;
106: (79) r1 = *(u64 *)(r10 -24) ; R1_w=4 R10=fp0 fp-24=4 refs=4
107: (73) *(u8 *)(r6 +22) = r1 ; R1_w=4 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
; PUSH_FIXED_SIZE_TO_RINGBUF(ringbuf, param, sizeof(int64_t));
108: (bf) r1 = r6 ; R1_w=ringbuf_mem(ref_obj_id=4,off=0,imm=0) R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
109: (0f) r1 += r7 ; R1_w=ringbuf_mem(ref_obj_id=4,off=34,imm=0) R7=34 refs=4
110: (bf) r2 = r9 ; R2_w=scalar(id=7) R9=scalar(id=7) refs=4
111: (77) r2 >>= 48 ; R2_w=scalar(umax=65535,var_off=(0x0; 0xffff)) refs=4
112: (73) *(u8 *)(r1 +6) = r2 ; R1_w=ringbuf_mem(ref_obj_id=4,off=34,imm=0) R2_w=scalar(umax=65535,var_off=(0x0; 0xffff)) refs=4
113: (bf) r2 = r9 ; R2_w=scalar(id=7) R9=scalar(id=7) refs=4
114: (77) r2 >>= 56 ; R2_w=scalar(umax=255,var_off=(0x0; 0xff)) refs=4
115: (73) *(u8 *)(r1 +7) = r2 ; R1_w=ringbuf_mem(ref_obj_id=4,off=34,imm=0) R2_w=scalar(umax=255,var_off=(0x0; 0xff)) refs=4
116: (bf) r2 = r9 ; R2_w=scalar(id=7) R9=scalar(id=7) refs=4
117: (77) r2 >>= 32 ; R2_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) refs=4
118: (73) *(u8 *)(r1 +4) = r2 ; R1_w=ringbuf_mem(ref_obj_id=4,off=34,imm=0) R2_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) refs=4
119: (bf) r2 = r9 ; R2_w=scalar(id=7) R9=scalar(id=7) refs=4
120: (77) r2 >>= 40 ; R2_w=scalar(umax=16777215,var_off=(0x0; 0xffffff)) refs=4
121: (73) *(u8 *)(r1 +5) = r2 ; R1_w=ringbuf_mem(ref_obj_id=4,off=34,imm=0) R2_w=scalar(umax=16777215,var_off=(0x0; 0xffffff)) refs=4
122: (bf) r2 = r9 ; R2_w=scalar(id=7) R9=scalar(id=7) refs=4
123: (77) r2 >>= 16 ; R2_w=scalar(umax=281474976710655,var_off=(0x0; 0xffffffffffff)) refs=4
124: (73) *(u8 *)(r1 +2) = r2 ; R1_w=ringbuf_mem(ref_obj_id=4,off=34,imm=0) R2_w=scalar(umax=281474976710655,var_off=(0x0; 0xffffffffffff)) refs=4
125: (bf) r2 = r9 ; R2_w=scalar(id=7) R9=scalar(id=7) refs=4
126: (77) r2 >>= 24 ; R2_w=scalar(umax=1099511627775,var_off=(0x0; 0xffffffffff)) refs=4
127: (73) *(u8 *)(r1 +3) = r2 ; R1_w=ringbuf_mem(ref_obj_id=4,off=34,imm=0) R2_w=scalar(umax=1099511627775,var_off=(0x0; 0xffffffffff)) refs=4
128: (73) *(u8 *)(r1 +0) = r9 ; R1_w=ringbuf_mem(ref_obj_id=4,off=34,imm=0) R9=scalar(id=7) refs=4
129: (77) r9 >>= 8 ; R9_w=scalar(umax=72057594037927935,var_off=(0x0; 0xffffffffffffff)) refs=4
130: (73) *(u8 *)(r1 +1) = r9 ; R1_w=ringbuf_mem(ref_obj_id=4,off=34,imm=0) R9_w=scalar(umax=72057594037927935,var_off=(0x0; 0xffffffffffffff)) refs=4
131: (b7) r1 = 8 ; R1_w=8 refs=4
132: (6b) *(u16 *)(r6 +26) = r1 ; R1_w=8 R6=ringbuf_mem(ref_obj_id=4,off=0,imm=0) refs=4
133: (18) r1 = 0x1 ; R1_w=1 refs=4
; if(bpf_core_enum_value_exists(enum bpf_func_id, BPF_FUNC_get_current_task_btf) &&
135: (15) if r1 == 0x0 goto pc+5 ; R1_w=1 refs=4
136: (18) r1 = 0x9e ; R1_w=158 refs=4
; if(bpf_core_enum_value_exists(enum bpf_func_id, BPF_FUNC_get_current_task_btf) &&
138: (55) if r1 != 0x9e goto pc+2 ; R1_w=158 refs=4
; return (struct task_struct *)bpf_get_current_task_btf();
139: (85) call bpf_get_current_task_btf#158 ; R0=trusted_ptr_task_struct(off=0,imm=0) refs=4
140: (05) goto pc+1
;
142: (bf) r7 = r0 ; R0=trusted_ptr_task_struct(off=0,imm=0) R7_w=trusted_ptr_task_struct(off=0,imm=0) refs=4
143: (18) r1 = 0x1 ; R1_w=1 refs=4
145: (79) r8 = *(u64 *)(r10 -32) ; R8_w=8 R10=fp0 fp-32=8 refs=4
146: (79) r9 = *(u64 *)(r10 -24) ; R9_w=4 R10=fp0 fp-24=4 refs=4
; READ_TASK_FIELD_INTO(&cap_struct, task, cred, cap_inheritable);
147: (15) if r1 == 0x0 goto pc+7 ; R1_w=1 refs=4
148: (18) r1 = 0x9e ; R1_w=158 refs=4
; READ_TASK_FIELD_INTO(&cap_struct, task, cred, cap_inheritable);
150: (55) if r1 != 0x9e goto pc+4 ; R1_w=158 refs=4
; READ_TASK_FIELD_INTO(&cap_struct, task, cred, cap_inheritable);
151: (79) r1 = *(u64 *)(r7 +1984) ; R1_w=rcu_ptr_or_null_cred(id=8,off=0,imm=0) R7_w=trusted_ptr_task_struct(off=0,imm=0) refs=4
152: (79) r1 = *(u64 *)(r1 +48)
R1 invalid mem access 'rcu_ptr_or_null_'
processed 146 insns (limit 1000000) max_states_per_insn 0 total_states 7 peak_states 7 mark_read 5
-- END PROG LOAD LOG --
Mon Nov 4 09:58:02 2024: [libs]: libbpf: prog 'capset_x': failed to load: -13
Mon Nov 4 09:58:02 2024: [libs]: libbpf: failed to load object 'bpf_probe'
Mon Nov 4 09:58:02 2024: [libs]: libbpf: failed to load BPF skeleton 'bpf_probe': -13
Mon Nov 4 09:58:02 2024: [libs]: libpman: failed to load BPF object (errno: 13 | message: Permission denied)
Mon Nov 4 09:58:02 2024: An error occurred in an event source, forcing termination...
Error: Initialization issues during scap_init
Events detected: 0
Rule counts by severity:
Triggered rules by rule name: |
|
@PierreBart thank you very much! this is an issue we already fixed in dev falcosecurity/libs#2118 let me check with other maintainers what we can do @falcosecurity/falco-maintainers |
|
Since lots of people are having the issue, my 2c is to definitely make a patch release of libs (0.18.2) and then a patch release for Falco 0.39 (0.39.2). |
|
I agree, I'm still investigating the fedora issue also reported here by @tiny-pangolin (#3323 (comment)), it would be great to have both of them in the patch |
|
Same issue on almalinux 8.7 hosts, no kubernetes. Some hosts fail to restart after making a config change or whenever I make a tweak in one of the rules files. After a while they do succeed to startup again. |
|
Uhm interesting, I would try to separate this issue into several separate ones:
More, in general, to understand under which category you fall please enable the libs logger:
libs_logger:
enabled: true
severity: debug
sudo ./usr/bin/falco -c ./etc/falco/falco.yaml -r ./etc/falco/falco_rules.yaml -o libs_logger.enabled=true -o libs_logger.severity=debug
|
|
@Andreagit97 a possible patch release was mentioned. Any news on that becoming patch becoming available anytime soon? |
|
We are still investigating the
issue. I'd say that we can expect a Falco patch release in a couple of weeks; sorry for the delay! |
|
/milestone 0.39.2 |
|
ei @tordenist issues 1 and 2 reported here (#3323 (comment)) are solved in dev. It would be great to understand also the third one before releasing a patch release, if you are experiencing the third issue could you please provide the logs required above (@OneideLuizSchneider you are the initial reporter of the restart issue, could you please provide additional logs as suggested above?) If we cannot reproduce the third issue, we may release just the fixes for the first 2. Maybe the third one is just an unhappy consequence of the first 2 but it would be great to understand it |
|
@OneideLuizSchneider as andrea said, can you enable libs logging and send us some logs? |
|
@FedeDP @Andreagit97 I'm using the image= I did test it on:
|
|
is there somewhere I can post debug logs to? the full run between setting is about 31000 lines |
|
ei @tiny-pangolin you can upload a txt file here on the issue or you can create a gists as you prefer |
|
Falco 0.39.2 is out, feel free to test it! |
Regarding my issue, it turns out the vm's that failed to restart falco didn't have enough free memory to start Falco. |
|
0.39.2 fixes the issue for me, thanks @FedeDP and @Andreagit97 for your help! |
|
@FedeDP @Andreagit97 |
I'm not the original person, but ran into this issue as well on self hosted Talos Linux nodes (version 1.8.1, kubernetes version 1.31.2). Noticed a pattern that falco ran on nodes without secureboot enabled, but on nodes with secureboot I was getting this error message. May be a separate issue, but here are debug logs from a good node and a bad node. |
|
@NachoxMacho this sounds like another issue so i opened it here #3416 |
|
Hello, I have the same error type. Bests regards, |
|
Thank you for reporting! This is issue describes multiple restart of Falco before a successful run, if you are facing other issues (like a verifier failure) please open a new github issue |
|
Unfortunately, upgrading from 0.38.2 to 0.39.2 does not fix the issue for me... Edit: Adding version |
|
ei @Jmorkcho can you share the Falco debug logs as explained here #3323 (comment)? |
|
The full log is here: |
it seems you have not enough memory to deploy Falco on that node |
|
Are there documented minimum requirements for running falco? I was planning on running falco on small virtual machines with 2 vcpus 4gb of memory |
|
^ I'm wondering the same as well, since at the time of starting the pod, there is almost 1GB free memory on the node, yet I receive the CrashLoopBackOff/scap_init error.
|
This comment has been minimized.
This comment has been minimized.
@devasmith This is pretty strange Falco with modern ebpf requires by default 8 MB for each 2 CPUs. resources:
limits:
cpu: 1000m
memory: 1024Mi
requests:
cpu: 100m
memory: 512MiBTW yes, using buf_size_preset and cpus_for_each_buffer as knobs is a great strategy to calibrate Falco memory usage. @tiny-pangolin @Jmorkcho could you try the above 2 configs and see if you can find a fit for your environment? |
Thanks for the quick reply! I had actually previously lowered the resource limits so that was why I hit it.. But maybe this help anyone else.. :) |
|
Hello, @devasmith and @Andreagit97, I've adjusted my config like this: In order to stop another problem that we experience in our environment, namely dropped syscalls and in this article I've found that increasing the buf_size_preset might fix the problem. I've also removed the resource limitations, but there are still some pods that will experience the scap_init problem, restart X amount of times and then run successfully. |
Describe the bug
After the POD restarted 8 times it worked.
ERROR:
Error: Initialization issues during scap_initJust Install it, details are below.
Expected behaviour
it should not need to restart to able to work
Screenshots
Environment
Falco version: 0.38.2 (x86_64)
Linux version 5.10.223-212.873.amzn2.x86_64 (mockbuild@ip-10-0-60-177) (gcc10-gcc (GCC) 10.5.0 20230707 (Red Hat 10.5.0-1), GNU ld version 2.35.2-9.amzn2.0.1) Digwatch compiler #1 SMP Wed Aug 7 16:53:32 UTC 2024
AWS Linux 2
5.10
EKS 1.29
values.yaml->Additional context
I saw many other folks reporting this here, but it's not clear why this happened and how to fix it if there is a fix.
The text was updated successfully, but these errors were encountered: