Logs are mixed

[mmatczuk]

I requested scylla logs got agent logs and scylla logs

michal@lh:scylla-operator-eval/aws (master)$ kubectl logs -n scylla scylla-cluster-eu-central-1-eu-central-1a-0 scylla
{"L":"INFO","T":"2020-05-29T10:55:23.731Z","M":"sidecar started","version":"0.1.6","build_date":"2020-04-30T08:16:18Z","commit":"9858a394f3f771a28f9cfe829a9a85e33ba0d4d3","built_by":"goreleaser","go_version":"go version go1.14.2 linux/amd64","options":{"Name":"scylla-cluster-eu-central-1-eu-central-1a-0","Namespace":"scylla","LogLevel":"info","CPU":"1"},"_trace_id":"_PITLRCPRyeQuG8y4PT_AQ"}
{"L":"INFO","T":"2020-05-29T10:55:23.788Z","M":"Registering Components.","_trace_id":"_PITLRCPRyeQuG8y4PT_AQ"}
{"L":"INFO","T":"2020-05-29T10:55:23.809Z","M":"Member loaded","member":{"Name":"scylla-cluster-eu-central-1-eu-central-1a-0","Namespace":"scylla","IP":"192.168.10.45","StaticIP":"10.100.53.52","Rack":"eu-central-1a","Datacenter":"eu-central-1","Cluster":"scylla-cluster"},"_trace_id":"Wngx1yi3TEOj9RjqsDc1cg"}
{"L":"INFO","T":"2020-05-29T10:55:23.859Z","N":"member_controller","M":"Setting up HTTP Checks...","_trace_id":"Wngx1yi3TEOj9RjqsDc1cg"}
{"L":"INFO","T":"2020-05-29T10:55:23.859Z","N":"member_controller","M":"Setting up config files","_trace_id":"Wngx1yi3TEOj9RjqsDc1cg"}
{"L":"INFO","T":"2020-05-29T10:55:23.859Z","N":"member_controller","M":"Setting up scylla.yaml","_trace_id":"Wngx1yi3TEOj9RjqsDc1cg"}
{"L":"INFO","T":"2020-05-29T10:55:23.859Z","N":"member_controller","M":"no scylla.yaml config map available"}
{"L":"INFO","T":"2020-05-29T10:55:23.860Z","N":"member_controller","M":"Setting up cassandra-rackdc.properties","_trace_id":"Wngx1yi3TEOj9RjqsDc1cg"}
{"L":"INFO","T":"2020-05-29T10:55:23.860Z","N":"member_controller","M":"unable to read properties","file":"/mnt/scylla-config/cassandra-rackdc.properties"}
{"L":"INFO","T":"2020-05-29T10:55:23.861Z","N":"member_controller","M":"Setting up entrypoint script","_trace_id":"Wngx1yi3TEOj9RjqsDc1cg"}
{"L":"INFO","T":"2020-05-29T10:55:23.871Z","N":"member_controller","M":"Scylla entrypoint","command":"/docker-entrypoint.py --listen-address=192.168.10.45 --broadcast-address=10.100.53.52 --broadcast-rpc-address=10.100.53.52 --seeds=10.100.53.52 --developer-mode=0 --smp=1 --cpuset=0-1","_trace_id":"Wngx1yi3TEOj9RjqsDc1cg"}

[dahankzter]

You get both containers logs like that. You can supply another argument specifying which container to display.

[mmatczuk]

See it's scylla-cluster-eu-central-1-eu-central-1a-0 scylla if you do not provide container k8s complains. How do I see the scylla logs only?

[dahankzter]

aha yes it's the controller not the agent. It actually runs in the same container. I have been hesitant to move it out in a separate container but perhaps it's better after all.

[dahankzter]

This is the process hierarchy in the scylla container:

root           1       0  0 11:54 ?        00:00:00 /mnt/shared/tini -- /mnt/shared/scylla-operator sidecar
root           6       1  0 11:54 ?        00:00:01 /mnt/shared/scylla-operator sidecar
root          17       6  0 11:54 ?        00:00:00  \_ /opt/scylladb/python3/bin/python3 /opt/scylladb/python3/bin/../libexec/python3.7.bin -s /docker-entrypoint.py --listen-address=172.18.0.5 --broadcast-address=10.102.130.128 --broadca
root          37      17  0 11:54 ?        00:00:00      \_ /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
root          40      37  2 11:54 ?        00:00:05          \_ /opt/scylladb/jmx/symlinks/scylla-jmx -Xmx256m -XX:+UseSerialGC -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremo
root          41      37  1 11:54 ?        00:00:03          \_ /usr/bin/scylla --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --developer-mode=1 --smp 1 --overprovisioned --listen-address 172.18.0.5 --
root          42      37  0 11:54 ?        00:00:00          \_ /bin/bash /node-exporter-service.sh
root          47      42  0 11:54 ?        00:00:00          |   \_ /usr/bin/node_exporter --collector.interrupts
root          43      37  0 11:54 ?        00:00:00          \_ /bin/bash /sshd-service.sh
root          84      43  0 11:54 ?        00:00:00          |   \_ /usr/sbin/sshd -D
root          44      37  0 11:54 ?        00:00:00          \_ /bin/bash /scylla-housekeeping-service.sh
root          93      44  0 11:54 ?        00:00:00              \_ sleep 1d

I agree we should really clean it up as much as possible but right now the operator sidecar is the process that actually launches scylla.

[scylla-operator-bot]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out

/lifecycle stale

[scylla-operator-bot]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out

/lifecycle rotten

[tnozicka]

The split is tracked in #1940