Structured logging

[serathius]

Enhancement Description

Define standard structure for Kubernetes log messages, add methods to klog to enforce this structure, add ability to configure Kubernetes components to produce logs in JSON format and initiate migration to structured logging.

• Kubernetes Enhancement Proposal: Structured Logging KEP
• Primary contact (assignee): @pohly
• Responsible SIGs: sig-instrumentation
• Enhancement target (which target equals to which milestone):
  • Alpha release target 1.19
  • Beta release target 1.23
  • GA release target 1.27

Alpha

• Implement text format by adding klog.InfoS, klog.ErrorS functions to k8s.io/klog (Implement InfoS and ErrorS functions klog#125, @yuzhiquan)

• Implement klog.KObj, klog.KRef to k8s.io/klog (Implement klog.KObj klog.KRef functions klog#127, @tahsinrahman)

• Upgrade klog to v2 (Upgrade to klogv2 kubernetes#89961. @dims)

• Implement --logging-format flag in k8s.io/component-base (Implement --logging-format flag in k8s.io/component-base kubernetes#89190, @yuzhiquan)

• Prepare migration instructions (@serathius, Add instructions for structured logging migration community#4793)

• Migrate selected log messages to klog.InfoSusing Migration Instructions

  • staging/src/k8s.io/apiserver (httplog.go:161, get.go:251, watcher.go:363, watcher.go:256) (@AkashSarda, Structured logging for Apiserver [Issue #1602 Structured logging] kubernetes#91833)
  • pkg/controller/ (garbagecollector.go:404, garbagecollector.go:517, controller_utils.go:603, replica_set.go:561, deployment_controller.go:484, deployment_controller.go:575, endpoints_controller.go:340, cloud_cidr_allocator.go:233, ttl_controller.go:271, replica_set.go:597, replica_set.go:225) (@tahsinrahman, Migrate to log calls to klog.InfoS and klog.ErroS for pkg/controller kubernetes#91576)
  • /plugin/pkg/auth/authorizer (node_authorizer.go:197) (@ankeesler, Migrate a single node_authorizer.go klog.Infof call to klog.InfoS kubernetes#91591)
  • pkg/scheduler (scheduler.go:604, factory.go:431, scheduler.go:715) (@tanjunchen, Migrate to log calls to klog.ErroS for pkg/scheduler kubernetes#91679)
  • pkg/kubelet (kubelet_getters.go:173) (@hase1128, Migrate klog to klog.InfoS in pkg/kubelet kubernetes#91624)
  • staging/src/k8s.io/client-go (event.go:278) (@KobayashiD27, Migrate log to klog.InfoS for staging/src/k8s.io/client-go kubernetes#91712)

• Implement json log format (@yuzhiquan, Implement json log format  kubernetes#91490)

• Update flag help to list both format options (@serathius, Update logging format flag help kubernetes#92177)

• Add --logging-format flag to kubernetes components

  • kube-apiserver (Add --logging-format flag for kube-apiserver kubernetes#91501, @tahsinrahman)
  • kube-controller-manager (Add '--logging-format' flag to kube-controller-manager kubernetes#91521, @SataQiu)
  • kubelet (add --logging-format flag to kubelet kubernetes#91532, @afrouzMashaykhi)
  • kube-scheduler (Add '--logging-format' flag to kube-scheduler kubernetes#91522, @SataQiu)

• Write documentation for new logging format (Document Structured Logging Enhancement website#21202, @serathius)

Follow-up from main tasks:

• ErrorS(nil, ...) should call loggr.Error(nil, ...) instead of loggr.Info(...) klog#153 (@tahsinrahman)
• Fix InfoS serializing Stringer subfields klog#156 (@yuzhiquan)
• InfoS uses %#s for errors instead of %q klog#158 (@tahsinrahman)
• Update klog in k/k to collect bug fixes above (@serathius, Upgrade klog to v2.1.0 kubernetes#91792)
• Structured logging behaviour changed from v2.0.0 to v2.1.0 klog#165 (@physcat)
• Update klog to collect bug fixes above (@serathius, Update k8s.io/klog to v2.2.0 kubernetes#92554)
• Remove locking from logFormatRegistry as it's not needed (@rahulchheda, Removed mutex thread locking and unlocking from logging Format registry kubernetes#92347)
• Validate if incompatible klog flags were used (@rahulchheda, (feat) Add validation for Unsupported Loggings Flags for component-base kubernetes#92394)
• Fix JSON logger verbosity (@yuzhiquan, Fix JSON logger verbose kubernetes#92788)

Beta

• Directories can be marked as migrated, preventing regression via verify script in hack. (Prevent regressions after structured logging migration kubernetes#98975, @adisky)
• Kubernetes Logging documentation is updated for structured logging. (Update Logging Conventions doc with structured logging community#5593)
• Structured logging interface design is verify by migrating one whole component (Kubelet) to structured logging. (Kubelet migration to structured logs kubernetes#98976)
• Log volume (i.e. size of output) will be benchmarked and documented for kubelet in non-JSON and JSON format. (Benchmark kubelet log sizes before and after structured log migration kubernetes#99803)
• Dedicated periodic e2e test scenario with Json enabled (Add an e2e test scenario with json logging enabled for kubelet kubernetes#99802)
• Refactor Options to use LoggingConfiguration (Move Logs options to component-base/config kubernetes#99267)
• Json format should support same set of feature flags as text format, minus those we will decide to deprecte. (Json format should support same set of feature flags as klog kubernetes#99270)
  • Add source code location to JSON logs (Add source code location information to JSON logs kubernetes#102353)
• Move klog flags to Logs flag group (Move not-deprecated klog flags to Logs flag group kubernetes#99265)
• Enabling Json format should work from component start (Kubelet logs start with unstructured logs then transition to structured logs kubernetes#100152)

• [] Beta (1.35)
  • [] KEP (k/enhancements) update PR(s): KEP-1602, KEP-3077: structured, contextual logging in 1.35 #5621
  • Code (k/k) update PR(s): will not be tracked individually here because there is no "required" PR for this feature, just several optional ones
  • Docs (k/website) update(s): none

Follow ups:

• Migrate trace.go)

Please let me know if someone is interested in working on one of the tasks.

[serathius]

/sig instrumentation

[serathius]

/help

[k8s-ci-robot]

@serathius:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[yuzhiquan]

interested in

[tanjunchen]

/cc

[serathius]

/milestone v1.19

[k8s-ci-robot]

@serathius: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[justaugustus]

/milestone v1.19