Logging infrastructure and configuration updates

[vkirkl]

Begin the migration from ad hoc, "printed", output to systematic and consistently formatted log messages that can be centrally aggregated from all Wildbook instances for searching, summarizing, and alerting with an "off the shelf" solution of Alloy + Loki + Grafana. High priority areas like API access, database access, interactions with ml services, and lossy operations will be migrated first. Flag certain actions that modify core objects (like encounters) as audited and update the object's "audit trail".

To do:

• Logging infrastructure

  • pom file updates, including log4j2 and alloy install and configuration
  • configs for log4j2 and alloy
  • dockerfile updates
  • loki install and configuration
  • grafana configuration: data source and dashboard
  • example LogQL query documentation
  • install and configuration documentation
  • standard for logging in code and code update examples; INFO / WARN / ERROR standards

• Logging code update priorities (migrations from System.out.println and new logging)

  • API / auth
  • Opensearch interactions
  • Shepherd
  • WBIA / ml-service calls
  • destructive actions (deletes, updates)

• Encounter update auditing

  • Redesign the encounter audit trail / "researcher comments" -> perhaps from a string to structured data?
  • Update as a logging action

• Other activities

  • Determine who or what is using slf4j? Can it be cleaned out entirely?
  • Other observability research into opentelemetry / Tempo

[vkirkl]

During Loki and Grafana work: consider migration of all shared Grafana-related services (incl. Prometheus) a small dedicated Azure VM.