Skip to content

#540 Kubernetes 관측성 개선 - #707

Merged
Neibce merged 147 commits into
developfrom
540/observability-improvements
Jun 19, 2026
Merged

#540 Kubernetes 관측성 개선#707
Neibce merged 147 commits into
developfrom
540/observability-improvements

Conversation

@Neibce

@Neibce Neibce commented Jun 19, 2026

Copy link
Copy Markdown
Member

Changelog

  • Kubernetes 운영 환경용 관측성 구성을 추가했습니다.
    • Prometheus scrape 대상, alert rule, Alertmanager routing
    • Grafana dashboards
    • Loki/Alloy 로그 수집
    • Tempo/OTel Collector trace 수집
  • backend에 /metrics, JSON request log, request ID 처리를 추가했습니다.
  • frontend/API 요청에 X-Request-ID 전파를 추가했습니다.
  • Sentry 설정을 환경변수 기반으로 정리했습니다.
  • 제출, 채점, AI 힌트 경로의 운영 metric을 추가했습니다.

Testing

  • Backend 영향권 테스트 통과: 158 tests OK
  • Frontend lint/build 성공
  • Kubernetes dev/prod/monitoring kustomize build 성공
  • Grafana dashboard JSON/YAML parse 확인
  • GitHub CodeQL checks pass

Ops Impact

  • DB migration 없음
  • API contract 변경 없음
  • Kubernetes monitoring CRD가 필요합니다.
  • Alertmanager webhook secret 주입이 필요합니다.
  • Swarm monitoring 구성은 변경하지 않습니다.

Version Compatibility

  • 기존 API/DB schema와 호환됩니다.

Comment thread backend/account/tests.py Fixed
Comment thread .github/workflows/observability-validation.yml Fixed
Comment thread backend/utils/celery_observability.py Fixed
Comment thread backend/utils/observability_tracing.py Dismissed
Comment thread backend/utils/observability_tracing.py Dismissed
Comment thread backend/utils/tests.py Fixed
@Neibce
Neibce force-pushed the 540/observability-improvements branch from 6055e7b to 582d02c Compare June 19, 2026 18:15
@Neibce Neibce changed the title 540/observability improvements #540 Kubernetes 관측성 개선 Jun 19, 2026
@Neibce
Neibce marked this pull request as ready for review June 19, 2026 19:41
@Neibce
Neibce merged commit dd8ec30 into develop Jun 19, 2026
6 checks passed
@Neibce
Neibce deleted the 540/observability-improvements branch June 19, 2026 19:44

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 87337442b0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

).only("judge_start_time", "judge_end_time")
for submission in submissions:
durations.append(max((submission.judge_end_time - submission.judge_start_time).total_seconds(), 0))
metric.add_metric([], cache.llen(CELERY_BROKER_QUEUE_KEY) or 0)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Read the Celery queue from the broker DB

Under the current Redis settings, Django's default cache points at db 0 (redis_config() uses the default cache DB) while the Celery broker/result backend use db 4, so cache.llen("celery") is checking the wrong Redis database. In Redis/Sentinel deployments the new /metrics value will stay at zero or reflect an unrelated key, which means the Celery backlog dashboard/alert will not catch queued judge tasks; query the Celery broker connection/db instead.

Useful? React with 👍 / 👎.

@Neibce
Neibce restored the 540/observability-improvements branch June 20, 2026 07:49
@Neibce
Neibce deleted the 540/observability-improvements branch June 20, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants