Skip to content

feat: 完善清洗模板字段处理 - #12027

Merged
yiqiwang-17 merged 7 commits into
TencentBlueKing:feat/clean_templatefrom
JunLong-Lin:fix/clean-template-etl-fields
Aug 19, 2026
Merged

feat: 完善清洗模板字段处理#12027
yiqiwang-17 merged 7 commits into
TencentBlueKing:feat/clean_templatefrom
JunLong-Lin:fix/clean-template-etl-fields

Conversation

@JunLong-Lin

Copy link
Copy Markdown
Collaborator

No description provided.

@JunLong-Lin JunLong-Lin added project/log project log feat A new feature. Correlates with MINOR in SemVer labels Aug 18, 2026
@JunLong-Lin JunLong-Lin changed the title fix: 完善清洗模板字段处理 feat: 完善清洗模板字段处理 Aug 18, 2026

@yiqiwang-17 yiqiwang-17 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

整体方向认可:etl_storage/base.py:1064 确实是 field["is_dimension"] 硬取,历史模板缺该键会 500;模板 serializer 从自由 DictField 收紧也能挡住脏数据。

但有 3 条 P1 建议修完再合,已按位置发在对应行:

  1. 兜底只补了 is_dimension,同一批历史模板缺 is_delete 等键时仍会 500(transfer.py)。
  2. etl_params 换成 Serializer 后会把 DRF 默认值写进模板,其中 record_parse_failure 会被强制置 True,与采集项侧联动语义相反(serializers.py:1177)。
  3. etl_fields 校验收紧属不兼容变更,且本次收紧零测试覆盖(serializers.py:1178)。

另外 PR 描述为空,建议把上述行为变化(尤其是模板 etl_params 持久化内容的变化)补进描述,方便后续回溯。

Comment thread bklog/apps/log_databus/handlers/etl/transfer.py Outdated
Comment thread bklog/apps/log_databus/serializers.py
Comment thread bklog/apps/log_databus/serializers.py
@yiqiwang-17
yiqiwang-17 merged commit 1c12685 into TencentBlueKing:feat/clean_template Aug 19, 2026
5 checks passed
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (feat/clean_template@79adaa9). Learn more about missing BASE report.

Additional details and impacted files
@@                  Coverage Diff                   @@
##             feat/clean_template   #12027   +/-   ##
======================================================
  Coverage                       ?   61.83%           
======================================================
  Files                          ?      927           
  Lines                          ?    68172           
  Branches                       ?        0           
======================================================
  Hits                           ?    42156           
  Misses                         ?    26016           
  Partials                       ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat A new feature. Correlates with MINOR in SemVer project/log project log

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants