We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
https://help.aliyun.com/document_detail/112903.html?spm=a2c4g.11186623.6.817.626f175bMJTh5c
bk-log-receiver 实现接收日志协议格式包括
节点管理
resource = Resource.create({ "service.name": settings.APP_CODE, "bk_data_id": 1 }), )
/v1/push/ { "bk_data_id": 113, "source": "xxxx", 来源ip或者其他 "logs": ["xxx", "xxx"] }
type LogMessage struct { Timestamp uint64 MessageLen uint32 Message []byte } // LogPack log package define type LogPack struct { PkgLen uint32 // 包总长度 (标识符不计算在内) PkgVer uint8 // 包版本号,0x02 SourceLen uint8 // 来源长度 Source []byte // 来源字符串 BkDataId int // bkdataid Compress uint8 // 压缩选项,0x00 无,0x01 gzip Messages []*LogMessage // 消息体(若 compress!=0x00 则为压缩后的 buffer) }
{ "bk_data_id": 113, 蓝鲸dataid "source": "xxxx", 来源ip "logs": ["xxx", "xxx"], 日志数组 "version": "v1", 协议版本 }
max_message_size: 512kb queue_limit: 50000 ip_limit: true
- type: otlp-trace dataid: 1111 - type: otlp-logs dataid: 1111 - type: http dataid: 1111 - type: udp dataid: 2222 - type: tcp dataid: 22223
相关trace采样 https://docs.newrelic.com/docs/distributed-tracing/concepts/how-new-relic-distributed-tracing-works/ trace数据量大 全量大数据上报对链路&存储都是巨大的压力
原始SDK采样配置 原始sdk会在入口侧进行采样判断,然后后续链路则不进行上报操作 但是会存在错误trace 触发采样导致无法上报
接收侧采样
otlp官方实现 https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/loadbalancingexporter/README.md
ulimit -n 1040000
测试条件
限制单核使用
工具 ghz
机器
8核心16G
CPU MHz: 2595.124
机器配置
8核
gse上报
自研云机器
测试工具 ab
主要消耗在json反序列上面
The text was updated successfully, but these errors were encountered:
liuwenping
zzhutianyu
No branches or pull requests
日志自定义上报方案(Logs Trace)
阿里云
日志上报
https://help.aliyun.com/document_detail/112902.html?spm=a2c4g.11186623.6.816.21329557cV5cMr
https://help.aliyun.com/document_detail/112903.html?spm=a2c4g.11186623.6.817.626f175bMJTh5c
腾讯云
https://cloud.tencent.com/document/product/614/12445
蓝鲸日志平台
架构设计
模块说明
bk-log-receiver
实现接收日志协议格式包括
通过以上上报协议接收日志或者Trace并通过协议内携带的dataid进行上报
读取平台下发自定义上报dataid,不符合的dataid进行丢弃
节点管理
上报协议
上报需要在resource携带bk_data_id
python如下所示
上报集群部署
上报服务配置
主配置
子配置
集群保护(QOS)
otlp-trace 采样&染色能力
采样
相关trace采样
https://docs.newrelic.com/docs/distributed-tracing/concepts/how-new-relic-distributed-tracing-works/
trace数据量大 全量大数据上报对链路&存储都是巨大的压力
原始SDK采样配置
原始sdk会在入口侧进行采样判断,然后后续链路则不进行上报操作
但是会存在错误trace 触发采样导致无法上报
接收侧采样
otlp官方实现
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/loadbalancingexporter/README.md
染色能力
需要提供SDK拉取染色配置,需要平台方提供SDK去拉取对应的染色配置
需要实现接收端多机分布下的数据汇聚能力,将trace进行采样豁免使其能够进行稳定上报
负载均衡方案
Saas配置
系统调优
ulimit -n 1040000
otlp-grpc 压测报告
测试条件
限制单核使用
工具 ghz
机器
8核心16G
CPU MHz: 2595.124
机器配置
8核
gse上报
Otlp-http-json 压测报告
机器配置
8核
自研云机器
gse上报
测试工具 ab
主要消耗在json反序列上面
Otlp-http-proto 压测报告
The text was updated successfully, but these errors were encountered: