Skip to content

Commit 570bdb6

Browse files
zero3233-bdtpfzCoda-botdsf86tpfz
authored
[feat]:automation task (#133)
* fix * add logs * change code frame * code change * add debug log * add debug log * add debug log * fix time * add run_id * add run_id * fix callback * fix time * fix wire gen * fix * fix * fix * feat: [Coda] add detailed logging to track context cancellation reasons in backfill.go (LogID: 20250925170706010091109004992A774) Co-Authored-By: Coda <[email protected]> * fix * fix panic * add SearchDatasets * fix SearchDatasets(ctx context.Context, workspaceID in * fix RawSpanConvertToLoopSpan * add debuglog * feat: [Coda] implement FilterFieldsDO2DTO conversion function (LogID: 20250928114148010091110241630A99A) Co-Authored-By: Coda <[email protected]> * feat: [Coda] implement combineFilters function in task service (LogID: 20250928114148010091110241630A99A) Co-Authored-By: Coda <[email protected]> * add coze proc * add wire * fix: [Coda] handle SubFilter in task repo filter processing (LogID: 20250928114148010091110241630A99A) Co-Authored-By: Coda <[email protected]> * fix panic * fix debug log * fix status * fix coze proc * fix * feat: [Coda] remove limit for scheduled task processing to handle all non-terminal tasks (LogID: 202509282010120100911102412713936) Co-Authored-By: Coda <[email protected]> * feat: [Coda] implement pagination loop for scheduled task processing (LogID: 202509282010120100911102412713936) Co-Authored-By: Coda <[email protected]> * fix debug log * fix runScheduledTask * delete useless code * fix * fix backfill * fix panic * fix * add ItemSource * fix backfill * add debug log * fix * add appid * fix * fix appid * add aid * add comment * fix task_detail * fix comment * add backfill proc * fix backfill done * add env * fix panic * add debug log * fix env * add fill ctx * add cache * fix taskCache * fix log * add debug log * fix env * fix env * fix env * fix * delete useless code * delete useless code * add debug log * chore: [Coda] translate tracehub backfill comments (LogID: 20251011125600192168031096958E98A) Co-Authored-By: Coda <[email protected]> * chore: [Coda] translate tracehub callback comments (LogID: 20251011125600192168031096958E98A) Co-Authored-By: Coda <[email protected]> * chore: [Coda] translate tracehub scheduled logs to english (LogID: 20251011130741192168031096951F15E) Co-Authored-By: Coda <[email protected]> * chore: [Coda] translate tracehub scheduled task to english (LogID: 20251011130741192168031096951F15E) Co-Authored-By: Coda <[email protected]> * chore: [Coda] translate subscriber comments to english (LogID: 20251011130741192168031096951F15E) Co-Authored-By: Coda <[email protected]> * chore: [Coda] translate trace hub comments to english (LogID: 20251011130741192168031096951F15E) Co-Authored-By: Coda <[email protected]> * chore: [Coda] translate auto evaluate processor to english (LogID: 20251011130741192168031096951F15E) Co-Authored-By: Coda <[email protected]> * add conts * delete useless code * add fill ctx * fix * delete useless code * fix file * fix comment * fix * fix * fix Change-Id: Ia38d5316b1c1f343b76a396afb474f4f5591cd64 * fix Change-Id: I123702b62cd8c3b4b517a1f591493693822f2706 * add wire gen * add debug log * add debuglog * add debug log * add CTX * add permission * add permission * fix permission * add permission * fix permission * fix * NA (LogID: 202510130112151921680310966149939) Co-Authored-By: Coda <[email protected]> * feat: [Coda] add tests for observability task application (LogID: 202510130112151921680310966149939) Co-Authored-By: Coda <[email protected]> * add debug log * feat: [Coda] guard transform task status with lock (LogID: 20251013073531192168031096625ABCA) Co-Authored-By: Coda <[email protected]> * feat: [Coda] lock backfill job to avoid deadlock (LogID: 20251013073531192168031096625ABCA) Co-Authored-By: Coda <[email protected]> * fix debug log * feat: [Coda] inject locker into trace hub service (LogID: 20251013073531192168031096625ABCA) Co-Authored-By: Coda <[email protected]> * fix dao * fix set * fix * fix debug log * fix * add wire gen * add debug log * fix: [Coda] add lock renewal for backfill (LogID: 20251013112813010091110127433B953) Co-Authored-By: Coda <[email protected]> * add debug log * debug log * refactor: [Coda] serialize backfill span processing (LogID: 20251013112813010091110127433B953) Co-Authored-By: Coda <[email protected]> * fix * fix count * fix bug * fix count * fix * fix error * fix permission * fix * fix * fix DO、DTO、PO * fix DTO * fix noop * fix * fix * fix * fix * fix * fix * fix * fix * fix time * fix panic * fix ttl * add ext * fix logs * fix log * fix * fix panic * fix * fix * fix * fix * fix * FIX * FIX TIME * fix loginfo * fix * fix * delete useless log * add mq * fix log * add lience * fix: [Coda] 更新任务应用单测中的参数错误 (LogID: 202510150018011921680310963441CE5) Co-Authored-By: Coda <[email protected]> * add license * fix: [Coda] 调整任务创建单测参数匹配 (LogID: 202510150018011921680310963441CE5) Co-Authored-By: Coda <[email protected]> * go mod * fix * fix lecense * go mod * fix * test: [Coda] add task service unit tests (LogID: 202510150018011921680310963441CE5) Co-Authored-By: Coda <[email protected]> * fix ut * add * fix ut * fix * delete useless log * fix ut * delete useless code * go fmt * delete useless code * test: [Coda] add auto evaluate processor tests (LogID: 202510150018011921680310963441CE5) Co-Authored-By: Coda <[email protected]> * add ut * add license * style: [Coda] gofumpt format task convertor (LogID: 202510150018011921680310963441CE5) Co-Authored-By: Coda <[email protected]> * fix * go fmt * go fmt * add cg config * test: [Coda] add tracehub unit tests (LogID: 20251015075943192168031096911BFD6) Co-Authored-By: Coda <[email protected]> * fix ctx * fix * test(tracehub): [Coda] Add unit tests for span_trigger, callback, and backfill (LogID: 20251015075943192168031096911BFD6) Co-Authored-By: Coda <[email protected]> * add license * fix * fix * fix panic * fix callback proc * fix auth * 增加泳道 * add debug log * fix * add debug log * fix * add callback proc * fix * add debug log * add * add tcc * delete useless code * add ttl * fix ut * fix: [Coda] align create task test context (LogID: 202510161119220100911101276073113) Co-Authored-By: Coda <[email protected]> * test: [Coda] align eval set batch create mocks (LogID: 202510161119220100911101276073113) Co-Authored-By: Coda <[email protected]> * test: [Coda] fix evaluator get mock expectations (LogID: 202510161119220100911101276073113) Co-Authored-By: Coda <[email protected]> * fix ut * test: [Coda] add tracehub backfill coverage (LogID: 202510161119220100911101276073113) Co-Authored-By: Coda <[email protected]> * fix * test: [Coda] add task convertor unit tests (LogID: 20251016131229010091110127197EBE3) Co-Authored-By: Coda <[email protected]> * test: [Coda] enhance tracehub backfill tests (LogID: 20251016131058010091110127178868C) Co-Authored-By: Coda <[email protected]> * fix * fix * test: [Coda] extend trace service unit coverage (LogID: 20251016131330010091110127442162F) Co-Authored-By: Coda <[email protected]> * add license * add ctx * add ut * add * add conf * add conf * add conf * fix * add conf * add wire * fix filter * fix getspan * fix * test: [Coda] improve processor unit coverage (LogID: 20251016204650010091110127645E12A) Co-Authored-By: Coda <[email protected]> * fix evalution * fix * fix ut * delete useless code * fix Change-Id: Ib29035860bf6b53ef7cf885f3a4e5494262a5811 * fix Change-Id: I3046726f3939032a410b0dfcd6a27b0895ac5fe1 * fix Change-Id: Ibd60bca9ea6d651aa3a1c52cdedc2aa8d03a46a5 * delete useless code * fix panic * add ut * fix panic --------- Co-authored-by: tpfz <[email protected]> Co-authored-by: Coda <[email protected]> Co-authored-by: “dsf86” <[email protected]> Co-authored-by: tpfz <[email protected]>
1 parent d144b9d commit 570bdb6

File tree

193 files changed

+52806
-2255
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

193 files changed

+52806
-2255
lines changed

backend/api/api.go

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backend/api/handler/coze/loop/apis/handler.go

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,13 @@ import (
1313
"github.com/cloudwego/kitex/client/callopt"
1414
"github.com/cloudwego/kitex/pkg/endpoint"
1515
"github.com/cloudwego/kitex/pkg/kerrors"
16+
"github.com/coze-dev/coze-loop/backend/kitex_gen/coze/loop/observability/task"
17+
"github.com/coze-dev/coze-loop/backend/loop_gen/coze/loop/observability/lotask"
1618

1719
"github.com/coze-dev/coze-loop/backend/infra/i18n"
1820
cachemw "github.com/coze-dev/coze-loop/backend/infra/middleware/ctxcache"
1921
logmw "github.com/coze-dev/coze-loop/backend/infra/middleware/logs"
22+
"github.com/coze-dev/coze-loop/backend/infra/middleware/session"
2023
"github.com/coze-dev/coze-loop/backend/infra/middleware/validator"
2124
"github.com/coze-dev/coze-loop/backend/kitex_gen/coze/loop/data/dataset"
2225
"github.com/coze-dev/coze-loop/backend/kitex_gen/coze/loop/data/tag"
@@ -194,20 +197,24 @@ type ObservabilityHandler struct {
194197
obapp.ITraceApplication
195198
obapp.ITraceIngestionApplication
196199
obapp.IObservabilityOpenAPIApplication
200+
obapp.ITaskApplication
197201
}
198202

199203
func NewObservabilityHandler(
200204
traceApp obapp.ITraceApplication,
201205
ingestApp obapp.ITraceIngestionApplication,
202206
openAPIApp obapp.IObservabilityOpenAPIApplication,
207+
taskApp obapp.ITaskApplication,
203208
) *ObservabilityHandler {
204209
h := &ObservabilityHandler{
205210
ITraceApplication: traceApp,
206211
ITraceIngestionApplication: ingestApp,
207212
IObservabilityOpenAPIApplication: openAPIApp,
213+
ITaskApplication: taskApp,
208214
}
209215
bindLocalCallClient(trace.TraceService(h), &observabilityClient, lotrace.NewLocalTraceService)
210216
bindLocalCallClient(traceopenapi.OpenAPIService(h), &observabilityOpenAPIClient, looptraceopenapi.NewLocalOpenAPIService)
217+
bindLocalCallClient(task.TaskService(h), &observabilityTaskClient, lotask.NewLocalTaskService)
211218
return h
212219
}
213220

@@ -224,6 +231,7 @@ func defaultKiteXMiddlewares() []endpoint.Middleware {
224231
return []endpoint.Middleware{
225232
logmw.LogTrafficMW,
226233
validator.KiteXValidatorMW,
234+
session.NewRequestSessionMW(),
227235
cachemw.CtxCacheMW,
228236
}
229237
}

backend/api/handler/coze/loop/apis/observability_task_service.go

Lines changed: 45 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backend/api/handler/coze/loop/apis/observability_trace_service.go

Lines changed: 18 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backend/api/handler/coze/loop/apis/wire.go

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ import (
1010
"context"
1111

1212
"github.com/cloudwego/kitex/pkg/endpoint"
13+
"github.com/coze-dev/coze-loop/backend/kitex_gen/coze/loop/evaluation/experimentservice"
14+
task_processor "github.com/coze-dev/coze-loop/backend/modules/observability/domain/task/service/taskexe/processor"
1315
"github.com/google/wire"
1416

1517
"github.com/coze-dev/coze-loop/backend/infra/ck"
@@ -92,6 +94,7 @@ var (
9294
obapp.InitTraceApplication,
9395
obapp.InitTraceIngestionApplication,
9496
obapp.InitOpenAPIApplication,
97+
obapp.InitTaskApplication,
9598
)
9699
)
97100

@@ -207,6 +210,10 @@ func InitObservabilityHandler(
207210
tagClient tagservice.Client,
208211
limiterFactory limiter.IRateLimiterFactory,
209212
datasetClient datasetservice.Client,
213+
redis redis.Cmdable,
214+
experimentClient experimentservice.Client,
215+
taskProcessor task_processor.TaskProcessor,
216+
aid int32,
210217
) (*ObservabilityHandler, error) {
211218
wire.Build(
212219
observabilitySet,

backend/api/handler/coze/loop/apis/wire_gen.go

Lines changed: 11 additions & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backend/api/router/coze/loop/apis/coze.loop.apis.go

Lines changed: 18 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backend/api/router/coze/loop/apis/middleware.go

Lines changed: 60 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

backend/cmd/consumer.go

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ func MustInitConsumerWorkers(
1919
experimentApplication exptapp.IExperimentApplication,
2020
datasetApplication dataapp.IJobRunMsgHandler,
2121
obApplication obapp.IObservabilityOpenAPIApplication,
22+
taskApplication obapp.ITaskApplication,
2223
) []mq.IConsumerWorker {
2324
var res []mq.IConsumerWorker
2425

@@ -38,7 +39,7 @@ func MustInitConsumerWorkers(
3839
if err != nil {
3940
panic(err)
4041
}
41-
workers, err = obconsumer.NewConsumerWorkers(loader, obApplication)
42+
workers, err = obconsumer.NewConsumerWorkers(loader, obApplication, taskApplication)
4243
if err != nil {
4344
panic(err)
4445
}

backend/cmd/main.go

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -60,10 +60,8 @@ func main() {
6060
if err := initTracer(handler); err != nil {
6161
panic(err)
6262
}
63-
64-
consumerWorkers := MustInitConsumerWorkers(c.cfgFactory, handler, handler, handler)
65-
consumerRegistry := registry.NewConsumerRegistry(c.mqFactory).Register(consumerWorkers)
66-
if err := consumerRegistry.StartAll(ctx); err != nil {
63+
consumerWorkers := MustInitConsumerWorkers(c.cfgFactory, handler, handler, handler, handler)
64+
if err := registry.NewConsumerRegistry(c.mqFactory).Register(consumerWorkers).StartAll(ctx); err != nil {
6765
panic(err)
6866
}
6967

0 commit comments

Comments
 (0)