You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
When the termination signal is sent, instead of forcefully shutting everything down, we should gracefully shut down the extractor, processor and sink. Furthermore, the way we shut down the stream today can result in panic. The panic occurs because there is a possibility that the agent.stream instance will try to send on a closed channel.
Describe the solution you'd like
There should be a way to signal the extractor to wind down without cancelling the ctx and disrupting the in-flight operations. The extractor should decide how many assets it wants to emit post the shutdown signal. For example, in some cases, the API call could complete in 100ms and the assets built using the response can still be emitted. The ctx cancellation can be used to provide a deadline for the graceful shutdown. Similarly, the sink should try to complete the execution for already-extracted assets within a reasonable deadline (again controlled using the ctx).
Additional context
To reproduce the error for send on closed channel, run a recipe to sync assets to Compass and terminate the run mid-way. Example output:
$ meteor run ./_recipes/merlin-stg.yaml
INFO[0000] running recipe recipe=merlin-stg
INFO[0013] Successfully extracted record recipe=merlin-stg record="urn:merlin:merlin-stg:model:10.38"
INFO[0013] sinking record to compass record="urn:merlin:merlin-stg:model:10.38"
...
INFO[0014] force closing run recipe=merlin-stg
ERRO[0014] panic recovered err="send on closed channel
...
INFO[0014] goroutine 48 [running]:
runtime/debug.Stack()
/opt/homebrew/Cellar/go/1.19.3/libexec/src/runtime/debug/stack.go:24 +0x64
github.com/odpf/meteor/plugins/extractors/merlin.(*Extractor).extractProject.func1()
/Users/skaranth/dev/work/odpf/meteor/plugins/extractors/merlin/merlin.go:180 +0xd8
panic({0x1039dc400, 0x103dc6ad8})
/opt/homebrew/Cellar/go/1.19.3/libexec/src/runtime/panic.go:884 +0x204
github.com/odpf/meteor/agent.(*stream).push(0x140002cebd0, {0x14000b76cc0?})
/Users/skaranth/dev/work/odpf/meteor/agent/stream.go:102 +0x110
github.com/odpf/meteor/plugins/extractors/merlin.(*Extractor).extractProject(0x14000efbea0, {0x103de85a8, 0x14000f70080}, {0x1, {0x14000bfc308, 0x6}, {0x140010c2000, 0x24}, {0x140010840c0, 0x2, ...}, ...}, ...)
/Users/skaranth/dev/work/odpf/meteor/plugins/extractors/merlin/merlin.go:234 +0x4d0
github.com/odpf/meteor/plugins/extractors/merlin.(*Extractor).startWorkers.func1()
/Users/skaranth/dev/work/odpf/meteor/plugins/extractors/merlin/merlin.go:159 +0x134
created by github.com/odpf/meteor/plugins/extractors/merlin.(*Extractor).startWorkers
/Users/skaranth/dev/work/odpf/meteor/plugins/extractors/merlin/merlin.go:137 +0x6c
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
When the termination signal is sent, instead of forcefully shutting everything down, we should gracefully shut down the extractor, processor and sink. Furthermore, the way we shut down the stream today can result in panic. The panic occurs because there is a possibility that the
agent.stream
instance will try to send on a closed channel.Describe the solution you'd like
There should be a way to signal the extractor to wind down without cancelling the
ctx
and disrupting the in-flight operations. The extractor should decide how many assets it wants to emit post the shutdown signal. For example, in some cases, the API call could complete in 100ms and the assets built using the response can still be emitted. Thectx
cancellation can be used to provide a deadline for the graceful shutdown. Similarly, the sink should try to complete the execution for already-extracted assets within a reasonable deadline (again controlled using thectx
).Additional context
To reproduce the error for
send on closed channel
, run a recipe to sync assets to Compass and terminate the run mid-way. Example output:The text was updated successfully, but these errors were encountered: