-
Notifications
You must be signed in to change notification settings - Fork 436
fix: faster update of file tasks for loki.source.file #4611
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
We also make sure to collect all targets and perform the export every time component is updated with new discovery data
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR optimizes the loki.source.file
component by splitting the main loop into two separate goroutines: one for consuming and forwarding log entries, and another for managing task updates. This architectural change eliminates blocking issues where task updates were delayed when the system was busy sending entries, removing the need for a drain job during updates.
Key changes:
- Separated log entry consumption and task scheduling into independent goroutines using a WaitGroup
- Removed the complex drain goroutine mechanism that was previously needed during task updates
- Enhanced
local.file_match
component with immediate update triggers and change detection to reduce unnecessary exports
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
File | Description |
---|---|
internal/component/loki/source/file/file.go | Refactored main loop into two goroutines (consumer and scheduler), removed drain mechanism, improved error handling for task updates |
internal/component/local/file_match/file.go | Added trigger channel for immediate updates, implemented target comparison to avoid redundant exports, changed RWMutex to Mutex |
Co-authored-by: Copilot <[email protected]>
PR Description
loki.source.file
have a main loop that is responsible to read and forward log entries from tailer tasks. When we get an update to add new tasks we store new tasks and add a signal to this main loop, this is done to keep calls to Update cheap.The problem is that we will not update running tasks if we are blocked on sending an entry. This pr splits up the consumer work and task scheduling into two different go routines. With this change we no longer need to start a drain job while we update tasks.
Which issue(s) this PR fixes
Notes to the Reviewer
PR Checklist