Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid reprocessing old Jaeger traces #3

Open
cloudronin opened this issue Sep 19, 2024 · 1 comment
Open

Avoid reprocessing old Jaeger traces #3

cloudronin opened this issue Sep 19, 2024 · 1 comment
Assignees
Labels
bug Something isn't working
Milestone

Comments

@cloudronin
Copy link
Member

RIght now Pythia re-processes all Jaeger traces by default once its up and has no memory of previously processed traces.

This leads to duplication of work and metrics becoming erroneus.

Need a watermarking solution to keep track of last processed timestamp so if the Pythia docker crashes and is brought back up it doesnt reprocess old traces

@cloudronin cloudronin added the bug Something isn't working label Sep 19, 2024
@cloudronin cloudronin added this to the v0.1 milestone Sep 19, 2024
@pratacosmin
Copy link
Collaborator

pratacosmin commented Sep 26, 2024

To fix this bug we filter the traces by start and end time in order to reduce the number and also keep track of old traces that were process already in order to avoid reprocessing. Maybe a better solution is to use a queue this will also allow for multiple instances of pythia to process traces in parallel

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants