Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance the queue failed jobs functionality. #8306

Closed
touhidurabir opened this issue Oct 3, 2022 · 17 comments
Closed

Enhance the queue failed jobs functionality. #8306

touhidurabir opened this issue Oct 3, 2022 · 17 comments
Assignees
@touhidurabir
Labels
Housekeeping:1:Todo Any dependency management or refactor that would be nice to have some day.
Milestone
3.4.0-0

Comments

@touhidurabir
Copy link
Member

touhidurabir commented Oct 3, 2022 •
edited
Loading

Describe the enhancement
We have a frontend UI implemented to view queue pending/waiting to execute queue jobs in #7171 also queue job execution/handling enhancement in #7105 . But have no UI or CLI functionality to view or manage failed jobs and associated error for those failed job. It can be a good enhancement to have following functionalities for better managing of failed jobs

  • UI for failed jobs with redispatch, remove and exception trace visualisation.
  • CLI enhancement for listing failed jobs with ability to redispatch and clear out failed jobs.
  • Schedule task to clear out much older failed jobs such as older than 180 days .

Who is asking for this one?
No one right now. But this will give both the devs and site admins each access to failed jobs with better visualisation.

Possible UI Update(screenshots)
CleanShot 2023-01-11 at 14 54 38@2x
CleanShot 2023-01-11 at 14 52 01@2x
omp-main test_index php_index%2Fadmin%2FfailedJobDetails_73

Previous Idea
Clear out much older failed jobs from the failed_jobs table. Suggestion is to run a schedule task once a week and clear out any failed jobs older that 2 weeks from the table to keep it cleaner . This can also include some enhancement to existing job cli tool for better failed jobs handling.

PRs

ojs --> pkp/ojs#3763
omp --> pkp/omp#1298
ops --> pkp/ops#472
pkp-lib --> #8534
ui-library --> pkp/ui-library#248
@touhidurabir touhidurabir self-assigned this Oct 3, 2022
@touhidurabir touhidurabir added the Housekeeping:1:Todo Any dependency management or refactor that would be nice to have some day. label Oct 3, 2022
@touhidurabir touhidurabir added this to the 3.4 milestone Oct 3, 2022
@NateWr
Copy link
Contributor

NateWr commented Oct 3, 2022

I like it but I suggest a much longer window before removing failed jobs. Maybe 6 months? I think that in some cases failed jobs will be one way that people audit when things have gone wrong, and they may not recognize a problem for a while. This may be especially true with metadata deposits (eg - Crossref) where someone might not realize that their work has failed to be deposited for a long time.

@touhidurabir
Copy link
Member Author

suggest a much longer window before removing failed jobs. Maybe 6 months?

How about we set it as a config option in number of days ? and if thats turned off or disabled , the schedule task will not run. better control than hard coded values .

@NateWr
Copy link
Contributor

NateWr commented Oct 4, 2022

Yes, that sounds good.

@touhidurabir touhidurabir changed the title Clear out older failed jobs from failed_jobs table. Enhance the queue failed jobs functionality. Jan 9, 2023
touhidurabir added a commit to touhidurabir/ui-library that referenced this issue Jan 9, 2023
@touhidurabir
pkp/pkp-lib#8306 Failed jobs UI
1087193
touhidurabir added a commit to touhidurabir/pkp-lib that referenced this issue Jan 9, 2023
@touhidurabir
pkp#8306 Failed jobs UI and CLI enhancement
d6a9948
touhidurabir added a commit to touhidurabir/omp that referenced this issue Jan 9, 2023
@touhidurabir
pkp/pkp-lib#8306 Failed jobs functionality added and enhancement
7cba7ae
touhidurabir added a commit to touhidurabir/omp that referenced this issue Jan 9, 2023
@touhidurabir
pkp/pkp-lib#8306 Submodule Update ##touhidurabir/i8306_main##
c7078b1
This was referenced Jan 9, 2023
Merged
Merged
Merged
touhidurabir added a commit to touhidurabir/pkp-lib that referenced this issue Jan 11, 2023
@touhidurabir
pkp#8306 Added failed job details page
07fce88
touhidurabir added a commit to touhidurabir/ui-library that referenced this issue Jan 11, 2023
@touhidurabir
pkp/pkp-lib#8306 Added failed job details page
2d5509b
touhidurabir added a commit to touhidurabir/omp that referenced this issue Jan 11, 2023
@touhidurabir
pkp/pkp-lib#8306 Added failed job details page
13df7d1
touhidurabir added a commit to touhidurabir/omp that referenced this issue Jan 11, 2023
@touhidurabir
pkp/pkp-lib#8306 Submodule Update ##touhidurabir/i8306_main##
bd59925
touhidurabir added a commit to touhidurabir/omp that referenced this issue Jan 11, 2023
@touhidurabir
pkp/pkp-lib#8306 Failed jobs functionality added and enhancement
602e30a
touhidurabir added a commit to touhidurabir/omp that referenced this issue Jan 11, 2023
@touhidurabir
pkp/pkp-lib#8306 Submodule Update ##touhidurabir/i8306_main##
78792e9
touhidurabir added a commit to touhidurabir/omp that referenced this issue Jan 11, 2023
@touhidurabir
pkp/pkp-lib#8306 Added failed job details page
fecc2b3
touhidurabir added a commit to touhidurabir/omp that referenced this issue Jan 11, 2023
@touhidurabir
pkp/pkp-lib#8306 Submodule Update ##touhidurabir/i8306_main##
1312f46
touhidurabir added a commit to touhidurabir/pkp-lib that referenced this issue Jan 11, 2023
@touhidurabir
pkp#8306 Fixed job test
f6929f4
touhidurabir added a commit to touhidurabir/omp that referenced this issue Jan 11, 2023
@touhidurabir
pkp/pkp-lib#8306 Submodule Update ##touhidurabir/i8306_main##
b181c33
touhidurabir added a commit to touhidurabir/pkp-lib that referenced this issue Jan 12, 2023
@touhidurabir
pkp#8306 Added 404 not found handler for failed job details page
7a39598
touhidurabir added a commit to touhidurabir/omp that referenced this issue Jan 12, 2023
@touhidurabir
pkp/pkp-lib#8306 Submodule Update ##touhidurabir/i8306_main##
64f32c1
@touhidurabir
Copy link
Member Author

@asmecher Please take a look at this.

@asmecher
Copy link
Member

Looks very promising so far, @touhidurabir! I like the simplification you've proposed as well. I've left a few comments.

touhidurabir added a commit to touhidurabir/pkp-lib that referenced this issue Jan 13, 2023
@touhidurabir
pkp#8306 Admin api handling update
0b9e9a9
touhidurabir added a commit to touhidurabir/pkp-lib that referenced this issue Jan 13, 2023
@touhidurabir
pkp#8306 Failed jobs UI and CLI enhancement
d9a477e
asmecher pushed a commit that referenced this issue Feb 16, 2023
@touhidurabir @asmecher
#8306 Fixed job test
3451248
asmecher pushed a commit that referenced this issue Feb 16, 2023
@touhidurabir @asmecher
#8306 Added 404 not found handler for failed job details page
eaed57e
asmecher pushed a commit that referenced this issue Feb 16, 2023
@touhidurabir @asmecher
#8306 Admin api handling update
060346d
asmecher pushed a commit that referenced this issue Feb 16, 2023
@touhidurabir @asmecher
#8306 Integrated laravel resource functionality and removed custom re…
591167b 
…source class dependence
asmecher pushed a commit that referenced this issue Feb 16, 2023
@touhidurabir @asmecher
#8306 Moved job data fetch to API
be48dc7
asmecher pushed a commit that referenced this issue Feb 16, 2023
@touhidurabir @asmecher
#8306 Translation and pagination handler update
5015344
asmecher pushed a commit that referenced this issue Feb 16, 2023
@touhidurabir @asmecher
#8306 UI enhancement and requeue all action added
8b6efd8
asmecher pushed a commit that referenced this issue Feb 16, 2023
@touhidurabir @asmecher
#8306 Test update
c60df53
asmecher pushed a commit that referenced this issue Feb 16, 2023
@touhidurabir @asmecher
#8306 code cleanup, indentation fixing and removed confusing param fr…
0838a90 
…om APIHandler::getEndpointPattern
asmecher pushed a commit that referenced this issue Feb 16, 2023
@touhidurabir @asmecher
#8306 Test updated, added wait after running requeue
ea8bf2f
asmecher pushed a commit to pkp/ui-library that referenced this issue Feb 16, 2023
@touhidurabir
pkp/pkp-lib#8306 Failed jobs UI (#248)
b4a76e4 
* pkp/pkp-lib#8306 Failed jobs UI

* pkp/pkp-lib#8306 Added failed job details page

* pkp/pkp-lib#8306 Moved job data fetch to API

* pkp/pkp-lib#8306 UI enhancement

* pkp/pkp-lib#8306 code cleanup by extending JobsPage in FailedJobsPage component
@asmecher
Copy link
Member

All merged -- thanks, @touhidurabir and @NateWr!

touhidurabir added a commit to touhidurabir/pkp-lib that referenced this issue Feb 16, 2023
@touhidurabir
pkp#8306 Patch for zero-length delimited identifier issue
5a73ae1
touhidurabir added a commit to touhidurabir/omp that referenced this issue Feb 16, 2023
@touhidurabir
pkp/pkp-lib#8306 Submodule Update ##touhidurabir/i8306_patch_main##
ffa6af3
This was referenced Feb 16, 2023
Merged
Merged
touhidurabir added a commit to touhidurabir/ops that referenced this issue Feb 16, 2023
@touhidurabir
pkp/pkp-lib#8306 Submodule Update ##touhidurabir/i8306_patch_main##
876b190
touhidurabir added a commit to touhidurabir/ojs that referenced this issue Feb 16, 2023
@touhidurabir
pkp/pkp-lib#8306 Submodule Update ##touhidurabir/i8306_patch_main##
c21ef9f
This was referenced Feb 16, 2023
Merged
Merged
touhidurabir added a commit that referenced this issue Feb 16, 2023
@touhidurabir
Merge pull request #8644 from touhidurabir/i8306_patch_main
083438b 
#8306 Patch for zero-length delimited identifier issue
touhidurabir added a commit to pkp/omp that referenced this issue Feb 16, 2023
@touhidurabir
Merge pull request #1341 from touhidurabir/i8306_patch_main
272af07 
pkp/pkp-lib#8306 Submodule Update ##touhidurabir/i8306_patch_main##
touhidurabir added a commit to pkp/ops that referenced this issue Feb 16, 2023
@touhidurabir
Merge pull request #476 from touhidurabir/i8306_patch_main
4441a0d 
pkp/pkp-lib#8306 Submodule Update ##touhidurabir/i8306_patch_main##
touhidurabir added a commit to pkp/ojs that referenced this issue Feb 16, 2023
@touhidurabir
Merge pull request #3771 from touhidurabir/i8306_patch_main
44b98eb 
pkp/pkp-lib#8306 Submodule Update ##touhidurabir/i8306_patch_main##
@touhidurabir
Copy link
Member Author

touhidurabir commented Feb 16, 2023 •
edited
Loading

Builds are failing for PostgreSQL because of zero-length delimited identifier .

Patch PRs
pkp-lib --> #8644
omp --> pkp/omp#1341 [Submodule Update Only]
ops --> pkp/ops#476 [Submodule Update Only]
ojs --> pkp/ojs#3771 [Submodule Update Only]

Changes are pushed and merged also tested in local Postgres 11/12/13 . Should fix it now but keeping this open until then .

@touhidurabir touhidurabir reopened this Feb 16, 2023
@touhidurabir
Copy link
Member Author

touhidurabir commented Feb 16, 2023 •
edited
Loading

All tests are passing now .

@touhidurabir touhidurabir mentioned this issue Aug 12, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@touhidurabir touhidurabir

Labels
Housekeeping:1:Todo Any dependency management or refactor that would be nice to have some day.
Projects
None yet
Milestone
3.4.0-0
Development

No branches or pull requests
3 participants
@asmecher @NateWr @touhidurabir