Add fix options for integrity issues #12495

priyanshu16095 · 2025-02-12T22:13:47Z

Fixes #11419

This PR introduces a functionality to fix integrity issues either individually or all at once by selecting a type.

Mandatory checks

I own the copyright of the code submitted and I licence it under the MIT license
Change in CHANGELOG.md described in a way that is understandable for the average user (if change is visible to the user)
Tests created for changes (if applicable)
Manually tested changed features in running JabRef (always required)
Screenshots added in PR description (for UI changes)
Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

priyanshu16095 · 2025-02-12T22:14:47Z

Recording.2025-02-13.mp4

There is a problem with selecting messages using the Shift key. One has to click on a message first, as the focus moves to the entry editor. However, when clicking again with the mouse, all the selected entries are lost.

priyanshu16095 · 2025-02-12T22:15:38Z

Recording.2025-02-13.mp4

github-actions

JUnit tests are failing. In the area "Some checks were not successful", locate "Tests / Unit tests (pull_request)" and click on "Details". This brings you to the test output.

You can then run these tests in IntelliJ to reproduce the failing tests locally. We offer a quick test running howto in the section Final build system checks in our setup guide.

priyanshu16095 · 2025-02-16T19:44:57Z

Please review

priyanshu16095 · 2025-02-19T19:42:07Z

I have refactored the code a lot and used better practices.
I have used an enum as, without it I would have to use switch-case in every method.
Please guide me on whether I am moving in the right direction.

github-actions

JUnit tests are failing. In the area "Some checks were not successful", locate "Tests / Unit tests (pull_request)" and click on "Details". This brings you to the test output.
You can then run these tests in IntelliJ to reproduce the failing tests locally. We offer a quick test running howto in the section Final build system checks in our setup guide.

github-actions

JUnit tests are failing. In the area "Some checks were not successful", locate "Tests / Unit tests (pull_request)" and click on "Details". This brings you to the test output.
You can then run these tests in IntelliJ to reproduce the failing tests locally. We offer a quick test running howto in the section Final build system checks in our setup guide.

github-actions

JUnit tests are failing. In the area "Some checks were not successful", locate "Tests / Unit tests (pull_request)" and click on "Details". This brings you to the test output.
You can then run these tests in IntelliJ to reproduce the failing tests locally. We offer a quick test running howto in the section Final build system checks in our setup guide.

priyanshu16095 · 2025-02-20T18:00:46Z

Recording.2025-02-20.mp4

The feature is completed.
The fixes for the issue are left to be added.

src/main/java/org/jabref/logic/integrity/IntegrityIssue.java

src/main/java/org/jabref/gui/integrity/IntegrityCheckDialog.java

src/main/java/org/jabref/gui/integrity/IntegrityCheckDialogViewModel.java

src/main/java/org/jabref/logic/integrity/BibTeXEntryTypeChecker.java

src/main/java/org/jabref/logic/integrity/CitationKeyDuplicationChecker.java

src/main/java/org/jabref/logic/integrity/DateChecker.java

src/main/java/org/jabref/logic/integrity/EntryLinkChecker.java

src/main/java/org/jabref/logic/integrity/HTMLCharacterChecker.java

src/main/java/org/jabref/logic/integrity/ISSNChecker.java

src/main/java/org/jabref/logic/integrity/IntegrityIssue.java

src/main/java/org/jabref/logic/integrity/NoBibtexFieldChecker.java

src/main/java/org/jabref/logic/integrity/NoteChecker.java

koppor · 2025-03-19T20:57:17Z

After this PR is merged, one can reply at https://discourse.jabref.org/t/how-to-solve-quality-integrity-issues/1454/4?u=koppor

src/main/java/org/jabref/gui/integrity/IntegrityCheckDialog.java

koppor

The issue seems to be too hard.

No use of formatters at package org.jabref.logic.formatter.bibtexfields, but some hard-coded fixes.

Will takle longer to get this right.

koppor · 2025-03-19T21:25:33Z

src/main/java/org/jabref/gui/integrity/IntegrityCheckDialogViewModel.java

+        }
+    }
+
+    public void maskTitle(IntegrityMessage message) {


This code seems like AI generated.

I think, I pointed you to org.jabref.logic.cleanup.CleanupJob / org.jabref.logic.cleanup.Formatter

Maybe, we should start on working on a single integrity issue - discuss the archtiecture and then do a second one.

For pages the formatter to use is org.jabref.logic.formatter.bibtexfields.NormalizePagesFormatter

I am not sure why it feels AI-generated, it is basic string manipulation, as described in the issue.
Sure, I will use the describe classes.

Following puzzled me very much

updateField(message, StandardField.PAGES, "1-10");

Please tell me about the confusion, I am not clear about it.
I created this method because this code was being used in every method for updating fields.

Why I created this method?
Capital letters are not masked using curly brackets {} --> Basic string manipulation as mentioned in the issue, so to update the field I found this way
Map<Field, String> fields = new HashMap<>();
fields.put(field, newValue);
message.entry().setField(fields);
Then to impove the code reusablity, I created the method updateField.

From next time I will ask rather then pushing based on assumptions.

Very OK to push - we can then read code. This is much more easy.

I think, this task is really large. (note the "size: big" label).

Thus, it takes time. Will be a back and forth.

The clenaup jobs should do the cleaning, not manually. Look into how the cleanup dialog works.

(I currently cannot do deeper hints. I can try to draft the architecture next week if we cannot manage to move forward efficiently here)

koppor · 2025-03-19T23:57:09Z

It lists all the messages that are present in the table. I will improve this. I will identify issues that cannot be fixed automatically and remove them from the dropdown or AI can be used to fix these.

No need to add AI. Just use existing cleanup actions (see docs.jabref.org/finding-sorting-and-cleaning-entries/saveactions for a list)

Note that I linked the actions in the screenshot at #12495 (review) - and I hoped, you could find the code starting from that 😅

priyanshu16095 · 2025-03-20T00:05:28Z

In some of methods like formatPageNumberRange, replaceNonASCIICharacters, ensureValidEdition, as well as writing tests for them, I have used AI due to lack of understanding about the fields, I should rather ask or search on the web.

I have planned to ask for improvements about the fixes, which is why I have created fixes for only some issues to limit the usage of AI.

Everything else was planned by me, it took me so much iterations from using switch-case in every method to make use of enum, then solving decoupling issues. I used AI just for writing fix methods due to lack of knowledge.

subhramit · 2025-03-20T00:29:26Z

In some of methods like formatPageNumberRange, replaceNonASCIICharacters, ensureValidEdition, as well as writing tests for them, I have used AI due to lack of understanding about the fields, I should rather ask or search on the web.

I have planned to ask for improvements about the fixes, which is why I have created fixes for only some issues to limit the usage of AI.

Everything else was planned by me, it took me so much iterations from using switch-case in every method to make use of enum, then solving decoupling issues. I used AI just for writing fix methods due to lack of knowledge.

Hi, it's fine that you learnt what AI does when you use it without full understanding (issue with >95% PRs we see these days) - and thanks for being honest about it. Please be more careful next time, we are all here to answer questions.

Also, review AI code very well before using. Even if you didn't have idea about fields, the hardcoding of pages "1-10" in updateField(message, StandardField.PAGES, "1-10"); should have given you a hint that something is out of place (I am guessing that's what @koppor was referring to).

Edit: this was answer to your question in #12495 (comment):

Please tell me about the confusion, I am not clear about it.
I created this method because this code was being used in every method for updating fields.

trag-bot · 2025-03-20T23:03:12Z

src/main/java/org/jabref/gui/integrity/IntegrityCheckDialogViewModel.java

+        entryTypes.remove(entry);
+    }
+
+    public void fix(IntegrityIssue issue, IntegrityMessage message) {


Method lacks JavaDoc despite being a public method with non-trivial functionality (handling integrity issues). Complex operations should be documented.

trag-bot · 2025-03-21T04:42:25Z

@trag-bot didn't find any issues in the code! ✅✨

Add fix options for integrity issues

9b11edf

This comment was marked as outdated.

Sign in to view

Fix the check

02d8e75

This comment was marked as outdated.

Sign in to view

Add a missing entry for Localization.lang

754b6ad

This comment was marked as outdated.

Sign in to view

priyanshu16095 added 2 commits February 13, 2025 05:06

Fix the check

696f904

Refactor code

4ab36a8

github-actions bot reviewed Feb 14, 2025

View reviewed changes

priyanshu16095 added 2 commits February 14, 2025 18:55

Fix the check

28de796

Add a CHANGELOG.md entry

70ae8eb

Refactor the code

77df6ea

github-actions bot reviewed Feb 19, 2025

View reviewed changes

priyanshu16095 added 2 commits February 20, 2025 16:47

Make use of IntegrityIssue enum

30c81ce

Add keys in Localization.lang

5b389da

github-actions bot reviewed Feb 20, 2025

View reviewed changes

priyanshu16095 added 2 commits February 20, 2025 22:10

Refactor the code

3a0d9a1

Remove use of enum

a04e06a

github-actions bot reviewed Feb 20, 2025

View reviewed changes

priyanshu16095 added 4 commits February 20, 2025 22:17

Remove use of enum from PageChecker class

8e9341a

Add fixAll method

235c74f

Add a check for fixAll and fixByType method

9839137

Fix a bug

5754680

priyanshu16095 added 2 commits February 23, 2025 01:09

Use dialogService.notify() to notify

8f7bdc2

Remove unused imports

f57719c

priyanshu16095 added 3 commits March 17, 2025 22:55

Remove csl modules

e11499a

Remove csl locales

157dfe8

Correct the spelling of BibTeX

6f86c65