Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issue #11189 part 00 refactor citation relation tab logic #11845

Open
wants to merge 24 commits into
base: main
Choose a base branch
from

Conversation

alexandre-cremieux
Copy link
Contributor

@alexandre-cremieux alexandre-cremieux commented Sep 28, 2024

High level description

This contributions aims to provide a local temporary local storage for citations relations using MVStore. Please see #11189 for more information.

Implementation details

No new dependency added.

Caching strategy

The implemented caching logic is following a 2 levels side-caching strategy with:

  • an in memory caching level powered by an LRU MAP
  • a local storage powered by a an h2.mvstore.MVStore

The interaction between the two levels of cache is backed by a responsibility chain pattern. This helps to:

  • avoid searching on file if the citations/references are already available from RAM
  • avoid purely technical code to leak into the service layer that should only orchestrate fetching, storing, reading
  • ease future optimization for each of the levels of caching
  • should ease change of technologies for this use case in the future

Search high level logic

  1. Check if the fetch is possible: if the cache does not contain anything or if the local storage TTL ran out
  2. If the fetch is possible: fetch and insert into cache (in memory and locally
  3. Read from cache: if the in-memory cache is empty then load from local storage and update update in-memory cache

Serialization

MVStoreDAO uses the canonical representation to serialize a BibEntry and the BibtexParser to deserialize it.

Configuration

  • the search service is a singleton injected in each new Citations Relations tab using the JabRef IoC provider
  • the store path is configured to target the citations folder under JabRef's app directory

Setting translation

The the local storage TTL setting message has been traduced in:

  • DE
  • FR
  • EN
  • PL
  • IT
  • BR

Refactoring

This contributions simplifies the citations/references fetching and caching logic by introducing two layers:

  • service
  • repository

This should help to make this feature more extendable without modifying orchestration logic following open/close principle.

Screen shots

Screenshot 2025-01-26 at 22 02 16 Screenshot 2025-01-26 at 22 03 02

Mandatory checks

  • Change in CHANGELOG.md described in a way that is understandable for the average user (if applicable)
  • Tests created for changes (if applicable)
  • Manually tested changed features in running JabRef (always required)
  • Screenshots added in PR description (for UI changes)
  • Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
  • Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

@alexandre-cremieux alexandre-cremieux marked this pull request as draft September 28, 2024 20:38
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your code currently does not meet JabRef's code guidelines.
We use Checkstyle to identify issues.
The tool reviewdog already placed comments on GitHub to indicate the places. See the tab "Files" in you PR.
Please carefully follow the setup guide for the codestyle.
Afterwards, please run checkstyle locally and fix the issues.

You can check review dog's comments at the tab "Files changed" of your pull request.

@alexandre-cremieux alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch from 9fe8522 to cbe9e96 Compare September 28, 2024 21:15
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your code currently does not meet JabRef's code guidelines.
We use Checkstyle to identify issues.
The tool reviewdog already placed comments on GitHub to indicate the places. See the tab "Files" in you PR.
Please carefully follow the setup guide for the codestyle.
Afterwards, please run checkstyle locally and fix the issues.

You can check review dog's comments at the tab "Files changed" of your pull request.

@alexandre-cremieux alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch from cbe9e96 to 8231340 Compare September 28, 2024 21:51
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your code currently does not meet JabRef's code guidelines.
We use OpenRewrite to ensure "modern" Java coding practices.
The issues found can be automatically fixed.
Please execute the gradle task rewriteRun, check the results, commit, and push.

You can check the detailed error output by navigating to your pull request, selecting the tab "Checks", section "Tests" (on the left), subsection "OpenRewrite".

@alexandre-cremieux alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch from 8231340 to 33967c2 Compare September 28, 2024 22:20
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your code currently does not meet JabRef's code guidelines.
We use OpenRewrite to ensure "modern" Java coding practices.
The issues found can be automatically fixed.
Please execute the gradle task rewriteRun, check the results, commit, and push.

You can check the detailed error output by navigating to your pull request, selecting the tab "Checks", section "Tests" (on the left), subsection "OpenRewrite".

@alexandre-cremieux alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch from 33967c2 to 592d4d7 Compare September 28, 2024 22:47
@koppor koppor changed the title Fix issue 11189 part 00 refactor citation relation tab logic Fix issue #11189 part 00 refactor citation relation tab logic Sep 29, 2024
@alexandre-cremieux alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch 2 times, most recently from d94f4d3 to 3155242 Compare September 29, 2024 13:24
Copy link
Contributor Author

@alexandre-cremieux alexandre-cremieux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add code explanations to the PR

@Siedlerchr
Copy link
Member

Please no force push if not needed. All commits will be squashed when merged

* Move repository, cache, and fetcher to logic package
* Move citations model to model/citations/semanticscholar package
* Introduce service layer
* Rename LRU cache implementation
* Add tests helpers for repository
* Move logic from repository to service
* Refactor repositories
* Update tab configuration
@alexandre-cremieux alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch from 3155242 to 18db75e Compare September 29, 2024 15:01
@alexandre-cremieux
Copy link
Contributor Author

Please no force push if not needed. All commits will be squashed when merged

Sorry, just re-based main branch locally.

…lation-tab-logic

# Conflicts:
#	src/main/java/org/jabref/gui/entryeditor/citationrelationtab/CitationRelationsTab.java
Copy link
Member

@koppor koppor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general looks good. Some minor comments.

Sorry for delay. Please go ahead with everything.

@koppor koppor marked this pull request as ready for review October 10, 2024 20:29
@koppor
Copy link
Member

koppor commented Oct 10, 2024

Small other comments - IntelliJ proposed to extract a method

private static SearchCitationsRelationsService getSearchCitationsRelationsService(BibEntry cited, List<BibEntry> citationsToReturn, BibEntryRelationsRepository citationsToReturn1) {

in the tests - maybe you can also include that.

@koppor
Copy link
Member

koppor commented Oct 10, 2024

@alexandre-cremieux Please pull before you continue working on it - I merged main for you (and resolved conflicts).

@alexandre-cremieux
Copy link
Contributor Author

@alexandre-cremieux Please pull before you continue working on it - I merged main for you (and resolved conflicts).

Thanks for the review and the merge. I will resume the work on this branch and apply the changes.

@koppor
Copy link
Member

koppor commented Nov 8, 2024

@alexandre-cremieux Sorry for the merge conflicts - can you handle them? I was always happy with IntelliJ's "resolve merge conflicts" dialog. Hope, it works in this case, too.

@alexandre-cremieux
Copy link
Contributor Author

alexandre-cremieux commented Nov 8, 2024

@alexandre-cremieux Sorry for the merge conflicts - can you handle them? I was always happy with IntelliJ's "resolve merge conflicts" dialog. Hope, it works in this case, too.

Hello @koppor . Seems that we have new conflicts to resolve to be able to merge main. But I will do that when the feature will be fully developed. Was quite busy last month, I resumed the work this week. PR comments were addressed.

… exhausted

* Remove the isForceUpdate boolean
* User is still able to trigger the fetch if an error occurs
@alexandre-cremieux
Copy link
Contributor Author

Code update:

  • The update will be done automatically after the guard delay of the repo is exhausted
  • The user is still able to re-trigger the fetch if on error occurs (UX improvement will be done later: Improve "Refresh button" of CSL Preview #12247)

* Instantiate service in JabRefGui
* Inject service in EntryEditor
@alexandre-cremieux
Copy link
Contributor Author

alexandre-cremieux commented Jan 11, 2025

Hello @koppor.

Now, only preference for the MV store TTL time is missing. I thought about adding it under Web search preferences like this:

Citation relations search cache time-to-live (in days): [value-here]

Default value would be 7.

It would be added under a new category: Citation relations web search

@alexandre-cremieux
Copy link
Contributor Author

alexandre-cremieux commented Jan 20, 2025

I assume, you are working on "Task 1" of the issue. Thus, I need to file "Task 2" as separate issue?
Task 2 is included here. Just a question: do you want Jabref to automatically fetch citations relations after 7 days since last search even if citations are referenced in the store ? (I guess you are asking that because of the cited by list).

Think, we need this configurable. With a higher default value. Maybe 30 days? - And also disable the feature by default.

Hello @koppor

Could you please help me a bit here ?

The single missing part is the configuration that you asked for in December. Could you please explain me how to implement this: new tab in Preferences or adding it into existing one (like Web search), naming of the parameter for the user, what about the localization, etc ?

Otherwise, could you please let me know if I misunderstood something ?

Thank in advance

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your code currently does not meet JabRef's code guidelines.
We use Checkstyle to identify issues.
Please carefully follow the setup guide for the codestyle.
Afterwards, please run checkstyle locally and fix the issues.

In case of issues with the import order, double check that you activated Auto Import.
You can trigger fixing imports by pressing Ctrl+Alt+O to trigger Optimize Imports.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your code currently does not meet JabRef's code guidelines.
We use Checkstyle to identify issues.
Please carefully follow the setup guide for the codestyle.
Afterwards, please run checkstyle locally and fix the issues.

In case of issues with the import order, double check that you activated Auto Import.
You can trigger fixing imports by pressing Ctrl+Alt+O to trigger Optimize Imports.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your code currently does not meet JabRef's code guidelines.
We use Checkstyle to identify issues.
Please carefully follow the setup guide for the codestyle.
Afterwards, please run checkstyle locally and fix the issues.

In case of issues with the import order, double check that you activated Auto Import.
You can trigger fixing imports by pressing Ctrl+Alt+O to trigger Optimize Imports.

@alexandre-cremieux alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch from 32aa56a to 4956095 Compare January 20, 2025 21:20
@alexandre-cremieux alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch from 4956095 to ea36f1a Compare January 20, 2025 21:24
Copy link
Contributor Author

@alexandre-cremieux alexandre-cremieux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Answered comments.

progress,
fetchedList,
observableList
))
.onFailure(exception -> {
LOGGER.error("Error while fetching citing Articles", exception);
hideNodes(abortButton, progress, importButton);
listView.setPlaceholder(new Label(Localization.lang("Error while fetching citing entries: %0",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@InAnYan This was here before I touched this code. I am leaving this for now in this PR as this out of the scope.

LOGGER.error("Error while fetching citations for entry {}", cited.getTitle(), e);
}
}
return this.relationsRepository.readCitations(cited);
Copy link
Contributor Author

@alexandre-cremieux alexandre-cremieux Jan 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It depends. Normally:

  • If the store was empty and nothing was fetch in any case (empty fetch, or error) then yes => empty list
  • If the store was not empty and nothing was fetch after a successful fetch => the store will be erased and the returned collection will be empty
  • If the store was not empty and an error occurs while fetching => will return the content of the store

MVStoreBibEntryRelationDAO(Path path, String mapName) {
this.mapName = mapName;
this.insertionTimeStampMapName = mapName + "-insertion-timestamp";
this.storeConfiguration = new MVStore.Builder().autoCommitDisabled().fileName(path.toAbsolutePath().toString());
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank, me also usually. Hopefully, it is a proof that I am not yet a machine 🤖 😄 🌹


MVStoreBibEntryRelationDAO(Path path, String mapName) {
this.mapName = mapName;
this.insertionTimeStampMapName = mapName + "-insertion-timestamp";
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

.orElse(true);
}

private static class BibEntrySerializer extends BasicDataType<BibEntry> {
Copy link
Contributor Author

@alexandre-cremieux alexandre-cremieux Jan 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @InAnYan,

Please do not take this comment in account. You were right, ad-hoc serialization does not seem to be necessary. It should possible to apply existing logic provided by BibDatabaseWriter' and 'BibTexParser classes.

This is the complicated part:

The BibEntry is used here in fact not to represent BibEntry but something else, a citation/reference fetched from SemanticsScholar's API. The output of the API is not a BibEntry, but a Citation or a Reference and the fetcher (JabRef) convert it to BibEntry. A BibEntry is larger in memory than a citation/reference. So to avoid using more space than needed the serializer is adapted to this use case.

Also, until the Citation/Reference is inserted into the bib, the user does not have access to as much information than for a BibEntry accessible from a bib, or even cannot edit it.

I had this discussion with @koppor. I proposed to change the type of the Citation/Reference to something else and only transform it into a BibEntry when necessary (search or insertion into the bib). I thought it would simplify the code but it seems that this is not really needed.

Why did I propose those changes: (all the discussion is in the PR)

* Managing citations/references in this context is another domain than managing a BibEntry
* Interpreting DDD here (sorry, sometimes I am dogmatic), as it is not the same domain then it seems normal to not use the same model (as we are not representing the same thing)
* Avoiding mixing the concerns between citation/relation could (maybe should) reduce the ramp-up for the developer on this use case in the future
* Managing duplicates in the citations/relations stores would be easier from my point of view because it would only rely on the DOI for equality tests (I am saying this but maybe it is already the case with BibEntry ?!?) - with another model you can decide to switch to a more suitable ID (as DDD encourage it).

I let you see with @koppor. I am still willing to do those changes: create a specific model for citations/references and use this one for fetching, storage, and view in citations tabs. But convert it to BibEntry only when needed (search/insertion from/into the bib).

However, if you have a plan that implies to store it as a full BibEntry then please let me know and please indicate me where to find the proper serializer (so I do not re-invent the wheel).

private static String toString(BibEntry entry) {
return String.join(
FIELD_SEPARATOR,
entry.getTitle().orElse("null"),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Than "null" would be serialized (and deserialized).

private static BibEntry fromString(String serializedString) {
var fields = serializedString.split(FIELD_SEPARATOR);
BibEntry entry = new BibEntry();
extractFieldValue(fields[0]).ifPresent(title -> entry.setField(StandardField.TITLE, title));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because there are the only ones available for a citations or reference.

progress,
fetchedList,
observableList
))
.onFailure(exception -> {
LOGGER.error("Error while fetching citing Articles", exception);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@InAnYan This was here before I touched this code. I am leaving this for now in this PR as this out of the scope.

@alexandre-cremieux
Copy link
Contributor Author

alexandre-cremieux commented Jan 25, 2025

Still missing:

  • Correcting the MVStore serialization using serialization tool already implemented by JabRef
  • Describe code addition in CHANGELOG.MD

@alexandre-cremieux
Copy link
Contributor Author

TODO here:

  • add description in CHANGELOG.MD
  • update PR description

After this, the PR should be ready for last review.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your code currently does not meet JabRef's code guidelines.
We use Checkstyle to identify issues.
Please carefully follow the setup guide for the codestyle.
Afterwards, please run checkstyle locally and fix the issues.

In case of issues with the import order, double check that you activated Auto Import.
You can trigger fixing imports by pressing Ctrl+Alt+O to trigger Optimize Imports.

@alexandre-cremieux alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch from e56497e to 16cfd2c Compare January 26, 2025 21:32
@alexandre-cremieux
Copy link
Contributor Author

Ready for review. Normally everything should be there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants