Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add annotation similar to @JsonUnwrapped on document fields #2774

Open
AlcipPopa opened this issue Nov 18, 2023 · 5 comments · May be fixed by #2969
Open

Add annotation similar to @JsonUnwrapped on document fields #2774

AlcipPopa opened this issue Nov 18, 2023 · 5 comments · May be fixed by #2969
Labels
type: enhancement A general enhancement

Comments

@AlcipPopa
Copy link

This is a feature request.

I'm migrating a platform from Apache Solr to ElasticSearch and I'm struggling in creating dynamic fields like Apache Solr used to have. I need facets (aka "aggregations" in ElasticSearch) on these dynamic fields. On Solr that was easily done with an annotation on a Map:

@Dynamic("*_str")
@Field("attributes")
private Map<String, String> attributes;

Instead, with ElasticSearch, I need to create a dynamic template for the index like this:

{
    "mappings": {
        "dynamic_templates": [
            {
                "_str": {
                    "match": "*_str",
                    "match_mapping_type": "string",
                    "mapping": {
                        "type": "keyword"
                    }
                }
            }
        ]
    }
}

The main issue is that when I now save a document having

@Field("attributes")
private Map<String, String> attributes;

, Spring is inputting a payload like

{
  // other document fields above
  "attributes": {
    "code1_str":"value1",
    "code2_str":"other-value"
  },
  // other document fields below
}

and I can't do any aggregation query to get a result like

{
    "aggregations": {
        "code1_str": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
                {
                    "key": "value1",
                    "doc_count": 1
                },
                {
                    "key": "value2",
                    "doc_count": 1
                }
            ]
        },
        "code2_str": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
                {
                    "key": "other-value",
                    "doc_count": 2
                }
            ]
        }
    }
}

The workaround for this issue is atm a custom Spring Converter<S, T> for the entire document: immagine having to convert 20 or 30 fields.. this would take a lot of work just because of 1 field serialization/deserialization issue.
A solution could be found Instead in having something like @JsonUnwrapped annotation from Jackson library to remove the attributes level from the payload, leading to Spring inputting a payload like:

{
  // other document fields above
  "code1_str":"value1",
  "code2_str":"other-value"
  // other document fields below
}

Could this be feasible in an upcoming release?

@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label Nov 18, 2023
@sothawo
Copy link
Collaborator

sothawo commented Nov 26, 2023

When pulling out the properties from a map into the entity that contains the map, how should this work when reading entities from Elasticsearch?

This works when writing data, but on reading it is not possible to determine which of the fields returned from Elasticsearch should go into properties and which into the map, there is no general way to determine where code1_str should be put into. What if the entity has two maps as properties? In which one should this be read?

What would happen on writing, when an entity has a property prop1, and the map that should be unwrapped as well has a key prop1? Which one should be written and which one overwritten?

@sothawo sothawo added status: waiting-for-feedback We need additional information before we can continue and removed status: waiting-for-triage An issue we've not yet triaged labels Nov 26, 2023
@AlcipPopa
Copy link
Author

AlcipPopa commented Nov 29, 2023

Hmm yes you're right: the issue is on serialization from JSON to POJO.
Would a wildcard ease the mapping of fields between POJOs and ElasticSearch documents? This would also require a validation of the POJO field names before saving data to ElasticSearch, to avoid having multiple matching fields.

So this should be possible to save on ElasticSearch:

Schermata del 2023-11-29 16-36-08

{
  // other document fields above
  "code1_str":"value1",
  "code2_str":"other-value"
  // other document fields below
}

While this should throw a RuntimeException before even saving into ElasticSearch, because of ambiguous mapping:
Schermata del 2023-11-29 16-40-13

P.S.: Don't mind the @dynamic annotation since it's Solr related.

@spring-projects-issues spring-projects-issues added status: feedback-provided Feedback has been provided and removed status: waiting-for-feedback We need additional information before we can continue labels Nov 29, 2023
@sothawo
Copy link
Collaborator

sothawo commented Dec 1, 2023

We'd need to use some new annotation to prevent a mappping to be written for these fields. That idea probably will work.

I'm not yet sure if this can be done only for primitive types in the map like Map<String, String> or if this can work for something like Map<String,E> , might be possible.

@sothawo sothawo added type: enhancement A general enhancement and removed status: feedback-provided Feedback has been provided labels Dec 1, 2023
@AlcipPopa
Copy link
Author

I wouldn't use a generic value for the map.. at least not for a 1st version :)

@youssef3wi
Copy link
Contributor

You can use the @DynamicTemplates annotation can generate the mapping definition you need.

After that the converter should handle the de/serialization process between the Elasticsearch request/response and the Document (Java object) to manage this definition.

@youssef3wi youssef3wi linked a pull request Aug 20, 2024 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: enhancement A general enhancement
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants