Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump JSON libs #2926

Merged

Conversation

willyborankin
Copy link
Collaborator

@willyborankin willyborankin commented Jul 2, 2023

Description

Bump JSON libraries to latest versions:

  • jjwt-api - to 0.11.5
  • json-flattener -0.16.4
  • zjsonpatch - 0.4.14
  • json-path -2.8
  • json-smart - 2.4.11

Regarding json-flattener I have a question. All functionality which we use in our case can be replaced with such class:

package org.opensearch.security.compliance;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.JsonNode;
import org.apache.kafka.common.protocol.types.Field;
import org.opensearch.core.common.Strings;
import org.opensearch.security.DefaultObjectMapper;

import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Set;

public class JsonFlattener {

    public static Map<String, Object> flatten(final byte[] bytes) {
        try {
            final TypeReference<Map<String, Object>> typeReference = new TypeReference<>() {
            };
            final Map<String, Object> jsonMap = DefaultObjectMapper.objectMapper.readValue(bytes, typeReference);
            final Map<String, Object> flattenMap = new LinkedHashMap<>();
            flattenEntries("", jsonMap.entrySet(), flattenMap);
            return flattenMap;
        } catch (final IOException ioe) {
            throw new IllegalArgumentException("Unparseable json", ioe);
        }
    }

    private static void flattenEntries(String prefix, final Iterable<Map.Entry<String, Object>> entries, final Map<String, Object> result) {
        if (!Strings.isNullOrEmpty(prefix)) {
            prefix += ".";
        }

        for (final Map.Entry<String, Object> e : entries) {
            flattenElement(prefix.concat(e.getKey()), e.getValue(), result);
        }
    }

    private void flattenElement(String prefix, final Object source, final Map<String, Object> result) {
        if (source instanceof Iterable) {
            flattenCollection(prefix, (Iterable<Object>) source, result);
        }
        if (source instanceof Map) {
            flattenEntries(prefix, ((Map<String, Object>) source).entrySet(), result);
        }
        result.put(prefix, source);
    }

    private void flattenCollection(String prefix, final Iterable<Object> objects, final Map<String, Object> result) {
        int counter = 0;
        for (final Object o : objects) {
            flattenElement(prefix + "[" + counter + "]", o, result);
            counter++;
        }
    }

}

So the question is: Do we need yet another library for such functionality?

Issues Resolved

Testing

[Please provide details of testing done: unit testing, integration testing and manual testing]

Check List

  • New functionality includes testing
  • New functionality has been documented
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@codecov
Copy link

codecov bot commented Jul 2, 2023

Codecov Report

Merging #2926 (05ccc63) into main (e5348eb) will decrease coverage by 0.15%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #2926      +/-   ##
============================================
- Coverage     62.47%   62.32%   -0.15%     
+ Complexity     3380     3369      -11     
============================================
  Files           267      267              
  Lines         19772    19772              
  Branches       3356     3356              
============================================
- Hits          12352    12323      -29     
- Misses         5780     5806      +26     
- Partials       1640     1643       +3     
Impacted Files Coverage Δ
...nsearch/security/compliance/FieldReadCallback.java 55.55% <100.00%> (ø)

... and 9 files with indirect coverage changes

Copy link
Contributor

@stephen-crawford stephen-crawford left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left one comment about extracting to a version variable for the JWT libraries but overall looks good to me. :)

build.gradle Outdated Show resolved Hide resolved
cwperks
cwperks previously approved these changes Jul 3, 2023
Copy link
Member

@cwperks cwperks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @willyborankin. If that is all we use the json-flattener for than I would be in favor of less dependencies. If I am understanding it correctly, it is recursively going through a JSON structure and flattening it so all.entries.are.dot.separated and there is no object nesting?

@stephen-crawford
Copy link
Contributor

HI @cwperks, I opened this issue as a follow up to address your concern: #2930.

That way we can get this PR unblocked and still review the use.

@willyborankin
Copy link
Collaborator Author

willyborankin commented Jul 3, 2023

HI @cwperks, I opened this issue as a follow up to address your concern: #2930.

That way we can get this PR unblocked and still review the use.

Hi @scrawfor99, I was about to do the same, thank you. I think it's better to investigate as a first step.

@cwperks
Copy link
Member

cwperks commented Jul 11, 2023

This PR looks good to me once the conflict in build.gradle is resolved

Signed-off-by: Andrey Pleskach <[email protected]>
@willyborankin
Copy link
Collaborator Author

This PR looks good to me once the conflict in build.gradle is resolved

done

@stephen-crawford stephen-crawford merged commit 0e6608d into opensearch-project:main Jul 11, 2023
54 of 55 checks passed
@peternied peternied added backport 2.x backport to 2.x branch v2.10.0 Issues targeting release v2.10.0 labels Aug 2, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/security/backport-2.x
# Create a new branch
git switch --create backport/backport-2926-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0e6608d938af44b8db31440f94cfbb39d9a97d98
# Push it to GitHub
git push --set-upstream origin backport/backport-2926-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-2926-to-2.x.

peternied added a commit to peternied/security that referenced this pull request Aug 3, 2023
This change combines the many updates from the following commits:
* 5f62e8a dependabot: bump commons-io:commons-io from 2.11.0 to 2.13.0 (opensearch-project#3074)
* 2f69a10 bump com.github.wnameless.json:json-base from 2.4.0 to 2.4.1 (opensearch-project#3062)
* c0e50da dependabot: bump org.cryptacular:cryptacular from 1.2.4 to 1.2.5 (opensearch-project#3071)
* d3488e8 dependabot: bump kafka_version from 3.5.0 to 3.5.1 (opensearch-project#3041)
* ab6778d Update ospackage, checker-qual, zcxvbn and error_prone_annotations, camel-xmlsecurity (opensearch-project#3023)
* 0e6608d Bump JSON libs (opensearch-project#2926)
* df07bea SAML 4.3.0 addition persmission (opensearch-project#2987)
* e5348eb Change maven repo location for compatibility check (opensearch-project#2980)
* 4a1ec53 Bump jaxb to 2.3.8 (opensearch-project#2977)
* 9599155 Bump guava to 32.1.1-jre (opensearch-project#2976)
* 06eed60 dependabot: bump org.glassfish.jaxb:jaxb-runtime from 2.3.4 to 4.0.3 (opensearch-project#2970)
* 1113244 Bump eventbus to 3.3.1 (opensearch-project#2965)
* 99ff7b3 dependabot: bump org.apache.bcel:bcel from 6.6.0 to 6.7.0 (opensearch-project#2969)
* 0794c3f dependabot: bump jakarta.xml.bind:jakarta.xml.bind-api (opensearch-project#2968)
* 9e6aab3 dependabot: bump com.google.j2objc:j2objc-annotations from 1.3 to 2.8 (opensearch-project#2963)
* 8227f64 dependabot: bump com.sun.istack:istack-commons-runtime (opensearch-project#2960)
* 8e044a6 dependabot: bump org.apiguardian:apiguardian-api from 1.0.0 to 1.1.2 (opensearch-project#2964)
* 49cbf52 Remove commons-collections 3.2.2 (opensearch-project#2924)
* 092e8f5 Bump SAML libs (opensearch-project#2927)
* 8ab7cb4 Resolve CVE-2023-2976 by forcing use of Guava 32.0.1 (opensearch-project#2937)
* 4eef662 Clean up and bump Apache libs (opensearch-project#2925)
* 9a72355 Bump BouncyCastle from jdk15on to jdk15to18 (opensearch-project#2901)
* e4f4817 [Enhancement] Parallel test jobs for CI (opensearch-project#2861)
* d871af3 Update snappy to 1.1.10.1 and guava to 32.0.1-jre (opensearch-project#2886)
* c808692 Format everything (opensearch-project#2866)

Signed-off-by: Peter Nied <[email protected]>
peternied added a commit that referenced this pull request Aug 3, 2023
Manual backports have been failing for a while, specifically the failure
on #2926 clued me
into the issue, we've been manually resolving these conflicts, but we
should be trying to keep everything in sync as much as possible, this
set of backports lumped together attempts to handle that.

### Issues Resolved
- Resolves #3090

### Testing
Lots of `/gradlew testClasses`, CI checks should verify everything else.

Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Andrey Pleskach <[email protected]>
Co-authored-by: Andrey Pleskach <[email protected]>
willyborankin pushed a commit that referenced this pull request Dec 2, 2023
### Description

Implement JsonFlattener helper class as written in #2926 to deprecate
the use of the unnecessary JsonFlattener third party module.

* Category (Enhancement, New feature, Bug fix, Test fix, Refactoring,
Maintenance, Documentation)

Maintenance

* Why these changes are required?

The JsonFlattener module was being utilized in only one place for one
specific purpose, so these functions can be implemented as part of the
OpenSearch codebase instead of importing an unnecessary third party
module.

* What is the old behavior before changes and new behavior after
changes?

Hopefully nothing.

### Issues Resolved
- #2930 

Is this a backport? If so, please add backport PR # and/or commits #

No

### Testing
Tests checked to make sure functions are not broken.

### Check List
- [x] New functionality includes testing
- [x] New functionality has been documented
- [x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

---------

Signed-off-by: Prabhas Kurapati <[email protected]>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Dec 2, 2023
### Description

Implement JsonFlattener helper class as written in #2926 to deprecate
the use of the unnecessary JsonFlattener third party module.

* Category (Enhancement, New feature, Bug fix, Test fix, Refactoring,
Maintenance, Documentation)

Maintenance

* Why these changes are required?

The JsonFlattener module was being utilized in only one place for one
specific purpose, so these functions can be implemented as part of the
OpenSearch codebase instead of importing an unnecessary third party
module.

* What is the old behavior before changes and new behavior after
changes?

Hopefully nothing.

### Issues Resolved
- #2930

Is this a backport? If so, please add backport PR # and/or commits #

No

### Testing
Tests checked to make sure functions are not broken.

### Check List
- [x] New functionality includes testing
- [x] New functionality has been documented
- [x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

---------

Signed-off-by: Prabhas Kurapati <[email protected]>
(cherry picked from commit 87de7e2)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
prabhask5 added a commit to prabhask5/opensearch-security that referenced this pull request Jan 11, 2024
)

### Description

Implement JsonFlattener helper class as written in opensearch-project#2926 to deprecate
the use of the unnecessary JsonFlattener third party module.

* Category (Enhancement, New feature, Bug fix, Test fix, Refactoring,
Maintenance, Documentation)

Maintenance

* Why these changes are required?

The JsonFlattener module was being utilized in only one place for one
specific purpose, so these functions can be implemented as part of the
OpenSearch codebase instead of importing an unnecessary third party
module.

* What is the old behavior before changes and new behavior after
changes?

Hopefully nothing.

### Issues Resolved
- opensearch-project#2930

Is this a backport? If so, please add backport PR # and/or commits #

No

### Testing
Tests checked to make sure functions are not broken.

### Check List
- [x] New functionality includes testing
- [x] New functionality has been documented
- [x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

---------

Signed-off-by: Prabhas Kurapati <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x backport to 2.x branch v2.10.0 Issues targeting release v2.10.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants