Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature/multi_tenancy] Fix SdkClient credential handling for AWS endpoints #2934

Merged
merged 3 commits into from
Sep 13, 2024
Merged

[Feature/multi_tenancy] Fix SdkClient credential handling for AWS endpoints #2934

merged 3 commits into from
Sep 13, 2024

Conversation

dbwiddis
Copy link
Member

Description

Refactors SdkClient code as required to support (working) DDB/AOSS integration tests

  • removes the temporary "local" DDB implementation
  • fetches credentials from system environment (and updates gradle build to put them there)

Check List

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

dhrubo-os
dhrubo-os reviewed Sep 13, 2024
View reviewed changes
plugin/src/main/java/org/opensearch/ml/sdkclient/SdkClientFactory.java Outdated
if (Strings.isBlank(remoteMetadataEndpoint) || Strings.isBlank(region)) {
throw new OpenSearchException(clientType + " client requires a metadata endpoint and region.");
}
if (!"es".equals(serviceName) && !"aoss".equals(serviceName)) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if an opensource community wants to use some other service other than es or aoss? I think this part needs to be generic? Isn't that the whole point we moved this service name to a settings REMOTE_METADATA_SERVICE_NAME_KEY?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are specific to case SdkClientSettings.AWS_OPENSEARCH_SERVICE: and case SdkClientSettings.AWS_DYNAMO_DB for the remoteMetadata type. Any other service or any other provider can be added to that configuration.

These two particular clients use the AWS SDK for connection and "es" and "aoss" are the only documented options right now, see https://opensearch.org/docs/latest/clients/java/#connecting-to-amazon-opensearch-service

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also relevant: https://github.com/opensearch-project/data-prepper/blob/627ae9ffdb2de5dda84566e7b6a4f130f5005210/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/source/opensearch/worker/client/OpenSearchClientFactory.java#L54-L55

Copy link
Member Author

@dbwiddis dbwiddis Sep 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if an opensource community wants to use some other service other than es or aoss? I think this part needs to be generic? Isn't that the whole point we moved this service name to a settings REMOTE_METADATA_SERVICE_NAME_KEY?

For a concrete example, let's say OCI wants to have a client connect to their service through the OCI OpenSearch SDK. They would:

  1. Define a new SdkClientSettings.REMOTE_METADATA_TYPE. We already have "RemoteOpenSearch" (a cluster anywhere) and "AWSOpenSearchService" (AOS). They'd add a new "OCISearchService" (or whatever name)
  2. Instead of calling validation via validateAWSParams() which requires an endpoint, region, and one of two service names, they'd create a new validateOCIParams() checker method for whichever settings they needed to support methods on https://github.com/oracle/oci-java-sdk. They could probably use the Service Name Key setting for "OPENSEARCHCLUSTER" if relevant (I'm guessing here, see here). So the setting itself is open-ended, it's only in combination with the metadata type that it gains any restrictions on what it can be.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for detailed explanation. Makes sense, sounds good to me.

@dhrubo-os dhrubo-os merged commit f5345da into opensearch-project:feature/multi_tenancy Sep 13, 2024
7 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dhrubo-os dhrubo-os dhrubo-os approved these changes

@b4sjoo b4sjoo Awaiting requested review from b4sjoo b4sjoo is a code owner

@jngz-es jngz-es Awaiting requested review from jngz-es jngz-es is a code owner

@model-collapse model-collapse Awaiting requested review from model-collapse model-collapse is a code owner

@rbhavna rbhavna Awaiting requested review from rbhavna rbhavna is a code owner

@ylwu-amzn ylwu-amzn Awaiting requested review from ylwu-amzn ylwu-amzn is a code owner

@zane-neo zane-neo Awaiting requested review from zane-neo zane-neo is a code owner

@Zhangxunmt Zhangxunmt Awaiting requested review from Zhangxunmt Zhangxunmt is a code owner

@austintlee austintlee Awaiting requested review from austintlee austintlee is a code owner

@HenryL27 HenryL27 Awaiting requested review from HenryL27 HenryL27 is a code owner

@samuel-oci samuel-oci Awaiting requested review from samuel-oci samuel-oci is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dbwiddis @dhrubo-os