OIDC auth, custom configs #63

ssi444 · 2022-05-17T16:06:18Z

Description

Added the ability to log in via OpenID
Added the ability to install custom configuration files for the cluster
Added the ability to reconfigure the cluster (in particular, update certificates) when expanding it
Added the ability not to change certificates if the cluster composition has not changed, but only the settings have changed.

Issues Resolved

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sergey Shubin <[email protected]>

…and set them /bin/false shell Signed-off-by: Sergey Shubin <[email protected]>

1. Added the ability to log in via OpenID 2. Added the ability to install custom configuration files for the cluster 3. Added the ability to reconfigure the cluster (in particular, update certificates) when expanding it 4. Added the ability not to change certificates if the cluster composition has not changed, but only the settings have changed. Signed-off-by: Sergey Shubin <[email protected]>

peterzhuamazon · 2022-05-17T21:32:15Z

@saravanan30erd @prudhvigodithi please take a look when you have time.
Thanks.

saravanan30erd · 2022-05-18T05:34:06Z

@ssi444 Thanks for the great work.

Since it have lot of changes, I suggest to have separate PR for each of these features which will help us to review and test it in easier way. Becoz we are supporting many linux distro so we might need to test this against most of them.

Signed-off-by: Sergey Shubin <[email protected]>

ssi444 · 2022-05-18T08:41:37Z

@ssi444 Thanks for the great work.

Since it have lot of changes, I suggest to have separate PR for each of these features which will help us to review and test it in easier way. Becoz we are supporting many linux distro so we might need to test this against most of them.

@saravanan30erd The fact is that these changes are logically related and it is better to test them together. For example, OpenID alone makes little sense without custom configuration files.

Also, all new settings in the "default" state should lead to exactly the same result when starting the playbook as before

peterzhuamazon · 2022-05-23T22:52:48Z

Calling @saravanan30erd @prudhvigodithi again for the review.
Thanks.

ssi444 · 2022-05-31T13:51:30Z

@peterzhuamazon @saravanan30erd @prudhvigodithi ping

peterzhuamazon · 2022-06-09T19:02:06Z

@saravanan30erd Please let me know if you have bandwidth to review this.
@ssi444 Sorry we are focusing on two minor releases in the past weeks that is why this is pretty slow.
Thanks.

ssi444 · 2022-06-21T13:59:16Z

@peterzhuamazon @saravanan30erd @prudhvigodithi ping

ssi444 · 2022-06-27T08:49:00Z

@peterzhuamazon @saravanan30erd @prudhvigodithi ping

prudhvigodithi · 2022-06-27T12:11:49Z

Thanks @ssi444, LGTM!
@saravanan30erd @peterzhuamazon any other thoughts and suggestions ?

prudhvigodithi · 2022-06-28T21:27:00Z

roles/linux/opensearch/tasks/security.yml

+#    - name: Security Plugin configuration | debug
+#      debug:
+#        msg: "{{ item }} password: {{ lookup('vars', item + '_password') }}"
+#      with_items: "{{ custom_users_filtered }}"
+#      run_once: true


Hey @ssi444 if this is not required please delete the lines.
Thank you

prudhvigodithi · 2022-06-28T21:30:41Z

roles/linux/opensearch/tasks/security.yml

+    - name: Security Plugin configuration | Set passwords for all users from custom config
+      shell: >
+        sed -i '/hash: / s,{{ lookup('vars', item + '_password') }},'$(bash {{ os_sec_plugin_tools_path }}/hash.sh -p {{ lookup('vars', item + '_password') }} | tail -1)','
+        {{ os_sec_plugin_conf_path }}/internal_users.yml


Hey @ssi444 can the hash value be passed by user directly instead of this being by ad-hoc shell script, as overtime these sed parsing logic would be hard to maintain.

In principle, this is possible. But we need to redo the logic of working with user passwords. This task is done by analogy with the task "- name: Security Plugin configuration | Set the Admin user password". Now passwords are transmitted in clear text and hashed when performing the role. If you transmit hashes, this will lead to the need for additional actions on the part of users (pre-downloading the container from opensearch locally, hashing each password), which is not very convenient, I think.

Ok with me @peterzhuamazon @saravanan30erd if you are ok with this, I can resolve the conversation.
Thank you

prudhvigodithi · 2022-06-28T21:34:47Z

roles/linux/opensearch/tasks/security.yml

@@ -10,6 +20,7 @@
  run_once: true
  register: configuration
  become: false
+# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<


Please remove this line and any of similar lines.

prudhvigodithi · 2022-06-28T21:35:42Z

roles/linux/opensearch/tasks/security.yml

+      environment:
+        JAVA_HOME: "{{ os_home }}/jdk"
+      run_once: true
+      when: configuration.changed or copy_custom_security_configs


@ssi444 so can you add some details on when this block will be triggered, when a password done by user using admin_password or similar?

In the file README.md in the "Prerequisite" section, I added a description of how to set passwords for custom users (line 92-94)
https://github.com/ssi444/opensearch-project_ansible-playbook/blob/4b4bf51a01ac2e6db9a42c78e18914dab695063d/README.md?plain=1#L92

That comes for copy_custom_security_configs I guess, just wanted to know about configuration.changed, when an admin user changed ?

prudhvigodithi · 2022-06-28T21:43:42Z

roles/linux/opensearch/templates/security_plugin_conf.yml

@@ -0,0 +1,287 @@
+---


@ssi444 having the file directly added might not work if backported to 1.x branch, as config file are differnt for 1.x and 2.x security plugin https://github.com/opensearch-project/security/tree/main/config can you add some details here? If they are different then the backpot PR for 1.x should have a different file.

The contents of the security_plugin_config.yml file are taken from OpenSearch v.1.3.2. At the time of writing the patch, version 2.0 has not yet been released. If you compare this file with the file https://github.com/opensearch-project/security/blob/main/config/config.yml then there is no change at all

thanks @ssi444, ok with me, just added a thought as we might not know if i can change for next major versions.
@peterzhuamazon @saravanan30erd

prudhvigodithi · 2022-06-28T21:45:19Z

inventories/opensearch/group_vars/all/all.yml

@@ -34,3 +34,65 @@ cluster_type: multi-node
 os_user: opensearch

 os_dashboards_user: opensearch-dashboards
+
+# Number of days that certificates are valid
+cert_valid_days: 730


Any specific reason for 730? Can we have an empty space if null default as 1yr ?

This value was before my edits, so I left it the same. Probably this period was optimal. I have set 10 years for myself, because I do not see the need for frequent certificate changes in the cluster.

prudhvigodithi · 2022-06-28T21:47:09Z

roles/linux/opensearch/tasks/security.yml

+      with_items: "{{ cert_stat_result.results }}"
+      when: item.stat.exists == False
+
+    - name: Security Plugin configuration | debug 1 - force_update_cert


@ssi444 what are these debug 1 debug 2 ? Can you please add some details.

prudhvigodithi · 2022-06-28T21:49:11Z

roles/linux/opensearch/tasks/security.yml

+
+    - name: Security Plugin configuration | Filter service keys from the list of users
+      set_fact:
+        custom_users_filtered: '{{ custom_users|dict2items|rejectattr("key", "equalto", "_meta")|list|items2dict }}'


@ssi444 can you add some details about this logic?

added comment

Signed-off-by: Sergey Shubin <[email protected]>

ssi444 · 2022-06-29T09:38:26Z

@prudhvigodithi check please

peterzhuamazon · 2022-06-30T01:29:36Z

Hi @ssi444,

Just so you know, I was reviewing the PR alongside @prudhvigodithi.
And we think it is best to tackle this PR specifically before touching the other two.

Recently we are a bit busy on our tasks so review will be slow, especially July4 incoming.
Thanks for the patience and we will get this through 😄

Thanks.

prudhvigodithi · 2022-07-12T14:09:14Z

LGTM.
@peterzhuamazon and @saravanan30erd

peterzhuamazon

Thanks @ssi444 for the patience and please create a backport PR by cherry-picking the commit of this PR merge.

I will close your original backport PR.

Edit: I create the backport PR here: #78

Thanks.

* Update dashboards.yml. Fix owner and permissions for home folder Signed-off-by: Sergey Shubin <[email protected]> * The validity period of certificates is set to a variable Signed-off-by: Sergey Shubin <[email protected]> * change HOME directory for {{ os_user }} and {{ os_dashboards_user }} and set them /bin/false shell Signed-off-by: Sergey Shubin <[email protected]> * auth_type (internal, openid). Custom configs, IaC 1. Added the ability to log in via OpenID 2. Added the ability to install custom configuration files for the cluster 3. Added the ability to reconfigure the cluster (in particular, update certificates) when expanding it 4. Added the ability not to change certificates if the cluster composition has not changed, but only the settings have changed. Signed-off-by: Sergey Shubin <[email protected]> * readme. description for OpenID, IaC, custom configuration files Signed-off-by: Sergey Shubin <[email protected]> * refactoring. see opensearch-project#63 Signed-off-by: Sergey Shubin <[email protected]>

* Update dashboards.yml. Fix owner and permissions for home folder Signed-off-by: Sergey Shubin <[email protected]> * The validity period of certificates is set to a variable Signed-off-by: Sergey Shubin <[email protected]> * change HOME directory for {{ os_user }} and {{ os_dashboards_user }} and set them /bin/false shell Signed-off-by: Sergey Shubin <[email protected]> * auth_type (internal, openid). Custom configs, IaC 1. Added the ability to log in via OpenID 2. Added the ability to install custom configuration files for the cluster 3. Added the ability to reconfigure the cluster (in particular, update certificates) when expanding it 4. Added the ability not to change certificates if the cluster composition has not changed, but only the settings have changed. Signed-off-by: Sergey Shubin <[email protected]> * readme. description for OpenID, IaC, custom configuration files Signed-off-by: Sergey Shubin <[email protected]> * refactoring. see #63 Signed-off-by: Sergey Shubin <[email protected]> Co-authored-by: ssi444 <[email protected]>

rodolfovillordo · 2022-07-20T13:40:44Z

Hi team,

Just to let you know, and make cross reference. This PR introduced a few breaking changes on the default deployment. Please see below a list of issues and PRs for the fixes;

Issue: [BUG][Security Plugin Configuration] local action requesting unnecessary privilege escalation #82
- PR: become: false on "Check that the files/internal_users.yml exists" #84
Issue: [BUG][Security Plugin Configuration] securityadmin.sh execution fails #83
- PR: Fix securityadmin.sh when copy_custom_security_configs is False #85
Issue: [BUG][Security Plugin Configuration] Unecessary user left on internal_users.yml #86
- PR: Remove unecessary logstash user from default deployment #87

Regards

peterzhuamazon · 2022-07-20T19:06:06Z

@ssi444 ^^ Thanks.

peterzhuamazon · 2022-07-20T19:07:13Z

Hi team,

Just to let you know, and make cross reference. This PR introduced a few breaking changes on the default deployment. Please see below a list of issues and PRs for the fixes;

* Issue: [[BUG][Security Plugin Configuration] local action requesting unnecessary privilege escalation #82](https://github.com/opensearch-project/ansible-playbook/issues/82)
  
  * PR: [become: false on "Check that the files/internal_users.yml exists" #84](https://github.com/opensearch-project/ansible-playbook/pull/84)

* Issue: [[BUG][Security Plugin Configuration] securityadmin.sh execution fails #83](https://github.com/opensearch-project/ansible-playbook/issues/83)
  
  * PR: [Fix securityadmin.sh when copy_custom_security_configs is False #85](https://github.com/opensearch-project/ansible-playbook/pull/85)

* Issue: [[BUG][Security Plugin Configuration] Unecessary user left on internal_users.yml #86](https://github.com/opensearch-project/ansible-playbook/issues/86)
  
  * PR: [Remove unecessary logstash user from default deployment #87](https://github.com/opensearch-project/ansible-playbook/pull/87)

Regards

Thanks @rodolfovillordo we will take a look soon.
Please bare with us as we are a bit busy with our Jenkins migration recently.
Will definitely get it merged eventually 😄

* Update dashboards.yml. Fix owner and permissions for home folder Signed-off-by: Sergey Shubin <[email protected]> * The validity period of certificates is set to a variable Signed-off-by: Sergey Shubin <[email protected]> * change HOME directory for {{ os_user }} and {{ os_dashboards_user }} and set them /bin/false shell Signed-off-by: Sergey Shubin <[email protected]> * auth_type (internal, openid). Custom configs, IaC 1. Added the ability to log in via OpenID 2. Added the ability to install custom configuration files for the cluster 3. Added the ability to reconfigure the cluster (in particular, update certificates) when expanding it 4. Added the ability not to change certificates if the cluster composition has not changed, but only the settings have changed. Signed-off-by: Sergey Shubin <[email protected]> * readme. description for OpenID, IaC, custom configuration files Signed-off-by: Sergey Shubin <[email protected]> * refactoring. see opensearch-project/ansible-playbook#63 Signed-off-by: Sergey Shubin <[email protected]>

ssi444 added 4 commits May 12, 2022 14:07

Update dashboards.yml. Fix owner and permissions for home folder

6e74ddd

Signed-off-by: Sergey Shubin <[email protected]>

The validity period of certificates is set to a variable

9925a0c

Signed-off-by: Sergey Shubin <[email protected]>

change HOME directory for {{ os_user }} and {{ os_dashboards_user }} …

0a5885e

…and set them /bin/false shell Signed-off-by: Sergey Shubin <[email protected]>

ssi444 requested a review from a team as a code owner May 17, 2022 16:06

peterzhuamazon requested review from saravanan30erd and prudhvigodithi May 17, 2022 21:32

readme. description for OpenID, IaC, custom configuration files

4b4bf51

Signed-off-by: Sergey Shubin <[email protected]>

ssi444 mentioned this pull request May 24, 2022

[Enhancement][IaC] Ability to manage ISM policies and index patterns via role configuration files #66

Open

prudhvigodithi reviewed Jun 28, 2022

View reviewed changes

refactoring. see opensearch-project#63

394a9d5

Signed-off-by: Sergey Shubin <[email protected]>

prudhvigodithi approved these changes Jul 13, 2022

View reviewed changes

peterzhuamazon approved these changes Jul 13, 2022

View reviewed changes

peterzhuamazon merged commit 436136d into opensearch-project:main Jul 13, 2022

peterzhuamazon mentioned this pull request Jul 13, 2022

[Backport 1.x] OIDC auth, custom configs #78

Merged

rodolfovillordo mentioned this pull request Jul 20, 2022

Remove unecessary logstash user from default deployment #87

Closed

OIDC auth, custom configs #63

OIDC auth, custom configs #63

Conversation

ssi444 commented May 17, 2022

Description

Issues Resolved

peterzhuamazon commented May 17, 2022

saravanan30erd commented May 18, 2022

ssi444 commented May 18, 2022 • edited Loading

peterzhuamazon commented May 23, 2022

ssi444 commented May 31, 2022

peterzhuamazon commented Jun 9, 2022

ssi444 commented Jun 21, 2022

ssi444 commented Jun 27, 2022

prudhvigodithi commented Jun 27, 2022

Choose a reason for hiding this comment

ssi444 Jun 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ssi444 Jun 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ssi444 Jun 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ssi444 commented Jun 29, 2022

peterzhuamazon commented Jun 30, 2022

prudhvigodithi commented Jul 12, 2022

peterzhuamazon left a comment • edited Loading

Choose a reason for hiding this comment

rodolfovillordo commented Jul 20, 2022

peterzhuamazon commented Jul 20, 2022

peterzhuamazon commented Jul 20, 2022

ssi444 commented May 18, 2022 •

edited

Loading

ssi444 Jun 29, 2022 •

edited

Loading

ssi444 Jun 29, 2022 •

edited

Loading

ssi444 Jun 29, 2022 •

edited

Loading

peterzhuamazon left a comment •

edited

Loading