Scaling #264

naredula-jana · 2018-11-23T12:59:54Z

Added scaling feature. scaling means vertical scaling of data nodes. details of scaling feature are available here: https://github.com/naredula-jana/elasticsearch-operator/blob/scaling/docs/scaling.md .

…nding.

stevesloka

Hey thanks for this PR! My biggest question is why we have a statefulset per data node? Also there are places with hard coded ip's, not sure why those are now requirements. We should have a system that's dynamic and adapts to change. Also, there are various code comments, new lines, etc all throughout, not sure if this is a final PR or still WIP. Thanks!

cmd/operator/main.go

docs/scaling.md

pkg/k8sutil/deployments.go

stevesloka · 2018-11-23T19:42:02Z

pkg/k8sutil/es_crud.go

+	}
+	return ret;
+}
+func es_checkForGreen(es_ip string)(error){ /* TODO */


My original thoughts around the ES status was to make a set of stats that are broken out. Doing this allows us to use the same ES status checks and also enable other features like rolling restarts.

Can you pls provide more details on this, Do you expect any change here.

stevesloka · 2018-11-27T16:57:06Z

@naredula-jana you can change how statefulsets roll out changes which allow the operator to manage the statefulset in the manner it sees fit: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#on-delete

naredula-jana · 2018-11-30T12:25:50Z

@naredula-jana you can change how statefulsets roll out changes which allow the operator to manage the statefulset in the manner it sees fit: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#on-delete

Changed to single statefulset using on-delete, and the code is committed.

stevesloka · 2018-12-08T03:02:12Z

Thanks! Let me test this out @naredula-jana.

The only other thing that would be great to have is tests for all this code.

naredula-jana · 2018-12-08T04:52:11Z

Thanks! Let me test this out @naredula-jana.

The only other thing that would be great to have is tests for all this code.
Regarding Testing, I was not able to test on a local storage, able to test scaling feature successfully on block network storage on azure with latest committed code.

without scaling patch: does not support/not tested on local storage.
With scaling patch + multiple statefullset : supports local storage/nfs, scaling was sucessfull tested with local and network storage.
With scaling patch + single statefullset (latest code) : does not support local storage, this is same as without scaling patch. I have placed TODO in k8util.go(line-449), the reason is all data nodes as same mount point, In the multiple statefullset each datanode as separate mount point so each dfatanodes stores in a multiple sub directories in local/nfs.

Is there any way to make local/nfs storage to work with the latest code or single statefullset?.

stevesloka · 2018-12-18T19:45:58Z

Hey @naredula-jana, I've been tied up in Kubecon last week. Could you look to add tests to this PR? I don't see any included.

naredula-jana · 2018-12-20T10:24:57Z

Hey @naredula-jana, I've been tied up in Kubecon last week. Could you look to add tests to this PR? I don't see any included.

I have added Test cases in the latest commit.

stevesloka · 2018-12-21T16:09:27Z

@naredula-jana what I'm looking for are unit tests, things that run on each build so we can verify that the code works as expected. Manual tests are ok, but they are difficult to run and are time-consuming.

naredula-jana · 2018-12-22T09:23:40Z

@naredula-jana what I'm looking for are unit tests, things that run on each build so we can verify that the code works as expected. Manual tests are ok, but they are difficult to run and are time-consuming.

I have added detail steps for each testcase. Automising the testcases to run during build will be difficult because in all testcases, the test case need to communicate with k8, operator and elastic cluster to verify the state of test if it is need to automise, communicating and verfying the test need lot code if it need to build from scratch.

naredula-jana · 2018-12-24T08:03:48Z

@naredula-jana what I'm looking for are unit tests, things that run on each build so we can verify that the code works as expected. Manual tests are ok, but they are difficult to run and are time-consuming.

I have added detail steps for each testcase. Automising the testcases to run during build will be difficult because in all testcases, the test case need to communicate with k8, operator and elastic cluster to verify the state of test if it is need to automise, communicating and verfying the test need lot code if it need to build from scratch.

Is there any sample/example unit test that runs at build time and communicate with k8,ES operator and ES cluster ?. In the above testcases the test script need to communicate with 3 components: a) K8, b) ES operator c) ES cluster.

stevesloka · 2019-01-02T16:42:16Z

@naredula-jana I think you are referencing integration tests which would run on a live cluster. I'm just asking for unit tests which test out the functionality of how each method functions. e.g. if you pass nil to it, what happens? If you pass the wrong value (or right value), does the code do what it's supposed to.

I've been very lax with unit tests, but that shouldn't be the case. We need to have good coverage to understand when a change breaks the logic of something somewhere.

naredula-jana · 2019-01-02T20:35:37Z

@naredula-jana I think you are referencing integration tests which would run on a live cluster. I'm just asking for unit tests which test out the functionality of how each method functions. e.g. if you pass nil to it, what happens? If you pass the wrong value (or right value), does the code do what it's supposed to.

I've been very lax with unit tests, but that shouldn't be the case. We need to have good coverage to understand when a change breaks the logic of something somewhere.

I am preparing unit tests, I will commit tests as soon as possible. Thanks Jana

naredula-jana · 2019-01-03T15:50:30Z

@naredula-jana I think you are referencing integration tests which would run on a live cluster. I'm just asking for unit tests which test out the functionality of how each method functions. e.g. if you pass nil to it, what happens? If you pass the wrong value (or right value), does the code do what it's supposed to.
I've been very lax with unit tests, but that shouldn't be the case. We need to have good coverage to understand when a change breaks the logic of something somewhere.

I am preparing unit tests, I will commit tests as soon as possible. Thanks Jana

unit test code added in the latest commit.

stevesloka · 2019-01-13T02:37:58Z

Looks like there needs a rebase @naredula-jana, some files are out of sync.

naredula-jana · 2019-01-13T12:04:05Z

Looks like there needs a rebase @naredula-jana, some files are out of sync.

Scaling feature needed an extra argument to this function "p.k8sclient.CreateDataNodeDeployment", this need code merge , due to this there are some conflicts.

naredula-jana · 2019-01-13T12:11:02Z

Looks like there needs a rebase @naredula-jana, some files are out of sync.

Scaling feature needed an extra argument to this function "p.k8sclient.CreateDataNodeDeployment", this need code merge , due to this there are some conflicts.

Do you want me to pull the latest master code and apply the "scaling" patch to it?.

naredula-jana · 2019-01-14T10:45:33Z

Looks like there needs a rebase @naredula-jana, some files are out of sync.

Created New PR (scaling-new #277 ) rebased with latest master.

naredula-jana · 2019-01-21T12:13:03Z

Looks like there needs a rebase @naredula-jana, some files are out of sync.

Created New PR (scaling-new #277 ) rebased with latest master.

@stevesloka Any update to this PR or scaling-new #277 ?. Waiting for the code merging.

stevesloka · 2019-01-23T02:56:26Z

Are the 2 PR's the same? Can you please close one or the other? Then we can work on getting the chosen one working.

naredula-jana · 2019-01-23T03:10:56Z

Are the 2 PR's the same? Can you please close one or the other? Then we can work on getting the chosen one working.

Yes, Both are same. Pls pick any one of the PR for merging, remaining one can be closed. The PR scaling-new #277 is the rebased on the new master. but PR scaling-new #277 getting cycling dependency in one of the unit test that is not related to the scaling patch.

naredula-jana · 2019-01-27T03:39:55Z

Scaling path is rebased to the latest master at PR scaling-new #277 . So closing this PR.

naredula-jana · 2019-01-28T08:20:57Z

Old PR is closed. the new PR is rebased on the latest master and all the unit test works.

…

On Wed, 23 Jan 2019 at 08:26, Steve Sloka ***@***.***> wrote: Are the 2 PR's the same? Can you please close one or the other? Then we can work on getting the chosen one working. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#264 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAmJq1ZmntcvaxDB11KZzi1vtA57d2ieks5vF89agaJpZM4Ywnyt> .

naredula-jana added 8 commits November 16, 2018 11:10

doc change

34acbaa

doc changes

a51e122

changes: scaling feature changes, code clean up and stress test is pe…

751dd45

…nding.

code cleanup.

b57c502

code cleanup

46d1339

code cleanup.

96ad2ca

compilation errors addressed.

388b21f

Document changes.

67218cc

stevesloka requested changes Nov 23, 2018

View reviewed changes

naredula-jana added 3 commits November 24, 2018 14:15

code cleanup.

06bed86

main.go changes.

890e511

learning masterip from service dynamically.

0865221

All data nodes in a single statefulset.

4cd85d8

Test Cases added to doc/scaling.md

791dc6f

Detail steps for each testcase are added.

ed52b79

Unit test code and some fixes added.

b8c9ee4

resolving conflicts from the master.

730dbe3

extra space removed.

572d6d6

naredula-jana mentioned this pull request Jan 14, 2019

Scaling new #277

Open

enabled verbose.

4b38fe2

naredula-jana closed this Jan 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scaling #264

Scaling #264

naredula-jana commented Nov 23, 2018 •

edited

Loading

stevesloka left a comment

stevesloka Nov 23, 2018

naredula-jana Dec 5, 2018

stevesloka commented Nov 27, 2018

naredula-jana commented Nov 30, 2018 •

edited

Loading

stevesloka commented Dec 8, 2018

naredula-jana commented Dec 8, 2018

stevesloka commented Dec 18, 2018

naredula-jana commented Dec 20, 2018

stevesloka commented Dec 21, 2018

naredula-jana commented Dec 22, 2018

naredula-jana commented Dec 24, 2018

stevesloka commented Jan 2, 2019

naredula-jana commented Jan 2, 2019

naredula-jana commented Jan 3, 2019

stevesloka commented Jan 13, 2019 •

edited

Loading

naredula-jana commented Jan 13, 2019

naredula-jana commented Jan 13, 2019

naredula-jana commented Jan 14, 2019

naredula-jana commented Jan 21, 2019

stevesloka commented Jan 23, 2019

naredula-jana commented Jan 23, 2019

naredula-jana commented Jan 27, 2019

naredula-jana commented Jan 28, 2019 via email

Scaling #264

Scaling #264

Conversation

naredula-jana commented Nov 23, 2018 • edited Loading

stevesloka left a comment

Choose a reason for hiding this comment

stevesloka Nov 23, 2018

Choose a reason for hiding this comment

naredula-jana Dec 5, 2018

Choose a reason for hiding this comment

stevesloka commented Nov 27, 2018

naredula-jana commented Nov 30, 2018 • edited Loading

stevesloka commented Dec 8, 2018

naredula-jana commented Dec 8, 2018

stevesloka commented Dec 18, 2018

naredula-jana commented Dec 20, 2018

stevesloka commented Dec 21, 2018

naredula-jana commented Dec 22, 2018

naredula-jana commented Dec 24, 2018

stevesloka commented Jan 2, 2019

naredula-jana commented Jan 2, 2019

naredula-jana commented Jan 3, 2019

stevesloka commented Jan 13, 2019 • edited Loading

naredula-jana commented Jan 13, 2019

naredula-jana commented Jan 13, 2019

naredula-jana commented Jan 14, 2019

naredula-jana commented Jan 21, 2019

stevesloka commented Jan 23, 2019

naredula-jana commented Jan 23, 2019

naredula-jana commented Jan 27, 2019

naredula-jana commented Jan 28, 2019 via email

naredula-jana commented Nov 23, 2018 •

edited

Loading

naredula-jana commented Nov 30, 2018 •

edited

Loading

stevesloka commented Jan 13, 2019 •

edited

Loading