Add initial support for Elastic Stack 9.x #2102

mrodm · 2024-09-13T11:15:17Z

Part of elastic/package-registry#1226

Add initial support for 9.0.0 stack in elastic-package.

Changes performed:

By default, elastic-package will use wolfi images as it was for 8.16 stack versions.
If no version is specified, it will report wolfi image instead (TBD).
Added integration test to check elastic-package stack command for 9.0.0-SNAPSHOT

Author's checklist

Enable target in updatecli configuration before merging.
Add integration test for elastic-package stack command with 9.0.0-SNAPSHOT
Test elastic-package stack up -v --version 9.0.0-SNAPSHOT.

mrodm · 2024-09-13T14:55:25Z

internal/install/application_configuration_test.go

 }

 func TestSelectCompleteElasticAgentImageName_ForceSystemDImage(t *testing.T) {
 	version := stackVersion8160
 	selected := selectElasticAgentImageName(version, "systemd")
-	assert.Equal(t, selected, elasticAgentImageName)
+	assert.Equal(t, elasticAgentImageName, selected)


Updated the order to ensure that the values are the expected and the actual ones as required by the assert functions.
https://pkg.go.dev/github.com/stretchr/testify/assert#Equal

mrodm · 2024-09-13T15:56:16Z

internal/install/application_configuration_test.go

@@ -14,43 +14,43 @@ import (
 func TestSelectElasticAgentImageName_NoVersion(t *testing.T) {
 	var version string
 	selected := selectElasticAgentImageName(version, "")
-	assert.Equal(t, selected, elasticAgentLegacyImageName)
+	assert.Equal(t, elasticAgentWolfiImageName, selected)


This is the test changing the value, it would return the wolfi image instead.

In principle it LGTM. Maybe another option is to return an error if the version is empty? Not sure about the implications of the change, but would look safer to me to know where/if we are passing an empty version.

Commands like elastic-package stack down do not set any version, and if a error is returned here, it will fail the command.

mrodm · 2024-09-16T15:54:19Z

/test

mrodm · 2024-09-16T17:01:28Z

/test

mrodm · 2024-09-17T10:46:00Z

.github/workflows/updatecli/updatecli.d/bump-latest-9x-snapshot-version.yml

+# targets:
+#   update-snapshot:
+#     name: '[updatecli] Update latest snapshot to {{ source "latestSnapshot" }}'
+#     kind: file
+#     sourceid: latestSnapshot
+#     scmid: default
+#     spec:
+#       file: Makefile
+#       matchpattern: '(./scripts/test-stack-command.sh) 9\.[^\s]+-SNAPSHOT'
+#       replacepattern: '$1 {{ source "latestSnapshot" }}-SNAPSHOT'


To be enabled before merging

mrodm · 2024-09-17T10:48:10Z

internal/install/application_configuration.go

@@ -153,13 +154,13 @@ func (ac *ApplicationConfiguration) SetCurrentProfile(name string) {
 // This is mandatory as "elastic-agent-complete" is available since 7.15.0-SNAPSHOT.
 func selectElasticAgentImageName(version, agentBaseImage string) string {
 	if version == "" { // as version is optional and can be empty
-		return elasticAgentLegacyImageName
+		return elasticAgentWolfiImageName


WDYT about changing these defaults values to use the wolfi image docker.elastic.co/elastic-agent/elastic-agent-wolfi (e.g. no version specified) instead of the legacy image (beats namespace) docker.elastic.co/beats/elastic-agent?

Or should we keep unchanged this default?

We can adapt this to the new default, yes, if we are sure that in old versions the beats namespace is used.

Yes, the same behaviour is kept.
As a reference, I add here the docker images used in different versions:

7.14.0
$ docker ps --format "{{.Names}} {{.Image}}" |grep elastic-agent elastic-package-stack-elastic-agent-1 docker.elastic.co/beats/elastic-agent:7.14.0 elastic-package-stack-fleet-server-1 docker.elastic.co/beats/elastic-agent:7.14.0

7.15.0
$ docker ps --format "{{.Names}} {{.Image}}" |grep elastic-agent elastic-package-stack-elastic-agent-1 docker.elastic.co/beats/elastic-agent-complete:7.15.0 elastic-package-stack-fleet-server-1 docker.elastic.co/beats/elastic-agent-complete:7.15.0

8.2.0
$ docker ps --format "{{.Names}} {{.Image}}" |grep elastic-agent elastic-package-stack-elastic-agent-1 docker.elastic.co/elastic-agent/elastic-agent-complete:8.2.0 elastic-package-stack-fleet-server-1 docker.elastic.co/elastic-agent/elastic-agent-complete:8.2.0

8.16.0 (SNAPSHOT)
$ docker ps --format "{{.Names}} {{.Image}}" |grep elastic-agent elastic-package-stack-elastic-agent-1 docker.elastic.co/elastic-agent/elastic-agent-wolfi:8.16.0-SNAPSHOT elastic-package-stack-fleet-server-1 docker.elastic.co/elastic-agent/elastic-agent-wolfi:8.16.0-SNAPSHOT

9.0.0 (SNAPSHOT)
$ docker ps --format "{{.Names}} {{.Image}}" |grep elastic-agent elastic-package-stack-elastic-agent-1 docker.elastic.co/elastic-agent/elastic-agent-wolfi:9.0.0-SNAPSHOT elastic-package-stack-fleet-server-1 docker.elastic.co/elastic-agent/elastic-agent-wolfi:9.0.0-SNAPSHOT

This reverts commit 7f12fe8.

mrodm · 2024-09-17T13:44:06Z

.buildkite/scripts/stack_tests_windows.ps1

@@ -45,7 +45,7 @@ fixCRLF

 withGolang $env:GO_VERSION
 withDocker $env:DOCKER_VERSION
-withDockerCompose $env:DOCKER_COMPOSE_VERSION
+withDockerCompose $env:DOCKER_COMPOSE_VERSION.Substring(1)


Required to remove v from the version string for the chocolotey tool. Currently, this environment variable has this value v2.24.1.

Interesting, how has it worked in the past?

Probably, it was failing in the past too. I realized that even if with those commands were failing, the execution of the script continued.
I see that at the beginning of the file is defined

$ErrorActionPreference = "Stop"

but it does not stopped the execution. Not sure if choco command does not return any error in that scenario, or when the error is inside a function there are some differences in how to manage the errors.

mrodm · 2024-09-17T14:05:50Z

internal/install/application_configuration.go

-		logger.Errorf("stack version not in semver format (value: %s): %v", v, err)
-		return elasticAgentLegacyImageName
+		logger.Errorf("stack version not in semver format (value: %s): %v", version, err)
+		return elasticAgentWolfiImageName
 	}

 	disableWolfiImages := false


As it is now, the Elastic Agent image is chosen as:

If stack >= 8.16.0-SNAPSHOT,

by default it is used wolfi image_ docker.elastic.co/elastic-agent/elastic-agent-wolfi.

if ELASTIC_PACKAGE_DISABLE_ELASTIC_AGENT_WOLFI=true, then it is selected the Elastic Agent image as before. So if stack >= 7.15.0-SNAPSHOT, it will be used the complete image.

If the configuration file of the system test has base_image: complete:

It will be used the complete version of the Elastic Agent image if the stack version allows it.

If the configuration file of the system test has base_image: systemd:

It will be used the systemd version of the Elastic Agent image. That means docker.elastic.co/elastic-agent/elastic-agent or docker.elastic.co/beats/elastic-agent depending on the stack version.

Would that be ok? Specially if stack version is set to 9.0.0 and the environment variable to disable wolfi is set to true.

If the same set of images is going to be available in 9.0 keeping the same logic sounds good to me.

jsoriano · 2024-09-17T19:20:06Z

.buildkite/scripts/stack_tests_windows.ps1

@@ -45,7 +45,7 @@ fixCRLF

 withGolang $env:GO_VERSION
 withDocker $env:DOCKER_VERSION
-withDockerCompose $env:DOCKER_COMPOSE_VERSION
+withDockerCompose $env:DOCKER_COMPOSE_VERSION.Substring(1)


Interesting, how has it worked in the past?

jsoriano · 2024-09-17T19:22:08Z

internal/install/application_configuration.go

@@ -153,13 +154,13 @@ func (ac *ApplicationConfiguration) SetCurrentProfile(name string) {
 // This is mandatory as "elastic-agent-complete" is available since 7.15.0-SNAPSHOT.
 func selectElasticAgentImageName(version, agentBaseImage string) string {
 	if version == "" { // as version is optional and can be empty
-		return elasticAgentLegacyImageName
+		return elasticAgentWolfiImageName


We can adapt this to the new default, yes, if we are sure that in old versions the beats namespace is used.

jsoriano · 2024-09-17T19:25:20Z

internal/install/application_configuration_test.go

@@ -14,43 +14,43 @@ import (
 func TestSelectElasticAgentImageName_NoVersion(t *testing.T) {
 	var version string
 	selected := selectElasticAgentImageName(version, "")
-	assert.Equal(t, selected, elasticAgentLegacyImageName)
+	assert.Equal(t, elasticAgentWolfiImageName, selected)


In principle it LGTM. Maybe another option is to return an error if the version is empty? Not sure about the implications of the change, but would look safer to me to know where/if we are passing an empty version.

…nt-9.0

elasticmachine · 2024-09-18T14:31:40Z

💚 Build Succeeded

Buildkite Build
Commit: 168ae0b

History

💔 Build #4032 failed afee3c6
💔 Build #4028 failed 40deb26
💔 Build #4027 failed d918cc2
💔 Build #4026 failed 568547b
💔 Build #4023 failed 3468618

cc @mrodm

mrodm self-assigned this Sep 13, 2024

mrodm mentioned this pull request Sep 13, 2024

EPR returns almost no packages when kibana.version of 9.0.0 is provided elastic/package-registry#1226

Closed

4 tasks

mrodm force-pushed the support-elastic-agent-9.0 branch from caca34f to 0f4873e Compare September 13, 2024 14:36

mrodm commented Sep 13, 2024

View reviewed changes

mrodm force-pushed the support-elastic-agent-9.0 branch from 2f60421 to 16f9a3c Compare September 16, 2024 09:01

mrodm commented Sep 17, 2024

View reviewed changes

mrodm added 10 commits September 17, 2024 12:49

Update default images if no version specified

164e1ca

Add test stack command for 9.x stack

1135e80

Update tests

2636638

Create updatecli config for 9.x-snapshot

031aa4b

Update names and ids github workflows

03ad1e9

disable target temporarily

0dd9078

Remove prefix 'v' from docker compose version

d447532

Test without installing docker/docker-compose

10a5d76

Show version in error log message

332c2e5

Revert "Test without installing docker/docker-compose"

d918cc2

This reverts commit 7f12fe8.

mrodm force-pushed the support-elastic-agent-9.0 branch from 568547b to d918cc2 Compare September 17, 2024 10:49

Remove TODO message

40deb26

mrodm commented Sep 17, 2024

View reviewed changes

mrodm marked this pull request as ready for review September 17, 2024 14:06

mrodm requested a review from a team as a code owner September 17, 2024 14:06

jsoriano approved these changes Sep 17, 2024

View reviewed changes

mrodm added 2 commits September 18, 2024 10:22

Enable updatecli configuration

afee3c6

Merge remote-tracking branch 'upstream/main' into support-elastic-age…

168ae0b

…nt-9.0

jsoriano approved these changes Sep 18, 2024

View reviewed changes

mrodm merged commit f64d3fc into elastic:main Sep 19, 2024
3 of 4 checks passed

mrodm deleted the support-elastic-agent-9.0 branch September 19, 2024 08:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial support for Elastic Stack 9.x #2102

Add initial support for Elastic Stack 9.x #2102

mrodm commented Sep 13, 2024 •

edited

Loading

mrodm Sep 13, 2024

mrodm Sep 13, 2024

jsoriano Sep 17, 2024

mrodm Sep 18, 2024

mrodm commented Sep 16, 2024

mrodm commented Sep 16, 2024

mrodm Sep 17, 2024

mrodm Sep 17, 2024 •

edited

Loading

jsoriano Sep 17, 2024

mrodm Sep 18, 2024

mrodm Sep 17, 2024

jsoriano Sep 17, 2024

mrodm Sep 18, 2024 •

edited

Loading

mrodm Sep 17, 2024

jsoriano Sep 17, 2024

jsoriano Sep 17, 2024

jsoriano Sep 17, 2024

jsoriano Sep 17, 2024

elasticmachine commented Sep 18, 2024

Add initial support for Elastic Stack 9.x #2102

Add initial support for Elastic Stack 9.x #2102

Conversation

mrodm commented Sep 13, 2024 • edited Loading

Author's checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrodm commented Sep 16, 2024

mrodm commented Sep 16, 2024

Choose a reason for hiding this comment

mrodm Sep 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrodm Sep 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Sep 18, 2024

💚 Build Succeeded

History

mrodm commented Sep 13, 2024 •

edited

Loading

mrodm Sep 17, 2024 •

edited

Loading

mrodm Sep 18, 2024 •

edited

Loading