Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add profile setting to enable serverless in compose #1766

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

jsoriano
Copy link
Member

Supersedes #1231.

@jsoriano jsoriano requested a review from ruflin April 12, 2024 14:10
@jsoriano jsoriano self-assigned this Apr 12, 2024
@ruflin
Copy link
Member

ruflin commented Apr 15, 2024

I think there are also some flags that have to be set for Kibana: https://github.com/elastic/kibana/blob/main/config/serverless.oblt.yml#L17

@ruflin
Copy link
Member

ruflin commented Apr 30, 2024

@jsoriano It would be great to get this moving forward to make testing on serverless much easier.

@jsoriano
Copy link
Member Author

@jsoriano It would be great to get this moving forward to make testing on serverless much easier.

Testing on actual serverless is already fully supported by using elastic-package stack up --provider serverless (given that the user has access to serverless).

I could not find any different behaviour in ES by setting es.serverless. How could we check that this is working as expected?

@ruflin
Copy link
Member

ruflin commented Apr 30, 2024

Testing on actual serverless is already fully supported by using elastic-package stack up --provider serverless (given that the user has access to serverless).

But we are talking here about local testing / during development / latest snapshots.

I could not find any different behaviour in ES by setting es.serverless. How could we check that this is working as expected?

@leehinman Maybe you can help with this one?

@jsoriano
Copy link
Member Author

jsoriano commented May 6, 2024

I think there are also some flags that have to be set for Kibana: https://github.com/elastic/kibana/blob/main/config/serverless.oblt.yml#L17

Added, but it looks like some flags are not available in the builds we use. Maybe we need to use different docker images.

@ruflin
Copy link
Member

ruflin commented May 7, 2024

@flash1293 Do you know by chance more about what configs must be used?

@flash1293
Copy link
Contributor

Unfortunately I don't know much about running serverless kibana/elasticsearch. The dev environment is using the docker.elastic.co/kibana-ci/elasticsearch-serverless:latest-verified image

@cmacknz
Copy link
Member

cmacknz commented May 8, 2024

When we were adding support to serverless for Beats we found that there was no local development environment that could easily be setup with Docker. That elasticsearch-serverless container has service dependencies expects things like a blob storage service to exist, it isn't (or wasn't) meant to run standalone.

All of the Beats serverless tests run against real Serverless projects provisioned by the agent test framework.

@flash1293
Copy link
Contributor

there was no local development environment that could easily be setup with Docker

It seems like the kibana dev setup is super similar - @afharo where can we learn more about how it works there?

@afharo
Copy link
Member

afharo commented May 8, 2024

HI! Thanks for the ping. I think the best folks to ask about how our yarn es serverless works are @elastic/kibana-operations, prob @delanni.

{{ $serverless_project_type := fact "serverless_project_type" }}
{{ if eq $serverless_enabled "true" }}
{{ if eq $serverless_project_type "observability" }}
# Observability Project config
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK, just setting serverless: oblt should be enough to load all other settings (if you use the Kibana Serverless docker image).

@Ikuni17
Copy link

Ikuni17 commented May 9, 2024

Most of how we run ES serverless for Kibana development is in this file: https://github.com/elastic/kibana/blob/main/packages/kbn-es/src/utils/docker.ts

The first few logs lines should be

Starting Serverless Elasticsearch...
Serverless default for [stateless.enabled] is overridden to [true]

We pass --env stateless.enabled=true (instead of serverless) so this may be difference, though I think the image starts ES in serverless mode anyways.

@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 28, 2024

💔 Build Failed

Failed CI Steps

History

cc @jsoriano

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants