Charts' device value files override component security contexts

[eero-t]

Several application values files change component securityContext, for example setting (container) root FS to be writable:

$ git grep -e '^[^#].*securityContext' -e ^[^#].*readOnlyRoot.*false | grep -v -e common/ -e templates
chatqna/gaudi-tgi-values.yaml:  securityContext:
chatqna/gaudi-tgi-values.yaml:    readOnlyRootFilesystem: false
chatqna/gaudi-vllm-values.yaml:  securityContext:
chatqna/gaudi-vllm-values.yaml:    readOnlyRootFilesystem: false
chatqna/guardrails-gaudi-values.yaml:  securityContext:
chatqna/guardrails-gaudi-values.yaml:    readOnlyRootFilesystem: false
searchqna/gaudi-values.yaml:  securityContext:
searchqna/gaudi-values.yaml:    readOnlyRootFilesystem: false

Those overrides are for tei & teirerank components, which seem to have same override for Gaudi, but I think that's wrong. Either component needs to write root FS or not, it should not depend on which device is used:

$ git grep ^[^#].*readOnlyRoot.*false | grep common/
common/agent/values.yaml:  readOnlyRootFilesystem: false
common/chathistory-usvc/values.yaml:  readOnlyRootFilesystem: false
common/data-prep/values.yaml:  readOnlyRootFilesystem: false
common/llm-uservice/values.yaml:  readOnlyRootFilesystem: false
common/lvm-serve/values.yaml:  readOnlyRootFilesystem: false
common/lvm-uservice/values.yaml:  readOnlyRootFilesystem: false
common/mm-embedding/values.yaml:  readOnlyRootFilesystem: false
common/prompt-usvc/values.yaml:  readOnlyRootFilesystem: false
common/retriever-usvc/values.yaml:  readOnlyRootFilesystem: false
common/speecht5/values.yaml:  readOnlyRootFilesystem: false
common/tei/gaudi-values.yaml:  readOnlyRootFilesystem: false        <====
common/teirerank/gaudi-values.yaml:  readOnlyRootFilesystem: false  <====
common/whisper/values.yaml:  readOnlyRootFilesystem: false

I'm pretty sure those overrides were made redundant with #613 and #642.

---

latest testing status for true readOnlyRootFileSystem settings of charts under common directory.

• agent: Verified support readOnlyRootFilesystem
• chathistory-usvc: verified support readOnlyRootFilesystem
• data-prep: PR Add readOnlyRootFileSystem for dataprep and retriever #867. depends on [CI Feature] functional gap and code duplications in dataprep CI test GenAIComps#1324, [Bug] can NOT ingest pptx files with embedded image to dataprep microservice GenAIComps#1325
• gpt-sovits: verified NOT support readOnlyRootFilesystem. nltk need to download data to local directory during runtime.
• llm-uservice: verified NOT support readOnlyRootFilesystem. docsum/faqgen needs to install langserve python module during runtime.
• lvm-serve: verified NOT support readOnlyRootFilesystem. llama-vision needs to write files to current local directory during runtime.
• lvm-uservice: verified support readOnlyRootFilesystem
• mm-embedding: verified NOT support readOnlyRootFilesystem. embedding-multimodal-bridgetower needs to create logfiles in the local directory during runtime. Found bug [Bug] service embedding-multimodal-bridgetower can not be launched GenAIComps#1370
• nginx: verified with minimal securityContext.
• prompt-usvc: verified support readOnlyRootFilesystem
• retriever-usvc: PR Add readOnlyRootFileSystem for dataprep and retriever #867. depends on dataprep.
• speecht5: verified NOT support readOnlyRootFilesystem. service itself needs to download files to local current directory.
• tei: verified NOT support readOnlyRootFilesystem in tei-gaudi case. Pod crashes for with no clue of where the tei-gaudi image is want to write into filesystem.
• teirerank: same as tei
• text2image: verified support readOnlyRootFilesystem
• whisper: verified NOT support readOnlyRootFilesystem. service needs to write to the local current directory during runtime.
• ui: depends on issue [UI] UI container should NOT run with root privilege GenAIExamples#517
• vllm: PR vllm: Add securityContxt settings #874

[poussa]

Also, the vllm dropped the entire security context via #564.

Now the init container carefully manages the security context but the main container runs as root and with all capabilities.

[eero-t]

UI and NGINX services are also missing user setting, i.e. running as root:

$ for i in $(ls common/*/values.yaml); do \
    if ! grep -q ^[^#]*sUser $i; then \
      echo "User missing: $i"; \
    fi;
  done
User missing: common/commonlib/values.yaml
User missing: common/nginx/values.yaml
User missing: common/ui/values.yaml
User missing: common/vllm/values.yaml

And this shows those same deployments also running with all the (default) capabilities):

$ for i in $(ls common/*/values.yaml); do \
    if ! grep -q ^[^#]*drop: $i; then \
      echo "Running with all capabilities: $i"; \
    fi; \
  done
<same results>

Larger number of containers is running with (default) root group:

$ for i in $(ls common/*/*values*.yaml); do \
    if grep -q ^[^#]*sGroup $i; then \
      grep -H ^[^#]*sGroup $i; \
    elif echo $i | grep -qF /values.yaml; then \
      echo "User group missing: $i"; \
    fi; \
  done | grep missing
....

EDIT: groups are not in k8s security, so last one can be ignored.

[eero-t]

In addition to disabling read-only FS setting, there are also some Helm charts that do not set it at all:

$ for i in $(ls common/*/*values*.yaml); do \
    if grep -q ^[^#]*readOnlyRootFilesystem $i; then \
      grep -H ^[^#]*readOnlyRootFilesystem $i; \
    elif echo $i | grep -qF /values.yaml; then \
      echo "No read-only FS setting: $i"; \
    fi; \
  done | grep -e ^No -e false | sort
common/agent/values.yaml:  readOnlyRootFilesystem: false
common/chathistory-usvc/values.yaml:  readOnlyRootFilesystem: false
common/data-prep/values.yaml:  readOnlyRootFilesystem: false
common/llm-uservice/values.yaml:  readOnlyRootFilesystem: false
common/lvm-serve/values.yaml:  readOnlyRootFilesystem: false
common/lvm-uservice/values.yaml:  readOnlyRootFilesystem: false
common/mm-embedding/values.yaml:  readOnlyRootFilesystem: false
common/prompt-usvc/values.yaml:  readOnlyRootFilesystem: false
common/retriever-usvc/values.yaml:  readOnlyRootFilesystem: false
common/speecht5/values.yaml:  readOnlyRootFilesystem: false
common/tei/gaudi-values.yaml:  readOnlyRootFilesystem: false
common/teirerank/gaudi-values.yaml:  readOnlyRootFilesystem: false
common/whisper/values.yaml:  readOnlyRootFilesystem: false
No read-only FS setting: common/commonlib/values.yaml
No read-only FS setting: common/gpt-sovits/values.yaml
No read-only FS setting: common/nginx/values.yaml
No read-only FS setting: common/text2image/values.yaml
No read-only FS setting: common/ui/values.yaml
No read-only FS setting: common/vllm/values.yaml

[lianhao]

Larger number of containers is running with (default) root group:

I don't think the k8s pod security standard ever mention any requirements for group settings, so we just run the container with the default group settings defined by the container image itself.

[lianhao]

In addition to disabling read-only FS setting, there are also some Helm charts that do not set it at all:

We need to double check if the new image can run with read-only FS now. It's set as false because when the helm chart was created, the container doesn't support running under read-only root FS by that time.

[eero-t]

I was not suggesting changing it blindly. This is more of a reminder to go through these items.

And yes, groups are not mentioned k8s security standards, so that part can be ignored: https://kubernetes.io/docs/concepts/security/pod-security-standards/

[lianhao]

Also, the vllm dropped the entire security context via #564.

Now the init container carefully manages the security context but the main container runs as root and with all capabilities.

The vllm security settings are disabled by this commit, the reason behind this was at that time, opea/vllm-hpu image doesn't support run as non-root, while opea/vllm supported. Also, since there is no way for the helm to unset/delete values for the child chart, so we're stuck with this no security settings situation.

[eero-t]

The vllm security settings are disabled by this commit, the reason behind this was at that time, opea/vllm-hpu image doesn't support run as non-root, while opea/vllm supported. Also, since there is no way for the helm to unset/delete values for the child chart, so we're stuck with this no security settings situation.

Then it should be changed to run as root (with comment indicating why), instead of dropping the whole security context. User 0 with the extra constraints is still fairly OK, without constraints, much less so.