Skip to content

whisper: support air gapped environment #1116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
poussa merged 1 commit into from
Jul 4, 2025
Merged

whisper: support air gapped environment #1116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion helm-charts/common/speecht5/templates/configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ data:
HF_ENDPOINT: {{ .Values.global.HF_ENDPOINT | quote}}
{{- end }}
HUGGINGFACE_HUB_CACHE: "/data"
HF_TOKEN: {{ .Values.global.HUGGINGFACEHUB_API_TOKEN | quote}}
HF_TOKEN: {{ .Values.global.HUGGINGFACEHUB_API_TOKEN | default .Values.global.HF_TOKEN | quote }}
{{- if .Values.global.offline }}
HF_HUB_OFFLINE: "1"
{{- end }}
43 changes: 39 additions & 4 deletions helm-charts/common/whisper/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,37 @@ export ASR_MODEL_PATH="openai/whisper-small"
helm install whisper whisper --set global.modelUseHostPath=${MODELDIR} --set ASR_MODEL_PATH=${ASR_MODEL_PATH}
```

### Install the microservice in air gapped (offline) mode

To run `whisper` microservice in an air gapped environment, users are required to pre-download the model `openai/whisper-small` to a shared storage.

#### Use node-local directory

Assuming the model data is shared using node-local directory `/mnt/opea-models`.

```
# On every K8s node, run the following command:
export MODEL_DIR=/mnt/opea-models
# Download model, assumes Python huggingface_hub[cli] module is already installed
huggingface-cli download --cache-dir "${MODEL_DIR}" openai/whisper-small

# On K8s master node, run the following command:
# Install using Helm with the following additional parameters:
helm install ... --set global.offline=true,global.modelUseHostPath=${MODEL_DIR}
```

#### Use persistent volume

Assuming we share the offline data on cluster level using a persistent volume (PV), first we need to create the persistent volume claim (PVC) with name `opea-model-pvc` to store the model data.

```
# Download model openai/whisper-small at the root directory of the corresponding PV
# ...
# Install using Helm with the following additional parameters:
# export MODEL_PVC=opea-model-pvc
# helm install ... --set global.offline=true,global.modelUsePVC=${MODEL_PVC}
```

## Verify

Use port-forward to access it from localhost.
Expand All @@ -26,7 +57,11 @@ curl http://localhost:1234/v1/asr \

## Values

| Key | Type | Default | Description |
| ---------------- | ------ | ---------------- | ----------- |
| image.repository | string | `"opea/whisper"` | |
| service.port | string | `"7066"` | |
| Key | Type | Default | Description |
| ----------------------- | ------ | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| image.repository | string | `"opea/whisper"` | |
| service.port | string | `"7066"` | |
| global.offline | bool | `false` | Whether to run the microservice in air gapped environment |
| global.modelUseHostPath | String | `""` | Cached models directory on Kubernetes node, service will not download if the model is cached here. The host path "modelUseHostPath" will be mounted to the container as /data directory. Setting this to null/empty will force the pod to download the model every time during startup. May not be set if `global.modelUsePVC` is also set. |
| global.modelUsePVC | String | `""` | Name of Persistent Volume Claim to use for model cache. The Persistent Volume will be mounted to the container as /data directory. Setting this to null/empty will force the pod to download the model every time during startup. May not be set if `global.modelUseHostPath` is also set. |
| global.HF_TOKEN | string | `insert-your-huggingface-token-here` | Hugging Face API token |
3 changes: 3 additions & 0 deletions helm-charts/common/whisper/templates/configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,6 @@ data:
{{- end }}
HUGGINGFACE_HUB_CACHE: "/data"
HF_TOKEN: {{ .Values.global.HUGGINGFACEHUB_API_TOKEN | default .Values.global.HF_TOKEN | quote }}
{{- if .Values.global.offline }}
HF_HUB_OFFLINE: "1"
{{- end }}
6 changes: 4 additions & 2 deletions helm-charts/common/whisper/templates/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ spec:
serviceAccountName: {{ include "whisper.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
{{- if not (hasPrefix "/data/" .Values.ASR_MODEL_PATH) }}
{{- if and (not .Values.global.offline) (not (hasPrefix "/data/" .Values.ASR_MODEL_PATH)) }}
initContainers:
- name: model-downloader
envFrom:
Expand Down Expand Up @@ -116,8 +116,10 @@ spec:
hostPath:
path: {{ .Values.global.modelUseHostPath }}
type: Directory
{{- else }}
{{- else if not .Values.global.offline }}
emptyDir: {}
{{- else }}
{{- fail "'global.modelUsePVC' or 'global.modelUseHostPath' must be set in offline mode" }}
{{- end }}
- name: tmp
emptyDir: {}
Expand Down
3 changes: 3 additions & 0 deletions helm-charts/common/whisper/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,9 @@ global:
# If set, and serviceAccount.create is false, it will assume this service account is already created by others.
sharedSAName: ""

# Running microservice in air gapped (offline) mode
# If offline is enabled, user must set either modelUseHostPath or modelUsePVC and download models `openai/whisper-small`.
offline: false
# Choose where to save your downloaded models
# Set modelUseHostPath for local directory, this is good for one node test. Example:
# modelUseHostPath: /mnt/opea-models
Expand Down