Implement SecretManager SPI for K8s/Vault integration

[treblereel]

Context

Currently, environment variable resolution in YAML configuration uses a simple ${VAR} placeholder mechanism that resolves at deserialization time via System.getProperty() and System.getenv().

Current implementation:

• EnvironmentPlaceholderModule (Jackson module)
• EnvironmentPlaceholderResolver (resolves ${VAR} → System property → env var)
• Works for: ${OPENAI_API_KEY}, ${OLLAMA_BASE_URL}, etc.
• Location: schema/src/main/java/io/casehub/model/util/

Problem

In microservices/Kubernetes environments, secrets are typically stored in:

• Kubernetes Secrets
• HashiCorp Vault
• AWS Secrets Manager
• Azure Key Vault
• Google Secret Manager

The current approach cannot integrate with these secret stores.

Security concerns:

• Environment variables are visible in process listings
• System properties can leak in logs and stack traces
• No rotation mechanism for credentials
• No audit trail for secret access

Proposed Solution

Implement a SecretManager SPI pattern similar to Serverless Workflow.

1. Define SPI Interface

package io.casehub.api.spi;

/**
 * SPI for resolving secrets from various backends (K8s, Vault, etc.)
 */
@FunctionalInterface
public interface SecretManager {
  /**
   * Resolve a secret by name.
   * 
   * @param secretName the name of the secret (e.g., "openai", "database")
   * @return map of secret properties (e.g., {apiKey: "sk-...", orgId: "..."})
   * @throws SecretNotFoundException if secret does not exist
   */
  Map<String, Object> secret(String secretName);
}

2. Multiple Implementations

• SystemPropertySecretManager (default, backward compatible)
• KubernetesSecretManager (reads from K8s Secrets API)
• VaultSecretManager (integrates with HashiCorp Vault)
• CompositeSecretManager (chain of responsibility)

3. YAML Declaration

dsl: "0.1"
namespace: example
name: sentiment-analysis

use:
  secrets:
    - openai      # Validates secret exists at load time
    - anthropic   # Fails fast if secret not available

spec:
  workers:
    - name: sentiment-analyzer
      agent:
        model:
          openai:
            apiKey: "$secret.openai.apiKey"
            orgId: "$secret.openai.organizationId"

Backward compatibility:

# Old syntax still works:
apiKey: "${OPENAI_API_KEY}"

# New syntax for secret stores:
apiKey: "$secret.openai.apiKey"

4. Configuration

application.properties:

# Secret provider: system | kubernetes | vault | composite
casehub.secrets.provider=kubernetes

# Kubernetes-specific
casehub.secrets.kubernetes.namespace=casehub-prod

# Vault-specific
casehub.secrets.vault.address=https://vault.example.com
casehub.secrets.vault.token=${VAULT_TOKEN}

Reference Implementation

Serverless Workflow has a mature implementation:

• io.serverlessworkflow.impl.config.SecretManager
• io.serverlessworkflow.impl.config.ConfigSecretManager
• Version: 7.13.4.Final

Implementation Plan

Phase 1: SPI Foundation

• Define SecretManager interface in api/src/main/java/io/casehub/api/spi/
• Implement SystemPropertySecretManager (default, current behavior)
• Update EnvironmentPlaceholderResolver to support $secret.* syntax
• Add secret validation at YAML load time
• Tests for SPI contract

Phase 2: Kubernetes Integration

• New module: casehub-secrets-kubernetes
• Implement KubernetesSecretManager
• Integration tests with k3s/kind
• Documentation for K8s deployment

Phase 3: Vault Integration (Optional)

• New module: casehub-secrets-vault
• Implement VaultSecretManager
• Support for AppRole auth
• Integration tests with Vault dev server

Phase 4: Advanced Features

• CompositeSecretManager for fallback chains
• Secret caching with configurable TTL
• Metrics: secret access count, cache hit/miss
• Audit logging for secret access

Acceptance Criteria

• SecretManager SPI defined and documented
• Backward compatibility: ${VAR} syntax still works
• New syntax: $secret.{name}.{property} supported
• Secrets declared in use.secrets validated at load time
• At least 2 implementations: System, Kubernetes
• Tests for all implementations
• Documentation:
  • How to implement custom secret providers
  • How to configure each provider
  • Migration guide from ${VAR} to $secret.*

Security Considerations

1. Secret Exposure

   • Never log resolved secret values
   • Mask secrets in toString() and error messages

2. Access Control

   • Secret provider should enforce RBAC (K8s RBAC, Vault policies)
   • Validate secret names against allowlist (prevent path traversal)

3. Rotation

   • Support TTL-based cache invalidation
   • Allow runtime secret refresh without restart

4. Audit

   • Log which secrets are accessed (not their values)
   • Integrate with external audit systems

Related

• Current implementation: EnvironmentPlaceholderModule (PR Add AI Agent support as Worker execution type #244)
• Target deployment: Kubernetes, microservices architecture
• Detailed design: docs/secret-manager-spi.md