genMetadata augment PoC (local llama.cpp)

.gitignore

@@ -31,3 +31,6 @@
 
 # OpenNLP models (downloaded by tools/fetch-opennlp-models.sh)
 /java/gateway-core/src/main/resources/opennlp/
+
+# genMetadata model cache/archives (tools/fetch-gen-metadata-model.sh)
+.build/

docs/configuration/remote-resources.md

@@ -28,53 +28,68 @@ mounted locally.
 
 ## Terraform Configuration
 
-By default, the host modules in this repository (`aws-host` and `gcp-host`) will configure the
-`REMOTE_RESOURCE_BUCKET` for you if you set the `enable_remote_resources` variable to `true`. This
-automatically wires the **artifacts bucket** (used for deployment bundles) as the remote resource bucket.
+Host modules (`aws-host` and `gcp-host`) enable remote resource loading **per connector** via `enable_remote_resources` on each `api_connectors`, `bulk_connectors`, or `webhook_collectors` entry. This wires the **artifacts bucket** (used for deployment bundles) as the remote resource bucket for only those instances.
 
 > [!IMPORTANT]
 > - If you configure an existing bucket (e.g., by providing `artifacts_bucket_name`), the bucket must already exist.
 > - The Terraform runner (the credentials running the `terraform` command) must have sufficient IAM permissions on that bucket to apply permissions (since it will grant read access to the proxy's service account or Lambda execution role).
 
+### Connector specs and custom connectors
+
+Prebuilt connectors in `worklytics-connector-specs` may set `enable_remote_resources` or `enable_gen_metadata` per connector (e.g. `msft-copilot` enables gen metadata). Those flags flow through to `aws-host` / `gcp-host` when the connector is enabled.
+
+For ad-hoc connectors, set the flags on `custom_api_connectors` or `custom_bulk_connectors` in your root module (see `infra/examples-dev/*/variables.tf`).
+
 ### AWS (`aws-host`)
 
 ```hcl
 module "psoxy" {
   source = "../../modules/aws-host"
 
-  # ... existing configuration ...
-
-  # Enable remote resource loading from the artifacts S3 bucket
-  enable_remote_resources = true
+  api_connectors = {
+    "my-api" = {
+      source_kind             = "..."
+      source_auth_strategy    = "..."
+      target_host             = "..."
+      enable_remote_resources = true  # OpenNLP, rules.yaml in bucket, etc.
+      enable_gen_metadata     = false
+    }
+  }
 }
 ```
 
-This will:
-- Set `REMOTE_RESOURCE_BUCKET` on every Lambda function to the artifacts bucket name
-- Grant `s3:GetObject` permission on the configured path prefixes in the bucket to each Lambda's execution role
+This will, for that connector only:
+- Set `REMOTE_RESOURCE_BUCKET` on the Lambda to the artifacts bucket name
+- Grant `s3:GetObject` permission on the configured path prefixes in the bucket to that Lambda's execution role
 
 ### GCP (`gcp-host`)
 
 ```hcl
 module "psoxy" {
   source = "../../modules/gcp-host"
 
-  # ... existing configuration ...
-
-  # Enable remote resource loading from the artifacts GCS bucket
-  enable_remote_resources = true
+  api_connectors = {
+    "gcal" = {
+      source_kind             = "..."
+      source_auth_strategy    = "..."
+      target_host             = "..."
+      enable_remote_resources = true
+    }
+  }
 }
 ```
 
-This will:
-- Set `REMOTE_RESOURCE_BUCKET` on every Cloud Function to the artifacts bucket name
-- Grant `roles/storage.objectViewer` on the bucket, scoped to the configured path prefixes using IAM conditions, to each function's service account
+This will, for that connector only:
+- Set `REMOTE_RESOURCE_BUCKET` on the Cloud Function to the artifacts bucket name
+- Grant `roles/storage.objectViewer` on the bucket, scoped to the configured path prefixes using IAM conditions, to that function's service account
+
+Remote resource paths use `/` as a hierarchy separator within the bucket (e.g. `psoxy-dev-erik/GCAL/rules.yaml` for shared prefix `psoxy-dev-erik/` and connector `gcal`). They are distinct from secret / parameter prefixes, which use a trailing `_` to separate names (e.g. `psoxy-dev-erik_GCAL_SOURCE`). When `INSTANCE_RESOURCE_PATH` / `SHARED_RESOURCE_PATH` are not set, psoxy falls back to the config paths and normalizes trailing `_` to `/` and strips any leading `/`.
 
 ## Environment Variables
 
 | Variable                  | Description                                                                                        | Required |
 |---------------------------|----------------------------------------------------------------------------------------------------|----------|
-| `REMOTE_RESOURCE_BUCKET`  | Name of the S3/GCS bucket containing remote resources.                                             | No       |
+| `REMOTE_RESOURCE_BUCKET`  | Name of the S3/GCS bucket from which to load remote resources.                                     | Yes      |
 | `INSTANCE_RESOURCE_PATH`  | Path prefix for instance-specific resources within the bucket. Defaults to `PATH_TO_INSTANCE_CONFIG`. | No       |
 | `SHARED_RESOURCE_PATH`    | Path prefix for shared resources (NLP models, etc.) within the bucket. Defaults to `PATH_TO_SHARED_CONFIG`. | No       |
 
@@ -83,12 +98,12 @@ This will:
 The Terraform modules automatically grant minimal read permissions following the Principle of
 Least Privilege. Access is limited to the configured resource path prefixes within the bucket:
 
-- **AWS**: `s3:GetObject` only for objects under `{INSTANCE_RESOURCE_PATH}/` and `{SHARED_RESOURCE_PATH}/`
-- **GCP**: object read access only for objects under `{INSTANCE_RESOURCE_PATH}/` and `{SHARED_RESOURCE_PATH}/`, enforced with IAM Conditions
-
-No write, delete, or list permissions are granted. When an object is missing or inaccessible, S3 may return 403 (citing `s3:ListBucket`) rather than 404 if the caller lacks bucket list permission; psoxy treats that as unavailable (non-fatal) and continues its resource lookup chain (e.g. falling back to prebuilt rules).
+- **AWS**: `s3:GetObject` only for objects under `{INSTANCE_RESOURCE_PATH}/` and
+  `{SHARED_RESOURCE_PATH}/`
+- **GCP**: object read access only for objects under `{INSTANCE_RESOURCE_PATH}/` and
+  `{SHARED_RESOURCE_PATH}/`, enforced with IAM Conditions
 
-Remote resource paths use `/` as a hierarchy separator within the bucket (e.g. `psoxy-dev-erik/GCAL/rules.yaml` for shared prefix `psoxy-dev-erik/` and connector `gcal`). They are distinct from secret / parameter prefixes, which use a trailing `_` to separate names (e.g. `psoxy-dev-erik_GCAL_SOURCE`). When `INSTANCE_RESOURCE_PATH` / `SHARED_RESOURCE_PATH` are not set, psoxy falls back to the config paths and normalizes trailing `_` to `/` and strips any leading `/`.
+No write, delete, or list permissions are granted.
 
 ## Use Cases
 
@@ -98,8 +113,7 @@ if no `RULES` config property (env var, parameter store entry, etc.) is found.
 
 ### NLP Models (alpha)
 OpenNLP model files (`en-sent.bin`, `en-pos-maxent.bin`, `en-chunker.bin`) are **not** bundled in
-deployment JARs. If your connector rules use `sentenceMetadata` augments, you must upload these
-models to the remote resources bucket yourself (requires `enable_remote_resources = true`).
+deployment JARs. If your connector rules use `sentenceMetadata` augments, set `enable_remote_resources = true` on that API connector and upload these models to the remote resources bucket.
 
 Place them under `{SHARED_RESOURCE_PATH}/opennlp/` (e.g.
 `{SHARED_RESOURCE_PATH}/opennlp/en-sent.bin`). `{SHARED_RESOURCE_PATH}` defaults to
@@ -130,9 +144,32 @@ aws s3 cp en-sent.bin s3://{REMOTE_RESOURCE_BUCKET}/{SHARED_RESOURCE_PATH}/openn
 gsutil cp en-sent.bin gs://{REMOTE_RESOURCE_BUCKET}/{SHARED_RESOURCE_PATH}/opennlp/en-sent.bin
 ```
 
-### LLM Weights (future)
-Smaller language models that fit in memory can be placed in the shared resource path for
-on-the-fly inference within the proxy.
+### LLM model archives (genMetadata BETA)
+Upload a **zip** of a Jlama SafeTensors model directory (must include `config.json`) for the **genMetadata** augment to `{SHARED_RESOURCE_PATH}/llm/` in the bucket. The archive name is derived from `PSOXY_GEN_MODEL` (default `tjake/Llama-3.2-1B-Instruct-JQ4`): slashes become `__`, with a `.zip` suffix — e.g. `llm/tjake__Llama-3.2-1B-Instruct-JQ4.zip`. Set per-connector `enable_gen_metadata = true` on `api_connectors` (which also enables remote resource loading for that connector). See [gen-metadata-augment.md](../development/gen-metadata-augment.md) for HuggingFace ids, cloud backends, and ops detail (cloud inference does not use `llm/` archives).
+
+**Helper script** (download from HuggingFace, zip, upload):
+
+```bash
+# AWS — PREFIX is your SHARED_RESOURCE_PATH within the bucket (trailing slash optional)
+./tools/fetch-gen-metadata-model.sh s3://REMOTE_RESOURCE_BUCKET/PREFIX/
+
+# GCP — optional MODEL_ID overrides PSOXY_GEN_MODEL / default
+./tools/fetch-gen-metadata-model.sh gs://REMOTE_RESOURCE_BUCKET/PREFIX/ tjake/Llama-3.2-1B-Instruct-JQ4
+
+# Use an existing local SafeTensors directory instead of downloading
+./tools/fetch-gen-metadata-model.sh --from-dir /path/to/model-dir s3://REMOTE_RESOURCE_BUCKET/PREFIX/
+```
+
+**Manual upload:**
+
+```bash
+cd /path/to/model-dir && zip -r ../tjake__Llama-3.2-1B-Instruct-JQ4.zip .
+aws s3 cp ../tjake__Llama-3.2-1B-Instruct-JQ4.zip \
+  s3://{REMOTE_RESOURCE_BUCKET}/{SHARED_RESOURCE_PATH}/llm/tjake__Llama-3.2-1B-Instruct-JQ4.zip
+
+gsutil cp ../tjake__Llama-3.2-1B-Instruct-JQ4.zip \
+  gs://{REMOTE_RESOURCE_BUCKET}/{SHARED_RESOURCE_PATH}/llm/tjake__Llama-3.2-1B-Instruct-JQ4.zip
+```
 
 ## Uploading Resources
 

docs/development/augments.md

@@ -40,14 +40,16 @@ Inspired by OData's `@`-annotation pattern (where metadata about a property `Foo
 +{sourceProperty}:{augmentFunction}
 ```
 
-When `innerJsonPath` is set and matches nested JSON inside a string field, each inner match is
-augmented in place within the parsed embedded JSON (mirroring the legacy `textDigest` transform
-with `isJsonEscaped`), then the modified structure is re-serialized as a string:
+When `innerJsonPath` is set and matches nested JSON inside a string field, all inner matches are
+grouped under a single augment property, keyed by normalized inner path suffix:
 
 ```jsonc
 {
   "content": "{ ... escaped AdaptiveCard JSON ... }",
-  "+content:textDigest": "{\"type\":\"AdaptiveCard\",\"version\":\"1.0\",\"body\":[{\"type\":\"TextBlock\",\"text\":\"{\\\"length\\\":2572,\\\"word_count\\\":154}\",\"wrap\":true},...]}"
+  "+content:textDigest": {
+    "body[0].text": { "length": 2572, "word_count": 154 },
+    "body[1].text": { "length": 980,  "word_count": 154 }
+  }
 }
 ```
 
@@ -58,8 +60,8 @@ with `isJsonEscaped`), then the modified structure is re-serialized as a string:
 | `:` | Separator. |
 | `{augmentFunction}` | The name of the augment function (e.g. `textDigest`). |
 
-For embedded JSON, inner matched fields (e.g. `text`) are replaced with serialized augment output
-and the whole embedded JSON is re-serialized as a string value.
+Inner map keys use `{innerPathSuffix}` — the normalized concrete path within parsed inner JSON
+(e.g. `body[0].text`).
 
 **Example.** For a response containing `body.content`, the augment property produced by the `textDigest` function would be:
 
@@ -193,7 +195,7 @@ public abstract class Augment {
 
 ### `outputSchema` — Output Validation
 
-Each augment rule carries an optional `outputSchema` property of type `JsonSchemaFilter`. This schema is applied as a **predicate** (not a filter) to the value produced by the augment's `compute()` method:
+Each augment rule may carry an `outputSchema` property of type `JsonSchemaFilter`. This schema is applied as a **predicate** (not a filter) to the value produced by the augment's `compute()` method. **`genMetadata` requires `outputSchema`.** Validation is implemented in `AugmentProcessor` (v0.6.x BETA); failures emit `X-Psoxy-Warning: augment-output-schema-mismatch`.
 
 | Outcome | Action |
 |---|---|

docs/development/gen-metadata-augment.md

@@ -0,0 +1,126 @@
+# genMetadata Augment (BETA)
+
+> **Status:** BETA · PoC on branch `s225-gen-metadata-poc`
+> **Since:** v0.6.x
+> **Relates to:** [augments.md](augments.md), [sentence-metadata-augment.md](sentence-metadata-augment.md), [remote-resources.md](../configuration/remote-resources.md)
+
+## Overview
+
+The **genMetadata** augment generates structured JSON metadata alongside a source field using a pluggable generative backend. All backends share a [LangChain4j](https://github.com/langchain4j/langchain4j) `ChatModel` integration inside `psoxy-core`. **Local** inference uses embedded [Jlama](https://github.com/tjake/Jlama) (pure Java). **Bedrock** and **Vertex** cloud backends are planned.
+
+Output appears as a sibling property: `+{sourceProperty}:genMetadata`.
+
+## Rule configuration (required)
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `jsonPaths` | yes | Source values to process |
+| `prompt` | yes | Task instruction (included in rules SHA) |
+| `outputSchema` | yes | JSON Schema predicate; invalid output is suppressed |
+
+No `model`, `backend`, `quality`, or `maxTokens` in rules. Generation limits and model selection are deployment settings (see env vars below). The `outputSchema` gate ensures only structured, expected fields reach the response; it does not cap generation length — use `PSOXY_GEN_MAX_TOKENS` for that.
+
+## Deployment configuration (env)
+
+Read via `ConfigService` / `ProxyConfigProperty`:
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `PSOXY_GEN_BACKEND` | `local` | `local` (Jlama), `bedrock` and `vertex` (future) |
+| `PSOXY_GEN_MODEL` | `tjake/Llama-3.2-1B-Instruct-JQ4` | Jlama HuggingFace id (`owner/name`) or logical id for a cached local archive |
+| `PSOXY_GEN_TIMEOUT_SECONDS` | `15` | Max seconds per genMetadata `chat()` call (and model-load wait) |
+| `PSOXY_GEN_MAX_INPUT_CHARS` | `4096` | Truncate source text before prompting |
+| `PSOXY_GEN_MAX_TOKENS` | `256` | Max tokens to generate per inference |
+| `PSOXY_GEN_META_RETRIES` | `2` | Total inference attempts per augment when output is unparseable or fails `outputSchema` (minimum 1; `2` = one retry) |
+| `ENABLE_GEN_METADATA` | unset | Set to `true` when that connector has Terraform `enable_gen_metadata = true` |
+
+When `enable_gen_metadata` is set on an API connector, Terraform **appends** Jlama JVM flags to any existing `JAVA_TOOL_OPTIONS` from `general_environment_variables` or that connector's `environment_variables` (`--add-modules=jdk.incubator.vector --enable-preview --enable-native-access=ALL-UNNAMED`).
+
+## Infrastructure
+
+Set **`enable_gen_metadata = true`** on individual `api_connectors` entries (API connectors only). The `msft-copilot` entry in `worklytics-connector-specs` sets this by default when that connector is enabled; custom API connectors can set it in `custom_api_connectors`:
+
+- Sets `ENABLE_GEN_METADATA=true` on that function
+- Sets `JAVA_TOOL_OPTIONS` for Jlama on that function's JVM
+- Floors memory at **4096 MB** on that connector unless a higher `memory_size_mb` / `available_memory_mb` is set
+- Enables remote resource loading for that connector (for `llm/*.zip` model archives)
+
+You do not need a separate runtime failure if memory is too low — enable the flag on connectors whose rules use `genMetadata`. Without the flag, augments still run but return `augment-gen-unavailable` if the model cannot load.
+
+Manual setup (without Terraform flags): set env vars yourself, `memory_size_mb = 4096`, enable remote bucket access, and upload a model archive (see below).
+
+## Java / LangChain4j
+
+BETA uses **LangChain4j** with the **Jlama** provider for local embedded inference (`langchain4j` **1.15.0**, `langchain4j-jlama` **1.15.0-beta25**, `jlama-native` in `psoxy-core`). Maven Central does not publish a non-beta `langchain4j-jlama` artifact for the 1.x `ChatModel` API (older **0.36.2** uses a different API). The beta suffix is LangChain4j’s release channel for this integration module, not a separate fork. This avoids JNI bindings to llama.cpp that can take down the JVM on native faults. The same `ChatModel` abstraction will back **Bedrock** (`langchain4j-bedrock`, AWS bundle only) and **Vertex** (`langchain4j-vertex-ai-gemini`, GCP bundle only) when implemented.
+
+Jlama loads models in **SafeTensors** layout (HuggingFace-style directory with `config.json`), not single-file GGUF.
+
+### Local model deployment
+
+1. **HuggingFace id (dev / first run):** set `PSOXY_GEN_MODEL` to a Jlama-compatible id such as `tjake/Llama-3.2-1B-Instruct-JQ4`. Jlama may download weights on first use (requires outbound network from the function).
+2. **Production (recommended):** zip a SafeTensors model directory and upload to `{SHARED_RESOURCE_PATH}/llm/{cache-dir-name}.zip`, where `{cache-dir-name}` is `PSOXY_GEN_MODEL` with `/` replaced by `__` (e.g. `tjake__Llama-3.2-1B-Instruct-JQ4.zip` for `tjake/Llama-3.2-1B-Instruct-JQ4`). The runtime extracts the archive into a temp cache before loading.
+
+**Example `PSOXY_GEN_MODEL` values** (Jlama / HuggingFace; prefer small instruct models on the 4096 MB floor):
+
+| `PSOXY_GEN_MODEL` | Notes |
+|-------------------|-------|
+| `tjake/Llama-3.2-1B-Instruct-JQ4` | Default; pre-quantized Jlama build |
+| `tjake/Llama-3.2-3B-Instruct-JQ4` | Higher quality; verify memory |
+| `tjake/Qwen2.5-1.5B-Instruct-JQ4` | Strong small instruct |
+| `tjake/Phi-3.5-mini-instruct-JQ4` | Compact |
+| `tjake/gemma-2-2b-it-JQ4` | Google small instruct |
+
+See [tjake on Hugging Face](https://huggingface.co/tjake) for other pre-quantized `-JQ4` builds. Logical ids without `/` (e.g. `llama-3.2-1b-instruct`) work when the matching `llm/llama-3.2-1b-instruct.zip` archive is present.
+
+## MS Copilot PoC: prompt classification
+
+Classifies `body.content` into one of 11 categories (`category` only). See commented example in `docs/sources/microsoft-365/msft-copilot/msft-copilot.yaml` and `MS_COPILOT_GEN_METADATA_AUGMENT` in `PrebuiltSanitizerRules.java`.
+
+## Local integration tests
+
+Opt-in Jlama tests live in `LangChain4jGenMetadataBackendIntegrationTest` (token-budget regressions from live Copilot PoC). Run from the `java/` directory:
+
+```bash
+cd java
+mvn test -pl core -am -Pgen-metadata-integration
+```
+
+Requires a Jlama model under `~/.jlama` (downloaded on first run) or set `PSOXY_GEN_MODEL_CACHE`. Optional: `PSOXY_GEN_MODEL` to override the default HuggingFace id.
+
+## Error handling
+
+Non-fatal: failures throw `AugmentProcessingException` (caught in `AugmentProcessor`) and surface as `X-Psoxy-Warning` headers on the response.
+
+| Code | Meaning |
+|------|---------|
+| `augment-gen-unavailable` | Model missing / not loaded / unsupported backend |
+| `augment-gen-inference-failed` | Inference or JSON parse failed |
+| `augment-output-schema-mismatch` | Output failed `outputSchema` predicate |
+| `augment-conflict-skipped` | Upstream `+` properties present |
+
+## Roadmap
+
+### Cloud backends (`PSOXY_GEN_BACKEND=bedrock` | `vertex`) — planned
+
+Implement via LangChain4j in `psoxy-core`, behind the existing `GenMetadataBackend` SPI:
+
+| Backend | Module | Bundle | Credentials |
+|---------|--------|--------|-------------|
+| `bedrock` | `langchain4j-bedrock` | **AWS** Lambda / `psoxy-aws` only | Lambda execution role |
+| `vertex` | `langchain4j-vertex-ai-gemini` | **GCP** Cloud Functions / `psoxy-gcp` only | Function service account |
+
+`PSOXY_GEN_MODEL` becomes the cloud model id. Use native JSON schema (`ResponseFormat`) where supported; no `llm/` remote weights. Wrong backend on the wrong platform should fail clearly. IAM for Bedrock / Vertex AI will ship with per-connector `enable_gen_metadata` extensions.
+
+### Alternative local engine: java-llama.cpp (`de.kherud`) — conditional
+
+[java-llama.cpp](https://github.com/kherud/java-llama.cpp) (GGUF via JNI) is **not** in use. Revisit only if **both** are true:
+
+1. Embedded local models prove valuable enough vs cloud inference, and
+2. Jlama performance is a significant bottleneck for production workloads.
+
+Rationale for deferring: JNI/native faults in llama.cpp can crash the JVM; LangChain4j + Jlama keeps inference in managed Java with a single `ChatModel` surface for local and cloud.
+
+### Deferred
+
+- **Rule-level `backend`, `model`, or `quality`** — env-only for BETA.
+- **ONNX Runtime GenAI, DJL, Ollama sidecars** — out of scope for in-process serverless.