[Integration]: TealTiger governance for local LLM calls via Ollama

[nagasatish007]

Summary

Ollama runs LLMs locally. When developers use Ollama for agent workflows, they lose the governance layer that cloud API providers offer (rate limits, content filtering). TealTiger can fill that gap — providing PII detection, secret scanning, cost estimation, and policy enforcement for local LLM calls.

Why this integration?

Ollama's OpenAI-compatible API means TealTiger already works via TealOpenAI pointed at http://localhost:11434. But there are Ollama-specific concerns:

• No built-in content filtering — Ollama models respond to anything
• No cost tracking — local models have no usage API, but compute cost still exists
• No audit trail — no provider-side logging of what was asked/answered
• Tool use without governance — Ollama supports function calling, but with no access control

Proposed approach

1. TealOllama client wrapper — wraps Ollama's API with governance (builds on OpenAI-compat)
2. Compute cost estimation — estimate local inference cost based on model size, tokens, and hardware
3. Full guardrails — PII, secrets, prompt injection for local models that have no built-in safety
4. Audit trail — structured TEEC receipts for every local model interaction

Example usage

from tealtiger.integrations.ollama import TealOllama

client = TealOllama(
    host="http://localhost:11434",
    guardrails={
        "pii_detection": True,
        "secret_detection": True,
        "prompt_injection": True,
    },
    budget={
        "max_requests_per_session": 100,  # Rate limit for local models
    },
)

response = client.chat(
    model="llama3.1",
    messages=[{"role": "user", "content": "Summarize user data"}],
)
# Governance evaluated before and after the local model call

Deliverables

• tealtiger.integrations.ollama module
• TealOllama client wrapping Ollama's OpenAI-compatible API
• Local compute cost estimation (tokens × model-size factor)
• Full guardrail suite for unfiltered local models
• Example: governed local agent with Ollama + TealTiger
• Blog post: "Why local LLMs need governance too"

Links

• Ollama repo: https://github.com/ollama/ollama
• Ollama API docs: https://github.com/ollama/ollama/blob/main/docs/api.md
• Ollama OpenAI compatibility: https://github.com/ollama/ollama/blob/main/docs/openai.md

[CleanDev-Fix]

I'd like to work on this one if it's still open to outside contributors.

I’d keep the first PR narrow: add a TealOllama wrapper around the OpenAI-compatible endpoint, wire it through the existing governance path, and include a local/no-network test or example that proves the policy evaluation happens around an Ollama-style chat call. I’d leave hardware-based compute-cost estimation and the blog post as follow-up unless you want them in the first pass.

If that split sounds right, please assign me and I’ll take it from there.

[nagasatish007]

Thanks @richardchen874-sys — good question. The main challenge is all three, but in priority order:

1. Policy enforcement — local models have zero built-in content filtering, so PII detection, prompt injection blocking, and secret scanning are critical since nothing else catches them
2. Usage visibility — no provider-side logging means the governance layer becomes the single source of truth for what was asked/answered
3. Cost estimation — this is the least urgent but still valuable for teams running local models on GPU instances where compute time = real cost

Your point about normalizing to a provider-agnostic event is exactly what TealTiger already does — every call (cloud or local) produces a typed Decision object with the same shape regardless of backend. The TealOllama wrapper would feed into the same pipeline as TealOpenAI, TealAnthropic, etc.

Re: the private-beta credit for cloud-compatible comparison — appreciate the offer. For this issue specifically we're focused on the local governance gap, but if you'd like to open a separate issue about multi-model API comparison testing, that could be interesting for the broader cost-estimation story.

[nagasatish007]

@CleanDev-Fix great — that split is exactly right. A narrow first PR is the way to go.

Scope for first PR:

• TealOllama wrapper around Ollama's OpenAI-compatible endpoint
• Wire through existing governance pipeline (TealEngine evaluate → Decision → audit record)
• Local test that proves policy evaluation happens around a chat call (mock the Ollama endpoint, no network required)
• Basic example script showing governed local chat

Follow-up PRs:

• Hardware-based compute-cost estimation
• Tool-call governance for Ollama function calling
• Blog post

I'll assign you. A few pointers to get started:

• Look at packages/tealtiger-sdk/src/providers/openai.ts for the pattern — TealOllama should follow the same structure but default to http://localhost:11434/v1
• The guardrails pipeline is provider-agnostic, so you get PII/secrets/injection for free once the wrapper feeds into TealEngine.evaluate()
• Tests live in __tests__/ alongside the source — use vitest

Let me know if you have questions on the architecture.

[CleanDev-Fix]

Implemented the approved first slice in the TypeScript SDK submodule: agentguard-ai/tealtiger-typescript-prod#30. It adds a governed TealOllama client for Ollama's OpenAI-compatible local endpoint, with TealEngine/audit wrapping, a no-network mocked test, provider export, and a local chat example.\n\nI did not update the parent repo submodule pointer yet because CleanDev-Fix cannot push the SDK commit directly to the upstream SDK repo; once #30 is merged, the parent pointer can be updated to the upstream commit cleanly.

[nagasatish007]

@CleanDev-Fix Merge conflicts from the v1.4.0 release that landed on main today. Can you rebase onto latest main and force-push? I'll squash-merge once it's green.

[nagasatish007]

@richardchen874-sys That sounds good. Once #30 lands (TealOllama), the Decision object shape will be proven for local endpoints. At that point a follow-up issue for multi-model cost estimation / provider comparison would be welcome — feel free to open one with sample response shapes and we can scope it together.

[nagasatish007]

@richardchen874-sys — cost estimation across providers is the most useful framing. Specifically:

"Unified cost estimation for local vs. cloud LLM calls" — sample response shapes showing how the same Decision object captures cost whether the backend is Ollama (compute-time estimate), OpenAI (token pricing), or a managed compatible endpoint.

The fields you listed (provider, model, tokens, estimated cost, latency, decision outcome, policy reason) are exactly right. Policy-aware routing can build on top of that later as a separate issue.

Open it whenever you're ready — tag it enhancement + cost-governance.