Skip to content

API.md

Latest commit

 

History

History
988 lines (780 loc) · 26 KB

API.md

File metadata and controls

988 lines (780 loc) · 26 KB
Raw

OctoHub API Reference

Base URL: http://127.0.0.1:8080 (configurable via octohub.toml)

Authentication

OctoHub has two authentication layers, both controlled by the api_key setting in octohub.toml.

If api_key is not set (default): the server starts without authentication. All client endpoints are open, admin endpoints are disabled. A warning is printed at startup.

If api_key is set: full authentication is enforced on both layers.

Client endpoints (/v1/completions, /v1/chat/completions, /v1/embeddings)

Authenticated with a client API key issued via the admin API. Pass it as a Bearer token:

Authorization: Bearer <client-api-key>

Admin endpoints (/v1/admin/*)

Authenticated with the master key set in octohub.toml (api_key):

Authorization: Bearer <master-key>

All requests to admin endpoints without a valid master key return 401 Unauthorized. Admin endpoints are unavailable when no master key is configured.

Error Format

All errors return JSON:

{
  "error": {
    "message": "Description of what went wrong",
    "type": "invalid_request_error"
  }
}

Client Endpoints

OctoHub exposes two completion APIs. Use whichever fits your client:

Endpoint Style Best for
POST /v1/completions OctoHub Responses API Octomind and native OctoHub clients. Multi-turn chains, reasoning replay, richer output shape.
POST /v1/chat/completions Classic OpenAI Chat Completions Any OpenAI-compatible SDK or tool (LangChain, LiteLLM, curl with openai libs, etc.). Drop-in replacement.
POST /v1/embeddings OpenAI-compatible Embedding clients.

Both completion endpoints hit the same proxy engine and write to the same completions table with the same id, auth, metrics, and logging. The only difference is the wire format.

Note: POST /v1/chat/completions does not support streaming ("stream": true). Requests with streaming enabled receive 501 Not Implemented.

POST /v1/completions

OctoHub's native Responses API. Supports multi-turn conversation chains, reasoning-block replay, and a richer output structure. Used by Octomind.

Request

{
  "model": "string",
  "input": "string | InputItem[]",
  "instructions": "string | null",
  "previous_completion_id": "string | null",
  "temperature": 1.0,
  "max_output_tokens": 0,
  "tools": "ToolDefinition[] | null"
}
Field Type Required Default Description
model string ✅ — Model identifier. Use "provider:model" format (e.g. "openai:gpt-4o") or a mapped name from config.
input string | array ✅ — Input content. A plain string becomes a user message. An array allows typed items (see below).
instructions string — null System instructions prepended to the conversation.
previous_completion_id string — null Chain to a previous completion for multi-turn conversations. OctoHub reconstructs the full history automatically.
temperature float — 1.0 Sampling temperature (0.0–2.0).
max_output_tokens integer — 0 Maximum tokens in the response. 0 = provider default.
tools array — null Function definitions for tool/function calling.

Input Items

When input is an array, each element is a tagged object:

Message:

{"type": "message", "role": "user", "content": "What is Rust?"}

Function call output (tool result, requires previous_completion_id):

{"type": "function_call_output", "call_id": "call_abc123", "output": "72°F sunny"}

Tool Definition

{
  "type": "function",
  "name": "get_weather",
  "description": "Get weather for a location",
  "parameters": {
    "type": "object",
    "properties": {
      "location": {"type": "string"}
    }
  }
}

Response

{
  "id": "cmpl_<uuid>",
  "object": "completion",
  "output": [OutputItem],
  "usage": Usage,
  "created_at": 1700000000
}

Output Items

Message:

{
  "type": "message",
  "id": "msg_<uuid>",
  "role": "assistant",
  "content": [
    {"type": "output_text", "text": "Hello!"}
  ]
}

Function call:

{
  "type": "function_call",
  "id": "fc_<uuid>",
  "call_id": "call_abc123",
  "name": "get_weather",
  "arguments": "{\"location\":\"NYC\"}"
}

Usage

{
  "input_tokens": 10,
  "output_tokens": 5,
  "total_tokens": 15,
  "cache_read_tokens": null,
  "cache_write_tokens": null,
  "cost": 0.0001,
  "request_time_ms": 500
}

cache_read_tokens, cache_write_tokens, cost, and request_time_ms are omitted when null.

Examples

Simple text completion
curl -X POST http://127.0.0.1:8080/v1/completions \
  -H "Authorization: Bearer <client-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai:gpt-4o",
    "input": "Explain Rust in one sentence."
  }'
With system instructions
curl -X POST http://127.0.0.1:8080/v1/completions \
  -H "Authorization: Bearer <client-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai:gpt-4o",
    "input": "What should I learn first?",
    "instructions": "You are a programming tutor. Be concise.",
    "temperature": 0.7
  }'
Multi-turn conversation
# First turn
curl -X POST http://127.0.0.1:8080/v1/completions \
  -H "Authorization: Bearer <client-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai:gpt-4o",
    "input": "What is the capital of France?"
  }'
# Response: {"id": "cmpl_abc123", ...}

# Second turn — chains automatically
curl -X POST http://127.0.0.1:8080/v1/completions \
  -H "Authorization: Bearer <client-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai:gpt-4o",
    "input": "And what is its population?",
    "previous_completion_id": "cmpl_abc123"
  }'
Function calling
# Step 1: Ask with tools
curl -X POST http://127.0.0.1:8080/v1/completions \
  -H "Authorization: Bearer <client-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai:gpt-4o",
    "input": "What is the weather in NYC?",
    "tools": [
      {
        "type": "function",
        "name": "get_weather",
        "description": "Get current weather",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {"type": "string"}
          },
          "required": ["location"]
        }
      }
    ]
  }'
# Response includes: {"type": "function_call", "call_id": "call_xyz", "name": "get_weather", ...}

# Step 2: Send tool result back
curl -X POST http://127.0.0.1:8080/v1/completions \
  -H "Authorization: Bearer <client-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai:gpt-4o",
    "input": [
      {"type": "function_call_output", "call_id": "call_xyz", "output": "72°F, sunny"}
    ],
    "previous_completion_id": "cmpl_from_step1"
  }'
Using a mapped model name

With this config:

[models]
"my-model" = ["openai:gpt-4o", "anthropic:claude-sonnet-4-20250514"]
curl -X POST http://127.0.0.1:8080/v1/completions \
  -H "Authorization: Bearer <client-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-model",
    "input": "Hello!"
  }'

OctoHub randomly picks one provider from the list for load balancing.

POST /v1/chat/completions

Classic OpenAI Chat Completions API. Compatible with any OpenAI-compatible client or SDK. Internally converts to the same engine call as /v1/completions — all completions are stored in the DB with the same id, auth key, tokens, and provider metadata.

Request

{
  "model": "string",
  "messages": [{"role": "user", "content": "Hello"}],
  "temperature": 1.0,
  "top_p": 1.0,
  "max_tokens": null,
  "stream": false,
  "tools": null,
  "tool_choice": null
}
Field Type Required Default Description
model string ✅ — Model identifier — same as /v1/completions.
messages array ✅ — Conversation history. Roles: system, user, assistant, tool.
temperature float — 1.0 Sampling temperature.
top_p float — 1.0 Nucleus sampling.
max_tokens integer — provider default Maximum output tokens.
stream bool — false Streaming is not supported — true returns 501.
tools array — null Classic tool definitions (nested function object).
tool_choice any — null Accepted and ignored — provider decides.

Message conversion rules:

  • role=system messages → instructions (concatenated with newline if multiple)
  • role=tool messages → function_call_output (keyed by tool_call_id)
  • role=assistant with tool_calls → one function_call input item per call
  • All other messages → message input items (content string or parts preserved)

Response

Standard OpenAI Chat Completions response shape:

{
  "id": "cmpl_01JXXXXXXXXXX",
  "object": "chat.completion",
  "created": 1749300000,
  "model": "openai:gpt-4o",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I help?",
        "tool_calls": null
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 8,
    "total_tokens": 18
  }
}
  • id is the same ULID-based completion ID stored in the DB — use it to look up the record via the admin API
  • finish_reason is always a string; falls back to "stop" when the upstream doesn't report one
  • Tool call responses: choices[0].message.content is null, tool_calls is populated
  • Reasoning blocks from thinking models are not surfaced in the classic response (invisible to classic clients)

Examples

Using the Python openai SDK
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8080/v1",
    api_key="<client-api-key>",
)

response = client.chat.completions.create(
    model="openai:gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain Rust in one sentence."},
    ],
)
print(response.choices[0].message.content)
Using curl
curl -X POST http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer <client-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai:gpt-4o",
    "messages": [
      {"role": "system", "content": "Be concise."},
      {"role": "user", "content": "What is Rust?"}
    ]
  }'
Tool calling
# Step 1: ask with tools
curl -X POST http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer <client-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai:gpt-4o",
    "messages": [{"role": "user", "content": "What is the weather in NYC?"}],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get current weather",
          "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"]
          }
        }
      }
    ]
  }'
# Response: choices[0].message.tool_calls[0] = {id, type, function: {name, arguments}}

# Step 2: send result back
curl -X POST http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer <client-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai:gpt-4o",
    "messages": [
      {"role": "user", "content": "What is the weather in NYC?"},
      {"role": "assistant", "tool_calls": [{"id": "call_xyz", "type": "function", "function": {"name": "get_weather", "arguments": "{\"location\":\"NYC\"}"}}]},
      {"role": "tool", "tool_call_id": "call_xyz", "content": "72°F, sunny"}
    ]
  }'

POST /v1/embeddings

Generate vector embeddings for text input.

Request

{
  "model": "string",
  "input": "string | string[]"
}
Field Type Required Default Description
model string ✅ — Model identifier. Use "provider:model" format (e.g. "voyage:voyage-3.5") or a mapped name from config.
input string | string[] ✅ — Text(s) to embed. A single string or an array of strings.

Response

Returns the embedding vector(s) directly:

Single input:

[0.0023, -0.0091, 0.0234, ...]

Batch input:

[
  [0.0023, -0.0091, ...],
  [0.0156, 0.0089, ...],
  [0.0034, -0.0123, ...]
]

Examples

Single text
curl -X POST http://127.0.0.1:8080/v1/embeddings \
  -H "Authorization: Bearer <client-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "voyage:voyage-3.5",
    "input": "The quick brown fox jumps over the lazy dog."
  }'
Batch embedding
curl -X POST http://127.0.0.1:8080/v1/embeddings \
  -H "Authorization: Bearer <client-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai:text-embedding-3-small",
    "input": [
      "First document to embed",
      "Second document to embed",
      "Third document to embed"
    ]
  }'
Using a mapped embedding model

With this config:

[embedding_models]
"my-embeddings" = ["voyage:voyage-3.5"]
curl -X POST http://127.0.0.1:8080/v1/embeddings \
  -H "Authorization: Bearer <client-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-embeddings",
    "input": "Embed this text"
  }'

GET /health

Health check endpoint. No authentication required.

Response

{"status": "ok"}
curl http://127.0.0.1:8080/health

Admin Endpoints

All admin endpoints require the master key from octohub.toml:

Authorization: Bearer <master-key>

POST /v1/admin/keys

Create a new client API key.

Request

{
  "name": "string",
  "allowed_models": ["string"],
  "owner": "string",
  "owner_concurrency": 10
}
Field Type Required Description
name string ✅ Human-readable label for this key (e.g. "mobile-app", "ci-pipeline").
allowed_models string[] — Whitelist of model values this key may request. Match is exact against the model field as sent — either an alias from [models]/[embedding_models] or a raw provider:model string. Omit the field for an unrestricted key. Requests to a non-allowed model return 403 Forbidden.
owner string — Opaque grouping label (e.g. the tenant/account id in YOUR system). All keys sharing an owner share one in-flight request budget — see owner_concurrency. Omit for an ungrouped key.
owner_concurrency int — Max in-flight requests (completions + embeddings together) shared by ALL keys with this owner. A saturated budget queues a request up to 30s, then returns 429 Too Many Requests. Omit or 0 = unlimited.

Response 201 Created

{
  "id": 1,
  "name": "mobile-app",
  "key": "abc123...xyz",
  "key_hint": "...xyz",
  "status": "active",
  "allowed_models": ["gpt", "openai:gpt-4o"],
  "owner": "acct-42",
  "owner_concurrency": 10,
  "created_at": 1700000000
}

Important: The key field is only returned on creation. Store it securely — it cannot be retrieved again.

Field Description
id Numeric key ID. Used in admin queries to filter by key.
name Label as provided.
key Full 43-character base64url key. Only shown once.
key_hint Last 4 characters prefixed with ... for identification.
status "active" or "revoked".
allowed_models The model whitelist as configured. null means unrestricted.
owner Grouping label for the shared concurrency budget. null means ungrouped.
owner_concurrency Shared in-flight budget for all keys of this owner. null/0 means unlimited.
created_at Unix timestamp.

Examples

# Unrestricted key — may call any model
curl -X POST http://127.0.0.1:8080/v1/admin/keys \
  -H "Authorization: Bearer <master-key>" \
  -H "Content-Type: application/json" \
  -d '{"name": "mobile-app"}'

# Restricted key — limited to two models
curl -X POST http://127.0.0.1:8080/v1/admin/keys \
  -H "Authorization: Bearer <master-key>" \
  -H "Content-Type: application/json" \
  -d '{"name": "ci-pipeline", "allowed_models": ["gpt", "anthropic:claude-haiku-4-5"]}'

GET /v1/admin/keys

List all API keys. The full key value is never returned — only the hint.

Response 200 OK

{
  "data": [
    {
      "id": 1,
      "name": "mobile-app",
      "key_hint": "...xyz",
      "status": "active",
      "allowed_models": null,
      "created_at": 1700000000
    },
    {
      "id": 2,
      "name": "ci-pipeline",
      "key_hint": "...abc",
      "status": "revoked",
      "allowed_models": ["gpt", "anthropic:claude-haiku-4-5"],
      "created_at": 1700001000
    }
  ]
}

Example

curl http://127.0.0.1:8080/v1/admin/keys \
  -H "Authorization: Bearer <master-key>"

GET /v1/admin/keys/:id

Get a single API key by its numeric ID.

Response 200 OK

{
  "id": 1,
  "name": "mobile-app",
  "key_hint": "...xyz",
  "status": "active",
  "allowed_models": null,
  "created_at": 1700000000
}

Returns 404 if the key does not exist.

Example

curl http://127.0.0.1:8080/v1/admin/keys/1 \
  -H "Authorization: Bearer <master-key>"

POST /v1/admin/keys/:id/revoke

Revoke an API key. Revoked keys are rejected on all client endpoints immediately. Keys cannot be deleted — only revoked — because usage records are linked to the key ID.

Response 200 OK

{"status": "revoked"}

Returns 404 if the key does not exist.

Example

curl -X POST http://127.0.0.1:8080/v1/admin/keys/1/revoke \
  -H "Authorization: Bearer <master-key>"

POST /v1/admin/keys/:id/models

Replace an active key's allowed_models list in place — the key value stays valid, so deployed credentials keep working while their model access changes. Absent/null = unrestricted.

curl -X POST http://127.0.0.1:8080/v1/admin/keys/1/models \
  -H "Authorization: Bearer <master-key>" \
  -H "Content-Type: application/json" \
  -d '{"allowed_models": ["gpt", "voyage-4"]}'

Returns 200 {"status": "updated"}, or 404 if the key does not exist or is not active.

POST /v1/admin/keys/:id/owner

Replace an active key's owner grouping and shared owner_concurrency budget in place (same contract as /models). Set this identically on every key of a tenant so all of them drain one budget — minting more keys never widens concurrency.

curl -X POST http://127.0.0.1:8080/v1/admin/keys/1/owner \
  -H "Authorization: Bearer <master-key>" \
  -H "Content-Type: application/json" \
  -d '{"owner": "acct-42", "owner_concurrency": 10}'

Returns 200 {"status": "updated"}, or 404 if the key does not exist or is not active. Absent/null owner ungroups the key; absent/null/0 concurrency means unlimited. A capacity change applies to new requests immediately (requests already in flight drain on the old budget).

GET /v1/admin/usage

Aggregated usage statistics. Optionally grouped by time bucket and filtered by key.

Query Parameters

Parameter Type Description
key_id string Comma-separated key IDs to filter by (e.g. key_id=1,3). Omit for all keys.
bucket string Time grouping: hour, day, week, month. Omit for a single total row per key.
since integer Unix timestamp — only include records after this time.
until integer Unix timestamp — only include records before this time.

Response 200 OK

{
  "data": [
    {
      "period": 1700000000,
      "key_id": 1,
      "key_name": "mobile-app",
      "completions_count": 42,
      "embeddings_count": 10,
      "total_input_tokens": 8500,
      "total_output_tokens": 3200
    }
  ]
}
Field Description
period Start of the time bucket as unix timestamp. null when no bucket is specified (total).
key_id API key ID.
key_name API key name at time of query.
completions_count Number of completion requests in this period.
embeddings_count Number of embedding requests in this period.
total_input_tokens Sum of input tokens across all requests.
total_output_tokens Sum of output tokens across completion requests.

Examples

# Total usage across all keys
curl "http://127.0.0.1:8080/v1/admin/usage" \
  -H "Authorization: Bearer <master-key>"

# Daily breakdown for key 1 in the last week
curl "http://127.0.0.1:8080/v1/admin/usage?key_id=1&bucket=day&since=1699395200" \
  -H "Authorization: Bearer <master-key>"

# Hourly breakdown for keys 1 and 3
curl "http://127.0.0.1:8080/v1/admin/usage?key_id=1,3&bucket=hour" \
  -H "Authorization: Bearer <master-key>"

GET /v1/admin/completions

List raw completion records with full input/output.

Query Parameters

Parameter Type Default Description
key_id string all Comma-separated key IDs to filter by.
since integer — Unix timestamp lower bound.
until integer — Unix timestamp upper bound.
limit integer 100 Max records to return.
offset integer 0 Pagination offset.

Response 200 OK

{
  "data": [
    {
      "id": "cmpl_<uuid>",
      "api_key_id": 1,
      "session_id": "<uuid>",
      "input_model": "my-model",
      "resolved_model": "gpt-4o",
      "provider": "openai",
      "usage": {
        "input_tokens": 10,
        "output_tokens": 5,
        "total_tokens": 15,
        "cost": 0.0001,
        "request_time_ms": 320
      },
      "input": [...],
      "output": [...],
      "created_at": 1700000000
    }
  ]
}

Example

# Last 50 completions for key 2
curl "http://127.0.0.1:8080/v1/admin/completions?key_id=2&limit=50" \
  -H "Authorization: Bearer <master-key>"

GET /v1/admin/embeddings

List raw embedding records.

Query Parameters

Parameter Type Default Description
key_id string all Comma-separated key IDs to filter by.
since integer — Unix timestamp lower bound.
until integer — Unix timestamp upper bound.
limit integer 100 Max records to return.
offset integer 0 Pagination offset.

Response 200 OK

{
  "data": [
    {
      "id": "emb_<uuid>",
      "api_key_id": 1,
      "input_model": "my-embeddings",
      "resolved_model": "voyage-3.5",
      "provider": "voyage",
      "usage": {
        "input_tokens": 8,
        "total_tokens": 8,
        "request_time_ms": 120
      },
      "input": ["text to embed"],
      "created_at": 1700000000
    }
  ]
}

Example

curl "http://127.0.0.1:8080/v1/admin/embeddings?key_id=1&limit=20&offset=40" \
  -H "Authorization: Bearer <master-key>"

Per-Provider Concurrency

Each model alias resolves to a real upstream provider (ollama, openai, anthropic, ...). OctoHub can cap the number of in-flight requests it forwards to each provider. Configure in octohub.toml:

[providers.ollama]
concurrency = 4

[providers.openai]
concurrency = 32

Provider names are case-insensitive. Omit a provider to leave it unlimited.

When a provider is at its limit, additional requests queue inside the OctoHub process — the client's HTTP connection stays open and blocks until a slot frees up. This is intentional throttling, not an error; clients see it as a longer-than-usual response, never a 429. The default HTTP/1 keep-alive is disabled at the server, so no stale pooled connection lingers across the wait.

The limiter is process-local. It tracks completions and embeddings together (both flow through the same provider connection).

Supported Providers

Completions (/v1/completions)

Use "provider:model" format. Available providers depend on octolib configuration and environment variables (API keys).

Provider Format Example
OpenAI openai:gpt-4o
Anthropic anthropic:claude-sonnet-4-20250514
Google google:gemini-2.0-flash
DeepSeek deepseek:deepseek-chat
MiniMax minimax:minimax-m2.7
Ollama ollama:llama3
OpenRouter openrouter:meta-llama/llama-3-70b

Embeddings (/v1/embeddings)

Provider Format Example
Voyage voyage:voyage-3.5
OpenAI openai:text-embedding-3-small
Jina jina:jina-embeddings-v3
Google google:gemini-embedding-001
OpenRouter openrouter:openai/text-embedding-3-small

Quick Start

# 1. Start the server (api_key must be set in octohub.toml)
./octohub

# 2. Create a client API key
curl -X POST http://127.0.0.1:8080/v1/admin/keys \
  -H "Authorization: Bearer <master-key>" \
  -H "Content-Type: application/json" \
  -d '{"name": "my-app"}'
# → {"id": 1, "key": "abc...xyz", ...}  ← save this key!

# 3. Use the client key for completions
curl -X POST http://127.0.0.1:8080/v1/completions \
  -H "Authorization: Bearer abc...xyz" \
  -H "Content-Type: application/json" \
  -d '{"model": "openai:gpt-4o", "input": "Hello!"}'

# 4. Check usage
curl "http://127.0.0.1:8080/v1/admin/usage?key_id=1&bucket=day" \
  -H "Authorization: Bearer <master-key>"

Error Responses

All errors follow this format:

{
  "error": {
    "message": "Description of what went wrong",
    "type": "invalid_request_error"
  }
}
HTTP Status Meaning
400 Bad request (missing/invalid fields, bad model name)
401 Missing or invalid API key
403 The API key is not permitted to use the requested model (see allowed_models)
404 Unknown endpoint
429 The key's shared owner concurrency budget stayed saturated for the 30s queue window — reduce parallelism and retry
500 Internal error (provider failure, storage error)