Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions apps/docs/en/manual/model-prices.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,15 @@ Open the [Model Prices Demo](https://seakee.github.io/CPA-Manager-Plus/#/demo/mo

Synchronization only occurs when the user triggers it and may use the current Manager Server proxy configuration.

The same models.dev model ID may be offered by multiple providers, and a real model ID may itself contain `/`. CPAMP compares source identities separately from original model IDs and only matches automatically when the complete pricing metadata, including tiers and experimental-mode prices, is consistent. Identity or price conflicts remain in the candidate-confirmation flow; a same-named LiteLLM or OpenRouter entry cannot bypass that conflict.
Automatic matching runs strictly in models.dev, LiteLLM, then OpenRouter order. CPAMP uses the canonical model metadata in the models.dev catalog to prefer the first-party official entry. A source is saved automatically only when it has one clear, strong identity match; fuzzy similarities are never auto-confirmed. An ambiguous source falls through to the next source. If none of the three sources yields a unique match, the confirmation list keeps candidates from each source separately, even when they share the same original model ID.

The current sync maps models.dev `cost.input`, `cost.output`, `cost.cache_read`, and `cost.cache_write`, converts valid `cost.tiers` context tiers into CPAMP billing rules, and maps `experimental.modes.fast.cost` to short-context Fast/Priority prices. The complete model object remains available in raw metadata; reasoning prices, unknown experimental modes, unknown tier types, and rules that cannot be validated safely do not activate automatic billing.

### Sync failures and last-known-good prices

- When models.dev is temporarily unavailable, CPAMP continues with LiteLLM and OpenRouter.
- A transient models.dev failure cannot automatically replace a stored models.dev price with a lower-priority source; fallback sources may still fill models that have no local price.
- A fallback source may replace a model normally when models.dev responds successfully but does not contain that model.
- When models.dev responds successfully but has no official entry or remains ambiguous, fallback sources are tried in order; only a unique strong identity match may replace the model.
- If every source fails, synchronization stops before any database write and existing prices remain unchanged.
- A synchronized price remains the last-known-good value until a later successful sync or a manual edit; `syncedAtMs` indicates its freshness.

Expand Down
4 changes: 2 additions & 2 deletions apps/docs/manual/model-prices.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,15 @@ description: 配置 CPA Manager Plus 模型价格、service tier、长上下文

同步只在用户主动触发时发生,可能使用当前 Manager Server 代理设置。

models.dev 中同一个模型 ID 可能由多个 Provider 提供,而且真实模型 ID 本身也可能包含 `/`。CPAMP 会分别比较来源身份和原始模型 ID;只有完整价格元数据(包括阶梯和实验模式价格)明确一致时才自动匹配。任何身份或价格冲突都会进入候选确认流程,LiteLLM 或 OpenRouter 的同名条目不会绕过该冲突
自动匹配会严格按 models.dev、LiteLLM、OpenRouter 的顺序进行。CPAMP 使用 models.dev catalog 的规范模型元数据优先识别第一方官方条目;每个来源都只有唯一、明确的模型身份匹配才会自动保存,模糊相似项不会自动确认。某个来源存在歧义时会继续尝试下一来源;三个来源都无法唯一确认时,待确认列表会分别保留各来源的候选,即使它们的原始模型 ID 相同也不会互相覆盖

当前同步会映射 models.dev 的 `cost.input`、`cost.output`、`cost.cache_read` 和 `cost.cache_write`,将有效的 `cost.tiers` 上下文阶梯转换为 CPAMP 计费规则,并将 `experimental.modes.fast.cost` 映射为 Fast/Priority 短上下文价格。完整模型对象仍保存在原始元数据中;reasoning、未知实验模式、未知阶梯类型或无法安全验证的规则不会激活自动计费。

### 同步失败与最后有效价格

- models.dev 暂时不可用时,CPAMP 会继续尝试 LiteLLM 和 OpenRouter。
- 已保存的 models.dev 价格不会因为本次网络失败而被低优先级来源自动覆盖;回退来源仍可补充本地没有价格的模型。
- 只有 models.dev 成功响应但明确不包含某个模型时,才允许回退来源正常替换该模型
- models.dev 成功响应但缺少官方条目或匹配存在歧义时,会按顺序尝试回退来源;只有唯一的强身份匹配才会替换该模型
- 如果所有来源都失败,同步在写入数据库前终止,现有价格保持不变。
- 同步价格会一直作为最后有效数据使用,直到后续成功同步或用户手动修改;`syncedAtMs` 可用于判断数据新鲜度。

Expand Down
2 changes: 1 addition & 1 deletion apps/manager-server/internal/httpapi/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ var embeddedPanel embed.FS

const serviceID = "cpa-manager-plus"

var modelsDevModelPriceSyncURL = "https://models.dev/api.json"
var modelsDevModelPriceSyncURL = "https://models.dev/catalog.json"
var modelPriceSyncURL = "https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json"
var openRouterModelPriceSyncURL = "https://openrouter.ai/api/v1/models"

Expand Down
56 changes: 28 additions & 28 deletions apps/manager-server/internal/httpapi/server_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -746,20 +746,25 @@ func TestModelPricesSyncFromLiteLLMFormat(t *testing.T) {
}
}

func TestModelPricesSyncPrefersModelsDevProviderScopedPrices(t *testing.T) {
func TestModelPricesSyncUsesModelsDevOfficialAndOrderedFallback(t *testing.T) {
modelsDevSource := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{
"openai": {"models": {
"gpt-test": {"cost":{"input":9,"output":10,"cache_read":1}},
"ambiguous": {"cost":{"input":3,"output":4}}
}},
"azure": {"models": {
"ambiguous": {"cost":{"input":5,"output":6}}
}},
"crossmodel": {"models": {
"openai/gpt-test": {"cost":{"input":11,"output":12}}
}}
"models": {
"openai/gpt-test": {"id":"openai/gpt-test","name":"GPT Test"}
},
"providers": {
"openai": {"models": {
"gpt-test": {"cost":{"input":9,"output":10,"cache_read":1}},
"ambiguous": {"cost":{"input":3,"output":4}}
}},
"azure": {"models": {
"ambiguous": {"cost":{"input":5,"output":6}}
}},
"crossmodel": {"models": {
"openai/gpt-test": {"cost":{"input":11,"output":12}}
}}
}
}`))
}))
t.Cleanup(modelsDevSource.Close)
Expand Down Expand Up @@ -809,22 +814,9 @@ func TestModelPricesSyncPrefersModelsDevProviderScopedPrices(t *testing.T) {
if len(response.Sources) != 2 || response.Sources[0] != "models.dev" || response.Sources[1] != "litellm" {
t.Fatalf("source order = %#v", response.Sources)
}
if response.Imported != 4 || len(response.Candidates) != 2 {
if response.Imported != 6 || len(response.Candidates) != 0 {
t.Fatalf("sync selection = %#v", response)
}
candidateSources := map[string]map[string]bool{}
for _, set := range response.Candidates {
candidateSources[set.Model] = map[string]bool{}
for _, candidate := range set.Candidates {
candidateSources[set.Model][candidate.SourceModelID] = true
}
}
if !candidateSources["ambiguous"]["openai/ambiguous"] ||
!candidateSources["ambiguous"]["azure/ambiguous"] ||
!candidateSources["openai/gpt-test"]["openai/gpt-test"] ||
!candidateSources["openai/gpt-test"]["crossmodel/openai/gpt-test"] {
t.Fatalf("candidate sources = %#v", candidateSources)
}
price, ok := response.Prices["gpt-test"]
if !ok || !closeFloat(price.Prompt, 9) || !closeFloat(price.Completion, 10) || price.Source != "models.dev" || price.SourceModelID != "openai/gpt-test" {
t.Fatalf("models.dev alias price = %#v", price)
Expand All @@ -837,8 +829,13 @@ func TestModelPricesSyncPrefersModelsDevProviderScopedPrices(t *testing.T) {
if !ok || !closeFloat(crossmodel.Prompt, 11) || crossmodel.SourceModelID != "crossmodel/openai/gpt-test" {
t.Fatalf("nested provider-scoped price = %#v", crossmodel)
}
if _, ok := response.Prices["openai/gpt-test"]; ok {
t.Fatalf("colliding model was imported without confirmation: %#v", response.Prices["openai/gpt-test"])
officialScoped, ok := response.Prices["openai/gpt-test"]
if !ok || !closeFloat(officialScoped.Prompt, 9) || officialScoped.Source != "models.dev" || officialScoped.SourceModelID != "openai/gpt-test" {
t.Fatalf("official scoped price = %#v", officialScoped)
}
ambiguous, ok := response.Prices["ambiguous"]
if !ok || !closeFloat(ambiguous.Prompt, 7) || ambiguous.Source != "litellm" || ambiguous.SourceModelID != "ambiguous" {
t.Fatalf("ordered ambiguity fallback = %#v", ambiguous)
}
fallback, ok := response.Prices["fallback-only"]
if !ok || !closeFloat(fallback.Prompt, 1) || fallback.Source != "litellm" || fallback.SourceModelID != "fallback-only" {
Expand All @@ -858,7 +855,10 @@ func TestModelPricesSyncCachesModelsDevAndSkipsCoveredFallbacks(t *testing.T) {
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("ETag", etag)
_, _ = w.Write([]byte(`{"openai":{"models":{"gpt-test":{"cost":{"input":9,"output":10}}}}}`))
_, _ = w.Write([]byte(`{
"models":{"openai/gpt-test":{"id":"openai/gpt-test"}},
"providers":{"openai":{"models":{"gpt-test":{"cost":{"input":9,"output":10}}}}}
}`))
return
}
if received := r.Header.Get("If-None-Match"); received != etag {
Expand Down
Loading
Loading