Skip to content

feat(models): support provider-owned image models - #164

Merged
MagicCube merged 4 commits into
deer-flow:mainfrom
alwaycoding:feat/provider-owned-image-models
Sep 9, 2026
Merged

feat(models): support provider-owned image models#164
MagicCube merged 4 commits into
deer-flow:mainfrom
alwaycoding:feat/provider-owned-image-models

Conversation

@alwaycoding

@alwaycoding alwaycoding commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • make image generation a provider-owned capability, with image-only custom providers and separate chat/image inventories
  • support standard OpenAI Images, Ark Images, and explicit extra_body compatibility modes
  • preserve provider/model/profile selection, Ark compatibility, and legacy generate_image behavior
  • validate and recover persisted image model configuration without affecting chat inventories

Review Fixes

  • Use OpenAI pixel dimensions and omit response_format by default for GPT Image; canonical DALL-E models explicitly request base64.
  • Preserve an explicit ark-images protocol on custom providers through save/reload.
  • Add an optional model-level responseFormat: "b64_json" setting for gateway aliases, wired through the editor, persistence, and request payload. No image-URL download path is introduced.
  • Normalize legacy 1K to 1024x1024 only for standard OpenAI Images, across the settings editor, inventory validation, and thread tools. Editing a model no longer replaces its size set with all options or resets its default to auto. Ambiguous legacy sizes require explicit correction; Ark and extra-body presets remain unchanged.
  • Apply the old GPT Image size restrictions only to known GPT Image 1 variants/snapshots. GPT Image 2 and its dated ID accept the configured 1792x1024 option while DALL-E/GPT Image 1 restrictions remain enforced.
  • Merge current main (f2bb156) and preserve its desktop internationalization. New image settings and provider selection labels are available in English and Chinese.

Latest integration: afa506bcbb9613cd0935f1cf958297d626d92a09.

Verification

  • full Bun suite: 904/904 tests passed across 164 files
  • focused image/runtime/UI/i18n suite: 85/85 tests passed, including 8 mounted React regressions
  • full repository ESLint: passed with zero warnings/errors
  • full cross-package TypeScript check: passed
  • Prettier and git diff --check: passed
  • desktop renderer production build and web production build: passed (existing chunk-size warnings; web also reports its tolerated target/top-level-await warning)
  • final PR diff scan: 27 files, no .trae/evaluation assets or credential matches

The prior live Agnes smoke at 9a93982 generated a 1K PNG. Live OpenAI/Agnes requests and native CEF interaction were not rerun for this integration commit; provider behavior is covered with injected responses and mounted React tests.

No Agnes-specific identifiers or endpoints are hard-coded in the implementation. Evaluation assets and credentials are excluded from this PR.

@MagicCube MagicCube left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found two issues with image protocol handling. The 33 focused tests I ran passed, but do not cover these cases.

return_base64: true,
extra_body: { response_format: "b64_json" },
}
: { response_format: "b64_json" }),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Build a compatible payload for standard OpenAI Images

The new openai-images mode still sends metadata.size verbatim, while configuration and runtime validation only permit 1K/2K/3K/4K. Standard OpenAI Images requires pixel dimensions such as 1024x1024. This branch also always sends response_format, which GPT Image models do not support. Consequently, configuring a custom provider against the standard OpenAI endpoint cannot generate an image with the available settings. Please support protocol/model-appropriate sizes and output parameters, and add coverage for the standard mode alongside the extra-body fixture. Official parameter definitions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9a93982. Standard OpenAI Images now uses the official pixel-size options, maps the legacy 1K value to 1024x1024, rejects ambiguous larger presets before fetch, and only sends response_format for DALL-E models (GPT Image omits it). The settings editor and generate_image contract now expose protocol-compatible sizes. Added focused GPT Image, DALL-E, and invalid-preset coverage.

]
: [];
return {
...(api !== "ark-images" ? { api } : {}),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve an explicit Ark protocol on custom providers

Normalization unconditionally drops api: "ark-images", but a custom provider defaults to openai-images when api is absent. I reproduced this by adding a custom provider, saving an image configuration with api: "ark-images" through updateProvider, and constructing a new ModelManager with the same settings directory: the API changes from ark-images to undefined. After restart, generation therefore uses a different payload and omits Ark's watermark/stream fields. Please only omit the API when it equals defaultApi, or preserve the normalized API explicitly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9a93982. Image protocol normalization now omits api only when it equals the owning provider default, so a custom provider explicitly configured with ark-images survives persistence and reload. Added a restart regression test and exposed Ark Images in the custom-provider image API selector.

@alwaycoding

Copy link
Copy Markdown
Contributor Author

Pushed follow-up fixes in 814298d:

  1. Gateway model aliases can explicitly request b64_json through a persisted model setting. GPT Image defaults still omit response_format.
  2. Legacy 1K standard-OpenAI models retain their size/default when edited. Both new and existing thread tools remain usable after save/reload, with matching UI display. Native Ark/extra-body presets are unchanged.
  3. GPT Image 2 no longer inherits the GPT Image 1 size whitelist; 1792x1024 works for its regular and dated IDs. Known GPT Image 1 and DALL-E limits remain checked.

Verification: 72/72 top-level focused tests, including an isolated suite of 8/8 mounted React regressions; changed-file lint/typecheck, Prettier, and desktop/web production builds passed. The PR description records build warnings and the live-provider/native-CEF checks that were not rerun.

@alwaycoding

Copy link
Copy Markdown
Contributor Author

Resolved the current main conflicts in merge commit afa506bcbb9613cd0935f1cf958297d626d92a09 (base f2bb15600d1934bad62cf2595374ef1b27056ac2).

Resolution details:

  • preserved upstream desktop/provider internationalization and localized the new image API, response-format, provider-selection, and validation labels in English and Chinese
  • preserved provider-owned image inventories, custom image-provider selection, connection persistence, and legacy OpenAI 1K normalization
  • adapted the mounted settings regression fixture to the new real I18nProvider

Local verification:

  • full test suite: 904/904 passed (the Python generator test ran with jinja2 in an isolated /tmp virtualenv)
  • full lint: passed
  • full cross-package typecheck: passed
  • desktop renderer and web production builds: passed
  • final PR diff scan: 27 files, no .trae/evaluation assets or credential matches

GitHub now reports the PR as mergeable; checks may still need to settle.

@MagicCube
MagicCube merged commit 7280f78 into deer-flow:main Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants