Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: improve OpenAI model APIs and examples to better support audio, images, and tool calling #707

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

mattjohnsonpint
Copy link
Member

@mattjohnsonpint mattjohnsonpint commented Jan 16, 2025

Description

This PR makes several improvements to the OpenAI model interface APIs, for both AssemblyScript and Go.

  • Ensures tool calling works in both languages. Previously, we were missing some key properties for the Go SDK.
  • Simplifies tool calling (somewhat) with a few well-placed helper functions and convenience methods.
  • Adds support to all message types to allow passing content as either strings (as before) or as a list of content parts. In particular, this enables text intput, audio input, or image input for user messages.
  • Adds support for audio output as well (for models that support it)
  • Adds support for passing either "system" or "developer" messages - which are identical. OpenAI has deprecated "system" messages and prefers "developer" now, but other OpenAI-compatible models may not yet be updated, so both are supported for now.
  • Makes a few other minor changes to account for updates to OpenAI's schema
  • Adds localtime.isValidTimeZone(tz: string): bool to both SDKs (used by one of the tools in the tool calling examples)

The changes conform to OpenAI's published OpenAPI Schema - which in some areas is more accurate or specific than the end-user documentation.

❗ There are some minor breaking changes with this update, such as the role field of messages now being immutable. It should have little to no impact for most users. Any breaking changes should be noticeable at compile time, and should be resolvable with minimal effort.

❗ This PR is blocked and will stay in draft mode until AssemblyScript/assemblyscript#2894 is merged and released. It will then be updated to include a bump to AssemblyScript in the SDK.

Additionally, the "text generation" examples in both SDKs have been updated as follows:

  • Examples showing structured outputs (ie, product generation) have been moved to a new file.
  • New examples for media input/output have been added to another new file.
  • A new example for tool calling has been added to yet another new file.

Checklist

All PRs should check the following boxes:

  • I have given this PR a title using the
    Conventional Commits syntax, leading with fix:,
    feat:, chore:, ci:, etc.
    • The title should also be used for the commit message when the PR is squashed and merged.
  • I have formatted and linted my code with Trunk, per the instructions in
    the contributing guide.

If the PR includes a code change, then also check the following boxes. (If not, then delete the
next section.)

  • I have added an entry to the CHANGELOG.md file.
    • Add to the "UNRELEASED" section at the top of the file, creating one if it doesn't yet exist.
    • Be sure to include the link to this PR, and please sort the section numerically by PR number.
  • I have manually tested the new or modified code, and it appears to behave correctly.
  • I have added or updated unit tests where appropriate, if applicable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

1 participant