feat(api): oasdiff OpenAI openAPI spec against ours #3529

cdoern · 2025-09-23T18:52:27Z

What does this PR do?

diff the /v1/ routes that are OpenAI compatible against the OpenAI openAPI spec. This will of course only trigger on PRs where the spec is changed.

This will catch errors with new handwritten additions to our openAI compat routes.

Instead of fetching the OpenAPI spec from a dynamic URL, which could cause non-deterministic build failures,

this change uses a local copy stored at docs/static/openai-spec.yml.

This makes the conformance check fully reproducible and prevents CI failures caused by uncontrolled upstream changes.

I am marking this test with continue-on-error: true, until we get rid of all of the errors. Nevertheless, this is a nice utility to have so folks know if their spec changes introduce more breaking changes or fix breakages when comparing to the OpenAI openapi spec.

Test Plan

test should pass.

.github/workflows/conformance.yml

raghotham · 2025-09-24T20:58:38Z

.github/workflows/conformance.yml

+        if: steps.cache-openapi.outputs.cache-hit != 'true'
+        run: |
+          mkdir -p ~/openai-openapi
+          curl -L https://app.stainless.com/api/spec/documented/openai/openapi.documented.yml -o ~/openai-openapi/openai-openapi.yml


do we not want to pin the version of the openai api?

sure, I can pin it. Do we have a version in mind we want to use?

I think the pinning is important, what's the latest version so we can get started with it?

see my comment below, added openapi spec to the repo, there is not a versioned URL.

.github/workflows/conformance.yml

raghotham · 2025-09-24T21:00:11Z

.github/workflows/conformance.yml

+        id: cache-openapi
+        uses: actions/cache@0400d5f644dc74513175e3cd8d07132dd4860809
+        with:
+          path: ~/openai-openapi


why use ~ here?

I believe the cache action by default has access only to a runners home dir? I could be wrong but that is where I have been storing and writing the cached binaries+files to/from

raghotham · 2025-09-24T21:00:42Z

.github/workflows/conformance.yml

+          key: openai-openapi.yml
+
      # Cache oasdiff to avoid checksum failures and speed up builds
      - name: Cache oasdiff


we should probably also support a force no-cache conformance check as well - we should probably run it at the time of release.

I agree, but should I make this a separate action so it can be triggered by maintainers manually?

leseb

I think this action could result in a "badge" on the repo once we feel confident, having the compatibility ratio again openai is very important :)

cdoern · 2025-10-03T16:08:04Z

this now checks all of our APIs against all of the openAI apis. The removal of /v1/openai/v1/ makes this kind of difficult

ashwinb · 2025-11-06T19:00:47Z

@cdoern do you plan to pick this up?

cdoern · 2025-11-06T19:08:06Z

yes @ashwinb I can get back on this today/tomorrow. thanks for the reminder!

cdoern · 2025-11-07T20:11:49Z

@ashwinb this should be set now for review

After my last commit, I realized that while the oasdiff command was working, our CI was still vulnerable to upstream changes in the OpenAI spec since we were fetching it live on every run.

My initial thought was to find a permanent, version-locked URL to pin against, but after some investigation, it turns out there isn't a stable, versioned URL available for the exact spec we need, openai only published the most recent one to a stainless URL.

To guarantee build stability and make our conformance test fully hermetic, I've changed my approach to vendor the OpenAI spec directly into our repository at docs/static/openai-spec.yml.

diff the `/v1/` routes that are OpenAI compatible against the OpenAI openAPI spec. This will of course only trigger on PRs where the spec is changed. This will catch errors with new handwritten additions to our openAI compat routes. Instead of fetching the OpenAPI spec from a dynamic URL, which could cause non-deterministic build failures, this change uses a local copy stored at `docs/static/openai-spec.yml`. This makes the conformance check fully reproducible and prevents CI failures caused by uncontrolled upstream changes. Signed-off-by: Charlie Doern <[email protected]>

franciscojavierarceo · 2025-11-07T20:15:08Z

.github/workflows/conformance.yml

+        continue-on-error: true
+        shell: bash
+        run: |
+          OPENAI_SPEC=docs/static/openai-spec.yml


Rather than manually download this, should we be downloading it dynamically?

In the OpenAI repo here: https://github.com/openai/openai-openapi/blob/master/README.md

They state the current version is here: https://app.stainless.com/api/spec/documented/openai/openapi.documented.yml

Obviously this will be brittle when things are updated but we could instead have versioned saved and versioned nightly or something.

I had it this way in a previous version of the PR, but there is a request here to stick with a specific version of the openai spec.

Since they don't seem to have a versioned URL there really isn't a way for me to "version" it if we pull it dynamically.

One idea I had is we could cache it and maybe if we pull it and the version has changed, don't cache it again? I don't know if this is possible though, wdyt?

yeah, that's reasonable. I think we can store this file to seed it and we could run a nightly job to check if it needs to be updated. We could either automatically update it or make a PR right?

We could add a version either to the name of the file or somewhere in the file itself

the file has a version: 2.3.0 which I was planning on using once the file is pulled to see if that has changed. I could do that or I could even use oasdiff To diff the old openai spec against the new one.

@franciscojavierarceo do you think I should do that in this PR or a separate one since this should probably be a different workflow that runs nightly, right?

yeah can do in a follow up (this PR is also fine to me tbh since it's heavily related)

leseb

Thanks!

leseb · 2025-11-10T14:08:18Z

.github/workflows/conformance.yml

+        shell: bash
+        run: |
+          OPENAI_SPEC=docs/static/openai-spec.yml
+          LOCAL_SPEC=docs/static/llama-stack-spec.yaml


Suggested change

LOCAL_SPEC=docs/static/llama-stack-spec.yaml

LLAMA_STACK_SPEC=docs/static/llama-stack-spec.yaml

If you feel like this is better :)

leseb · 2025-11-10T14:10:28Z

docs/static/openai-spec.yml

+info:
+  title: OpenAI API
+  description: The OpenAI REST API. Please see https://platform.openai.com/docs/api-reference for more details.
+  version: 2.3.0


I'm wondering if we should either:

put this into a 2.3.0 directory

or add -2.3.0 to the file name

Or perhaps, are we always replacing / overwritting with a new spec?

meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 23, 2025

cdoern force-pushed the oasdiff-for-openai branch 10 times, most recently from 67b7c6e to b509e85 Compare September 24, 2025 14:49

cdoern marked this pull request as ready for review September 24, 2025 14:51

cdoern requested review from ashwinb, bbrowning, ehhuang, hardikjshah, leseb, mattf, raghotham, reluctantfuturist, slekkala1, terrytangyuan and yanxi0830 as code owners September 24, 2025 14:51

cdoern force-pushed the oasdiff-for-openai branch from b509e85 to 8a2bb5f Compare September 24, 2025 14:51

raghotham requested changes Sep 24, 2025

View reviewed changes

cdoern force-pushed the oasdiff-for-openai branch 4 times, most recently from da078af to c5bd55e Compare September 27, 2025 19:19

cdoern requested a review from raghotham September 27, 2025 19:20

leseb reviewed Oct 3, 2025

View reviewed changes

cdoern force-pushed the oasdiff-for-openai branch 3 times, most recently from d7e8b77 to 5e68691 Compare October 3, 2025 15:52

cdoern force-pushed the oasdiff-for-openai branch 2 times, most recently from f423104 to d9abb2e Compare October 3, 2025 16:48

cdoern force-pushed the oasdiff-for-openai branch from d9abb2e to 867e0a4 Compare November 7, 2025 20:10

cdoern requested a review from franciscojavierarceo as a code owner November 7, 2025 20:10

franciscojavierarceo reviewed Nov 7, 2025

View reviewed changes

cdoern force-pushed the oasdiff-for-openai branch from 867e0a4 to a8928b6 Compare November 7, 2025 20:15

leseb reviewed Nov 10, 2025

View reviewed changes

	LOCAL_SPEC=docs/static/llama-stack-spec.yaml
	LLAMA_STACK_SPEC=docs/static/llama-stack-spec.yaml

feat(api): oasdiff OpenAI openAPI spec against ours #3529

Are you sure you want to change the base?

feat(api): oasdiff OpenAI openAPI spec against ours #3529

Conversation

cdoern commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Test Plan

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leseb left a comment

Choose a reason for hiding this comment

Uh oh!

cdoern commented Oct 3, 2025

Uh oh!

ashwinb commented Nov 6, 2025

Uh oh!

cdoern commented Nov 6, 2025

Uh oh!

cdoern commented Nov 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cdoern Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leseb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cdoern commented Sep 23, 2025 •

edited

Loading

cdoern Nov 7, 2025 •

edited

Loading