FirebaseVertexAI Properties in Object Schema not accepting all Schema types

[pauze]

Description

I'm trying to supply Object schema to VertexAI with some basic properties.

    let firebaseSchema = Schema.object(
      properties: [
        "name": .string(),
        "duration:": .double()
      ]
    )

However when I send this schema to the generative model I get the following error:

{
  "error": {
    "code": 400,
    "message": "Unable to submit request because one or more response schemas specified incorrect schema type field. For schema with format, schema type should be STRING. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output",
    "status": "INVALID_ARGUMENT"
  }
}

If I change duration to be of type .integer() then I get back a successful response. I've only tested a handful of types but so far .double() and .float() are not working as expected.

Reproducing the issue

   
   func generate(prompt: String) async {
    let firebaseSchema = Schema.object(
      properties: [
        "name": .string(),
        "duration:": .double()
      ]
    )
    
    let vertex = VertexAI.vertexAI()
    let generativeModel = vertex.generativeModel(modelName: "gemini-1.5-flash", generationConfig: GenerationConfig(responseMIMEType: "application/json", responseSchema: firebaseSchema))
    let response = try await generativeModel.generateContent(prompt)
    
    print("Response: \(response.text ?? "nil")")
    }

Firebase SDK Version

11.4.0

Xcode Version

16.0

Installation Method

Swift Package Manager

Firebase Product(s)

VertexAI

Targeted Platforms

iOS

Relevant Log Output

he server responded with an error: <NSHTTPURLResponse: 0x6000002dc680> { URL: <redacted>/us-central1/publishers/google/models/gemini-1.5-flash:generateContent } { Status Code: 400, Headers {
    "Alt-Svc" =     (
        "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
    );
    "Cache-Control" =     (
        private
    );
    "Content-Encoding" =     (
        gzip
    );
    "Content-Length" =     (
        262
    );
    "Content-Type" =     (
        "application/json; charset=UTF-8"
    );
    Date =     (
        "Sun, 27 Oct 2024 16:46:53 GMT"
    );
    Server =     (
        ESF
    );
    Vary =     (
        Origin,
        "X-Origin",
        Referer
    );
    "x-content-type-options" =     (
        nosniff
    );
    "x-frame-options" =     (
        SAMEORIGIN
    );
    "x-xss-protection" =     (
        0
    );
} }
11.4.0 - [FirebaseVertexAI][I-VTX002002] Response payload: {
  "error": {
    "code": 400,
    "message": "Unable to submit request because one or more response schemas specified incorrect schema type field. For schema with format, schema type should be STRING. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output",
    "status": "INVALID_ARGUMENT"
  }
}

If using Swift Package Manager, the project's Package.resolved

Expand Package.resolved snippet

Replace this line with the contents of your Package.resolved.

If using CocoaPods, the project's Podfile.lock

Expand Podfile.lock snippet

Replace this line with the contents of your Podfile.lock!

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[andrewheard]

Hi @pauze, sorry for the delay in getting back to you. I've been able to reproduce the issue and have let the backend team know about the regression.

In case this is blocking you, I created PR #13990. The branch ah/vertex-float-workaround contains the Firebase 11.4 release you're using and a temporary workaround of removing the "double" or "float" format specifier from Schema.float() and Schema.double(). If using Swift Package Manager, you could temporarily depend on the branch ah/vertex-float-workaround, instead of specifying a version, to try this out.

Thanks for providing code snippets and details about the types you tried. It made it easy to reproduce on my end. And sorry for the trouble, I'll keep this issue open until it's resolved in the backend or if we decide to to merge the workaround for Firebase 11.5.

[pauze]

Perfect, thanks for the update. I'll keep my eyes open for the release :)