Skip to content

fix(tokenizer): resolve TypeError #914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(tokenizer): resolve TypeError #914

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions apps/gateway/src/chat/chat.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,11 @@ import { getProviderEnv } from "./tools/get-provider-env.js";
import { parseProviderResponse } from "./tools/parse-provider-response.js";
import { transformResponseToOpenai } from "./tools/transform-response-to-openai.js";
import { transformStreamingToOpenai } from "./tools/transform-streaming-to-openai.js";
import { type ChatMessage, DEFAULT_TOKENIZER_MODEL } from "./tools/types.js";
import {
type ChatMessage,
DEFAULT_TOKENIZER_MODEL,
extractTextFromMessageContent,
} from "./tools/types.js";
import { validateFreeModelUsage } from "./tools/validate-free-model-usage.js";

import type { ServerTypes } from "@/vars.js";
Expand Down Expand Up @@ -716,10 +720,7 @@ chat.openapi(completions, async (c) => {
try {
const chatMessages: ChatMessage[] = messages.map((m) => ({
role: m.role as "user" | "assistant" | "system" | undefined,
content:
typeof m.content === "string"
? m.content
: JSON.stringify(m.content),
content: extractTextFromMessageContent(m.content),
name: m.name,
}));
requiredContextSize = encodeChat(
Expand Down Expand Up @@ -2530,7 +2531,7 @@ chat.openapi(completions, async (c) => {
// Convert messages to the format expected by gpt-tokenizer
const chatMessages: any[] = messages.map((m) => ({
role: m.role as "user" | "assistant" | "system" | undefined,
content: m.content || "",
content: extractTextFromMessageContent(m.content) || "",
name: m.name,
}));
calculatedPromptTokens = encodeChat(
Expand Down
9 changes: 6 additions & 3 deletions apps/gateway/src/chat/tools/calculate-prompt-tokens.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
import { encodeChat } from "gpt-tokenizer";

import { type ChatMessage, DEFAULT_TOKENIZER_MODEL } from "./types.js";
import {
type ChatMessage,
DEFAULT_TOKENIZER_MODEL,
extractTextFromMessageContent,
} from "./types.js";

/**
* Transforms streaming chunk to OpenAI format for non-OpenAI providers
Expand All @@ -10,8 +14,7 @@ export function calculatePromptTokensFromMessages(messages: any[]): number {
try {
const chatMessages: ChatMessage[] = messages.map((m: any) => ({
role: m.role,
content:
typeof m.content === "string" ? m.content : JSON.stringify(m.content),
content: extractTextFromMessageContent(m.content),
name: m.name,
}));
return encodeChat(chatMessages, DEFAULT_TOKENIZER_MODEL).length;
Expand Down
11 changes: 6 additions & 5 deletions apps/gateway/src/chat/tools/estimate-tokens.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,11 @@ import { encode, encodeChat } from "gpt-tokenizer";

import { logger } from "@llmgateway/logger";

import { type ChatMessage, DEFAULT_TOKENIZER_MODEL } from "./types.js";
import {
type ChatMessage,
DEFAULT_TOKENIZER_MODEL,
extractTextFromMessageContent,
} from "./types.js";

import type { Provider } from "@llmgateway/models";

Expand All @@ -27,10 +31,7 @@ export function estimateTokens(
// Convert messages to the format expected by gpt-tokenizer
const chatMessages: ChatMessage[] = messages.map((m) => ({
role: m.role,
content:
typeof m.content === "string"
? m.content
: JSON.stringify(m.content),
content: extractTextFromMessageContent(m.content),
name: m.name,
}));
calculatedPromptTokens = encodeChat(
Expand Down
20 changes: 20 additions & 0 deletions apps/gateway/src/chat/tools/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,26 @@ export interface ChatMessage {
name?: string;
}

/**
* Extracts text content from a message content field, handling both string and array formats
* This function is necessary because BaseMessage.content can be string | MessageContent[]
* but gpt-tokenizer expects only strings
*/
export function extractTextFromMessageContent(content: string | any[]): string {
if (typeof content === "string") {
return content;
}

if (Array.isArray(content)) {
return content
.filter((part: any) => part.type === "text")
.map((part: any) => part.text || "")
.join(" ");
}

return "";
}
Comment on lines +15 to +28
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Replace any with precise content-part types; tighten signature and add safe narrowing

Project guideline forbids any. Define a typed union for message parts and use a type guard. Also handle null/undefined and normalize spacing.

 export const DEFAULT_TOKENIZER_MODEL = "gpt-4";
 
 // Define ChatMessage type to match what gpt-tokenizer expects
 export interface ChatMessage {
   role: "user" | "system" | "assistant" | undefined;
   content: string;
   name?: string;
 }
 
+// Message content parts (mirror zod schema in chat.ts)
+export interface TextPart {
+  type: "text";
+  text: string;
+}
+export interface ImageUrlPart {
+  type: "image_url";
+  image_url: { url: string; detail?: "low" | "high" | "auto" };
+}
+export type MessageContentPart = TextPart | ImageUrlPart;
+
 /**
  * Extracts text content from a message content field, handling both string and array formats
  * This function is necessary because BaseMessage.content can be string | MessageContent[]
  * but gpt-tokenizer expects only strings
  */
-export function extractTextFromMessageContent(content: string | any[]): string {
-	if (typeof content === "string") {
-		return content;
-	}
-
-	if (Array.isArray(content)) {
-		return content
-			.filter((part: any) => part.type === "text")
-			.map((part: any) => part.text || "")
-			.join(" ");
-	}
-
-	return "";
-}
+export function extractTextFromMessageContent(
+  content: string | MessageContentPart[] | null | undefined,
+): string {
+  if (typeof content === "string") return content;
+  if (Array.isArray(content)) {
+    return content
+      .filter((part): part is TextPart => part.type === "text" && typeof (part as TextPart).text === "string")
+      .map((part) => part.text)
+      .join(" ")
+      .replace(/\s+/g, " ")
+      .trim();
+  }
+  return "";
+}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
export function extractTextFromMessageContent(content: string | any[]): string {
if (typeof content === "string") {
return content;
}
if (Array.isArray(content)) {
return content
.filter((part: any) => part.type === "text")
.map((part: any) => part.text || "")
.join(" ");
}
return "";
}
export interface TextPart {
type: "text";
text: string;
}
export interface ImageUrlPart {
type: "image_url";
image_url: { url: string; detail?: "low" | "high" | "auto" };
}
export type MessageContentPart = TextPart | ImageUrlPart;
export function extractTextFromMessageContent(
content: string | MessageContentPart[] | null | undefined,
): string {
if (typeof content === "string") return content;
if (Array.isArray(content)) {
return content
.filter((part): part is TextPart => part.type === "text" && typeof (part as TextPart).text === "string")
.map((part) => part.text)
.join(" ")
.replace(/\s+/g, " ")
.trim();
}
return "";
}
🤖 Prompt for AI Agents 
In apps/gateway/src/chat/tools/types.ts around lines 15 to 28, replace the use
of `any` and the loose signature by defining a MessagePart union type (e.g.,
TextPart { type: "text"; text?: string | null }, ImagePart | OtherPart as
needed), change the function signature to accept string | MessagePart[] | null |
undefined, add a type guard isTextPart(part): part is TextPart to narrow safely,
filter out null/undefined and only text parts, map to (part.text ?? "") and
normalize whitespace by trimming parts and joining with a single space, and
return an empty string for other inputs; ensure no `any` remains and the types
are exported if reused.


// Define OpenAI-compatible image object type
export interface ImageObject {
type: "image_url";
Expand Down
Loading