Skip to content

Commit

Permalink
fix(normalization): stop returning "Sorry, I am unable to normalize t…
Browse files Browse the repository at this point in the history
…he text as it seems to contain random characters and does not appear to be a valid input for normalization."
  • Loading branch information
AmitMY committed Nov 16, 2024
1 parent fde8a5d commit 86f7191
Showing 1 changed file with 7 additions and 2 deletions.
9 changes: 7 additions & 2 deletions functions/src/text-normalization/model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,16 @@ import type OpenAI from 'openai';
import type {ChatCompletionMessageParam} from 'openai/resources/chat/completions';

const SYSTEM_PROMPT = `
You are a proficient assistant, responsible for data sanitization for a machine translation model. Your main task involves operating the \`normalize\` function:
You are a proficient assistant, responsible for data sanitization for a machine translation model.
Your main task involves operating the \`normalize\` function:
Normalizes the given text for sign language translation.
Takes in a text that might contain misspellings, incorrect capitalization,
missing hyphenation, numbers, units, or other phenomena requiring special
vocalization. Returns the text normalized for sign language, with corrections in
capitalization, spelling, and hyphenation, and special vocalizations for
numbers and units.
numbers and units. If the text can not be normalized, the function should return the original text.
Parameters:
- language_code (str): The language in which the text is in.
Expand All @@ -25,6 +26,10 @@ const FEW_SHOTS = [
user: 'normalize("en", "hello")',
assistant: 'Hello',
},
{
user: 'normalize("en", "mjyti7fasodevwftxzdj")',
assistant: 'mjyti7fasodevwftxzdj',
},
{
user: 'normalize("en", "$123")',
assistant: 'One hundred twenty-three dollars',
Expand Down

0 comments on commit 86f7191

Please sign in to comment.