Brevit

Brevit (from "Brevity") - A high-performance, multi-language library ecosystem for semantically compressing and optimizing data before sending it to Large Language Models (LLMs). Dramatically reduce token costs by 40-60% while maintaining data integrity and readability.

What is Brevit?

Brevit is a comprehensive token optimization solution designed to help developers and organizations reduce LLM API costs significantly. Named after the word "brevity" (meaning concise and exact use of words), Brevit transforms verbose data structures into token-efficient formats that LLMs can understand just as well, but at a fraction of the cost.

Why Brevit?

Cost Savings

40-60% Token Reduction: Flatten nested JSON structures into efficient key-value pairs, dramatically reducing token counts
Lower API Costs: Pay less for the same quality LLM responses by sending optimized data
Scalable Savings: The more you use LLMs, the more you save with Brevit

Developer Experience

Multi-Language Support: Consistent API across C#, JavaScript, and Python - use the same patterns everywhere
Zero Learning Curve: Simple, intuitive API that works out of the box
Type-Safe: Built with modern language features for reliability and IDE support
First-Class POCO Support: Optimize C# objects directly without manual serialization

Performance & Reliability

Lightweight: Minimal dependencies, maximum performance
Production-Ready: Battle-tested architecture designed for real-world applications
Extensible Architecture: Plugin system for custom optimizers (LangChain, Semantic Kernel, Azure AI, etc.)
Memory Efficient: Processes data in-place where possible, minimizing memory footprint

Real-World Cost Example

If you're processing 1 million API calls per month:

Without Brevit: ~100 tokens per call × $0.002/1K tokens = $200/month
With Brevit: ~50 tokens per call × $0.002/1K tokens = $100/month
Savings: $100/month (50% reduction) or $1,200/year

Key Features

1. JSON Optimization

Transform complex nested JSON into flat, token-efficient formats:

Flatten Mode: Convert {"user": {"name": "J"}} → user.name: J (saves ~40% tokens)
Tabular Arrays: Automatically detects uniform object arrays and formats them in compact tabular format (saves ~50-60% tokens)
Primitive Arrays: Comma-separated format for arrays of primitives (saves ~70% tokens)
YAML Mode: Convert JSON to YAML for better readability and token efficiency
Filter Mode: Extract only the fields you need, ignoring unnecessary data
Smart Array Handling: Hybrid approach - uses tabular format when possible, falls back to indexed paths for mixed data

Example - Tabular Optimization:

// Before (160 tokens)
{
  "friends": ["ana", "luis", "sam"],
  "hikes": [
    {"id": 1, "name": "Blue Lake Trail", "distanceKm": 7.5, "elevationGain": 320},
    {"id": 2, "name": "Ridge Overlook", "distanceKm": 9.2, "elevationGain": 540},
    {"id": 3, "name": "Wildflower Loop", "distanceKm": 5.1, "elevationGain": 180}
  ]
}

// After (95 tokens - 40% reduction)
friends[3]: ana,luis,sam
hikes[3]{distanceKm,elevationGain,id,name}:
  7.5,320,1,Blue Lake Trail
  9.2,540,2,Ridge Overlook
  5.1,180,3,Wildflower Loop

Example - Nested Objects:

// Before (45 tokens)
{
  "user": {
    "id": "u-123",
    "name": "Javian",
    "contact": {
      "email": "[email protected]",
      "phone": "+1-555-0123"
    }
  }
}

// After (28 tokens - 38% reduction)
user.id: u-123
user.name: Javian
user.contact.email: [email protected]
user.contact.phone: +1-555-0123

2. Text Optimization

Intelligently process long text documents:

Automatic Detection: Automatically identifies long text and applies optimization
Boilerplate Removal: Cleans signatures, headers, and repetitive content
Summarization: Integrate with LLM services to create concise summaries
Configurable Thresholds: Set custom thresholds for when to optimize

3. Image Optimization

Extract and optimize content from images:

OCR Integration: Extract text from receipts, invoices, documents, and more
Metadata Extraction: Get image metadata without processing the entire image
Multi-Provider Support: Works with Azure AI Vision, Tesseract, and custom OCR services
Format Conversion: Convert images to text that LLMs can process efficiently

4. Smart Data Handling

Type Detection: Automatically detects JSON strings, objects, text, and images
POCO Support: Direct optimization of C# objects without manual serialization
Async/Await: Full async support for high-performance, scalable applications
Error Handling: Graceful error handling with informative messages

5. Automatic Strategy Selection (`.brevity()`)

Intelligent Analysis: Automatically analyzes data structure (depth, arrays, complexity)
Optimal Strategy: Selects the best optimization method based on data characteristics
Zero Configuration: Works out of the box without manual configuration
Extensible: Register custom strategies for domain-specific optimizations
Scoring System: Uses a scoring algorithm (0-100) to rank optimization strategies

When Not to Use Brevit

Brevit is designed for optimizing data before sending to LLMs. Consider alternatives when:

Human-Readable Format Required: If you need JSON for human consumption or API responses, use standard JSON
Complex Nested Structures: For deeply nested structures where hierarchy is critical, standard JSON may be better
Small Data Sets: For data under 100 tokens, the optimization overhead may not be worth it
Real-Time APIs: If you're building REST APIs that return JSON to clients, use standard JSON
Data Validation: If you need strict JSON schema validation, use standard JSON with validators

When Brevit Shines:

✅ LLM prompt optimization
✅ Reducing API costs
✅ Processing large datasets for AI
✅ Document summarization workflows
✅ OCR and image processing pipelines

Benchmarks

Token Reduction Benchmarks

Data Type	Original Tokens	Brevit Tokens	Reduction	Example
Simple JSON	45	28	38%	User object with contact info
Complex JSON	234	127	46%	E-commerce order with items
Nested Arrays	156	89	43%	Product catalog
Mixed Data	312	178	43%	API response with metadata

Performance Benchmarks

Operation	Time (ms)	Memory (MB)
Flatten JSON (1KB)	0.5	2.1
Flatten JSON (10KB)	2.3	8.5
Flatten JSON (100KB)	18.7	45.2
Text Clean (1KB)	0.3	1.8
Text Clean (10KB)	1.2	6.3

Benchmarks run on: Intel i7-12700K, 32GB RAM, Node.js 20.x / .NET 8 / Python 3.11

Installation & Quick Start

Choose Your Language

C# (.NET) - For .NET applications
```
dotnet add package Brevit.NET
```
JavaScript - For Node.js and browser applications
```
npm install brevit-js
```
Python - For Python applications
```
pip install brevit-py
```

Quick Start Example

C#:

using Brevit.NET;

var config = new BrevitConfig(JsonMode: JsonOptimizationMode.Flatten);
var brevit = new BrevitClient(config, 
    new DefaultJsonOptimizer(), 
    new DefaultTextOptimizer(), 
    new DefaultImageOptimizer());

// Automatic optimization - analyzes data and selects best strategy
var optimized = await brevit.BrevityAsync(new { user = new { name = "John" } });

// Or use explicit optimization
var explicit = await brevit.OptimizeAsync(new { user = new { name = "John" } });

JavaScript:

import { BrevitClient, BrevitConfig, JsonOptimizationMode } from 'brevit-js';

const brevit = new BrevitClient(new BrevitConfig({ 
  jsonMode: JsonOptimizationMode.Flatten 
}));

// Automatic optimization - analyzes data and selects best strategy
const optimized = await brevit.brevity({ user: { name: 'John' } });

// Or use explicit optimization
const explicit = await brevit.optimize({ user: { name: 'John' } });

Python:

from brevit import BrevitClient, BrevitConfig, JsonOptimizationMode

brevit = BrevitClient(BrevitConfig(json_mode=JsonOptimizationMode.Flatten))

# Automatic optimization - analyzes data and selects best strategy
optimized = await brevit.brevity({"user": {"name": "John"}})

# Or use explicit optimization
explicit = await brevit.optimize({"user": {"name": "John"}})

Automatic Optimization with `.brevity()`

The .brevity() method intelligently analyzes your data structure and automatically selects the best optimization strategy. No configuration needed!

Example - Automatic Tabular Detection:

const data = {
  friends: ["ana", "luis", "sam"],
  hikes: [
    {id: 1, name: "Blue Lake Trail", distanceKm: 7.5},
    {id: 2, name: "Ridge Overlook", distanceKm: 9.2}
  ]
};

// Automatically detects uniform arrays and applies tabular format
const optimized = await brevit.brevity(data);
// Output: Automatically uses tabular format for hikes array

Example - Automatic Text Detection:

long_text = "This is a very long document..." * 100

# Automatically detects long text and applies summarization
optimized = await brevit.brevity(long_text)
# Output: Automatically summarizes if text exceeds threshold

Example - Extensibility:

// Register custom optimization strategy
brevit.registerStrategy('custom', (data, analysis) => {
  // Custom analysis logic
  if (analysis.hasSpecialPattern) {
    return { score: 95, reason: 'Custom optimization needed' };
  }
  return { score: 0 };
}, async (data) => {
  // Custom optimization logic
  return customOptimizedData;
});

Playgrounds

Try Brevit online:

Web Playground: https://brevit.dev/playground (Coming Soon)
VS Code Extension: Install "Brevit" extension for inline optimization
Online Demo: https://brevit.dev/demo (Coming Soon)

Local Playground

Run the interactive playground locally:

# Clone the repository
git clone https://github.com/JavianDev/Brevit.git
cd Brevit

# Run playground (Node.js)
cd Brevit.js
npm install
node playground.js

# Or use Python
cd Brevit.py
pip install -e .
python playground.py

CLI

Brevit includes command-line tools for quick optimization:

Installation

# npm
npm install -g brevit-cli

# Python
pip install brevit-cli

# .NET
dotnet tool install -g Brevit.CLI

Usage

# Optimize a JSON file
brevit optimize input.json -o output.txt

# Optimize from stdin
cat data.json | brevit optimize

# Optimize with custom config
brevit optimize input.json --mode flatten --threshold 1000

# Help
brevit --help

Examples

# Flatten JSON file
brevit optimize order.json --mode flatten

# Convert to YAML
brevit optimize data.json --mode yaml

# Filter specific paths
brevit optimize data.json --mode filter --paths "user.name,order.id"

Format Overview

Flattened Format (Hybrid Optimization)

Brevit's default format intelligently converts nested structures to flat key-value pairs with automatic tabular optimization:

Input:

{
  "order": {
    "id": "o-456",
    "items": [
      {"sku": "A-88", "quantity": 1},
      {"sku": "T-22", "quantity": 2}
    ]
  }
}

Output (with tabular optimization):

order.id: o-456
order.items[2]{quantity,sku}:
  1,A-88
  2,T-22

For non-uniform arrays (fallback):

{
  "items": [
    {"sku": "A-88", "quantity": 1},
    "special-item",
    {"sku": "T-22", "quantity": 2}
  ]
}

Output (fallback to indexed format):

items[0].sku: A-88
items[0].quantity: 1
items[1]: special-item
items[2].sku: T-22
items[2].quantity: 2

Key Features of Flattened Format

Dot Notation: Nested objects use dot notation (user.contact.email)
Tabular Arrays: Uniform object arrays use compact tabular format (items[3]{field1,field2}:)
Primitive Arrays: Comma-separated format (friends[3]: ana,luis,sam)
Hybrid Approach: Automatically detects optimal format, falls back to indexed format for mixed data
No Syntactic Overhead: Minimal braces, brackets, or commas - just keys and values
LLM-Friendly: Easy for LLMs to parse and understand
Token Efficient: 40-60% token reduction compared to JSON

API

Core Classes

BrevitClient

The main client for optimization operations.

Methods:

brevity(rawData, intent?) - NEW! Automatically analyzes data and selects optimal strategy
optimize(rawData, intent?) - Optimize any data type with explicit configuration
optimizeAsync(rawData, intent?) - Async version (C#)
BrevityAsync(rawData, intent?) - Async automatic optimization (C#)
registerStrategy(name, analyzer, optimizer) - Register custom optimization strategies

Parameters:

rawData: Object, string, or byte array to optimize
intent: Optional hint about optimization goal

Returns: Optimized string

When to Use:

.brevity(): Use when you want automatic strategy selection based on data analysis
.optimize(): Use when you want explicit control over optimization mode

BrevitConfig

Configuration object for BrevitClient.

Properties:

jsonMode: JsonOptimizationMode (Flatten, ToYaml, Filter, None)
textMode: TextOptimizationMode (Clean, SummarizeFast, SummarizeHighQuality, None)
imageMode: ImageOptimizationMode (Ocr, Metadata, None)
jsonPathsToKeep: string[] - Paths to keep in Filter mode
longTextThreshold: number - Character threshold for text optimization

Enums

JsonOptimizationMode

None - No optimization
Flatten - Flatten to key-value pairs (default)
ToYaml - Convert to YAML format
Filter - Keep only specified paths

TextOptimizationMode

None - No optimization
Clean - Remove boilerplate
SummarizeFast - Fast summarization
SummarizeHighQuality - High-quality summarization

ImageOptimizationMode

None - Skip processing
Ocr - Extract text via OCR
Metadata - Extract metadata only

Interfaces

IJsonOptimizer

Task<string> OptimizeJsonAsync(string jsonString, BrevitConfig config);

ITextOptimizer

Task<string> OptimizeTextAsync(string longText, BrevitConfig config);

IImageOptimizer

Task<string> OptimizeImageAsync(byte[] imageData, BrevitConfig config);

Using Brevit in LLM Prompts

Best Practices

Context First: Always provide context before optimized data
Clear Instructions: Tell the LLM what format to expect
Examples: Include examples of the format in your prompt

Example Prompt Template

You are analyzing order data. The data is in Brevit flattened format:

Context:
{optimized_data}

Task: Extract the order total and shipping address.

Format your response as JSON with keys: total, address

Real-World Example

from brevit import BrevitClient, BrevitConfig, JsonOptimizationMode

brevit = BrevitClient(BrevitConfig(json_mode=JsonOptimizationMode.Flatten))
order_data = {
    "orderId": "o-456",
    "total": 99.99,
    "items": [{"name": "Product A", "price": 49.99}]
}

optimized = await brevit.optimize(order_data)

prompt = f"""Analyze this order data:

{optimized}

Questions:
1. What is the order total?
2. How many items are in the order?
3. What is the average item price?

Respond in JSON format."""

Syntax Cheatsheet

Flattened Format Syntax

JSON Structure	Brevit Format	Example
Object property	`key.value`	`user.name: John`
Nested object	`key.nested.value`	`user.contact.email: [email protected]`
Primitive array	`key[count]: val1,val2,val3`	`friends[3]: ana,luis,sam`
Uniform object array	`key[count]{field1,field2}:` `val1,val2` `val3,val4`	`items[2]{sku,qty}:` `A-88,1` `T-22,2`
Array element (fallback)	`key[index].value`	`items[0].name: Product A`
Nested array	`key[index].nested[index]`	`orders[0].items[1].sku: A-88`
Root value	`value: content`	`value: Hello World`
Null value	`key: null`	`phone: null`
Boolean	`key: true`	`isActive: true`
Number	`key: 123`	`quantity: 5`

Special Cases

Empty Objects: key: {} → key: {}
Empty Arrays: key: [] → key: []
Nested Empty: user.metadata: {} → user.metadata: {}
Mixed Types: Arrays can contain objects, primitives, or mixed (falls back to indexed format)
Tabular Arrays: Automatically detected when all objects have same keys
Primitive Arrays: Automatically detected when all elements are primitives

Other Implementations

Brevit is available in multiple languages:

Language	Package	Repository	Status
C# (.NET)	`Brevit.NET`	brevit-net	✅ Stable
JavaScript	`brevit-js`	brevit-js	✅ Stable
Python	`brevit-py`	brevit-py	✅ Stable

Community Implementations

Have you created a Brevit implementation in another language? Submit a PR to add it here!

Full Specification

Format Specification

The Brevit flattened format follows these rules:

Key-Value Pairs: Each line contains exactly one key-value pair (or tabular header)
Separator: Key and value are separated by : (colon + space)
Key Format: Keys use dot notation for nesting and bracket notation for arrays
Value Format: Values are strings, numbers, booleans, or null
Tabular Arrays: Uniform object arrays use compact tabular format with header
Primitive Arrays: Arrays of primitives use comma-separated format
Line Endings: Lines are separated by newline characters (\n)

Grammar

brevit := line*
line := key ": " value "\n" | tabular_header "\n" tabular_rows
key := identifier ("." identifier | "[" number "]")*
value := string | number | boolean | null
tabular_header := key "[" number "]{" field_list "}:"
tabular_rows := ("  " row "\n")+
row := value ("," value)*
field_list := identifier ("," identifier)*
identifier := [a-zA-Z_][a-zA-Z0-9_]*
number := [0-9]+

Examples

Simple Object:

user.name: John
user.age: 30

Nested Object:

user.contact.email: [email protected]
user.contact.phone: +1-555-0123

Primitive Array:

friends[3]: ana,luis,sam

Tabular Array (Uniform Objects):

items[2]{sku,quantity,price}:
  A-88,1,29.99
  T-22,2,39.99

Array (Fallback for Mixed/Non-Uniform):

items[0].name: Product A
items[0].price: 29.99
items[1]: special-item
items[2].name: Product B
items[2].price: 39.99

Complex Structure:

order.id: o-456
order.customer.name: John Doe
order.items[2]{quantity,sku}:
  2,A-88
  1,T-22
order.shipping.address.street: 123 Main St
order.shipping.address.city: Toronto

Libraries

Brevit.NET (C#)

A high-performance, type-safe .NET library built with .NET 8.

Features:

First-class POCO support
Dependency Injection ready
Full async/await support
Extensible optimizer interfaces

Quick Start:

dotnet add package Brevit.NET

Brevit.js (JavaScript)

A lightweight JavaScript library for Node.js and browsers.

Features:

Zero dependencies (core)
Universal module support (ESM/CommonJS)
Works in browsers and Node.js
Optional YAML support
Full TypeScript definitions

Quick Start:

npm install brevit-js

Brevit.py (Python)

A modern Python library with full async/await support.

Features:

Type hints throughout
Async/await patterns
LangChain integration ready
FastAPI/Flask compatible

Quick Start:

pip install brevit-py

Use Cases

E-Commerce Platforms: Optimize product catalogs, order data, and customer information before sending to LLMs for recommendations
Document Processing: Process legal documents, contracts, and reports efficiently
Customer Support: Optimize support tickets, chat logs, and customer data for AI-powered support systems
Financial Services: Process invoices, receipts, and financial documents via OCR and optimization
Content Management: Optimize articles, blog posts, and content before AI analysis
API Integrations: Optimize API responses before sending to LLM-powered integrations
Data Analytics: Prepare data for LLM-based analytics and insights

Performance

Token Reduction: 40-60% reduction in token count
Memory Efficient: Processes data in-place where possible
Fast: Optimized algorithms for minimal overhead
Scalable: Async/await patterns for high concurrency

Architecture

All Brevit libraries follow a consistent architecture:

BrevitClient
├── BrevitConfig (Configuration)
├── IJsonOptimizer (JSON optimization)
├── ITextOptimizer (Text optimization)
└── IImageOptimizer (Image optimization)

Contributing

We welcome contributions! Each library has its own contributing guidelines:

License

MIT License - see LICENSE file for details.

Support

Documentation: https://brevit.dev/docs
Issues: https://github.com/JavianDev/Brevit/issues
Email: [email protected]

Roadmap

Version History

0.1.0 (Current): Initial release with core optimization features

Created by Javian - Optimizing LLM interactions, one token at a time.

Name		Name	Last commit message	Last commit date
Latest commit History 11 Commits
.github/workflows		.github/workflows
Brevit.NET.Tests		Brevit.NET.Tests
Brevit.NET		Brevit.NET
Brevit.js		Brevit.js
Brevit.py		Brevit.py
.gitignore		.gitignore
BUILD_STATUS.md		BUILD_STATUS.md
DEPLOYMENT.md		DEPLOYMENT.md
DEPLOY_PACKAGES.md		DEPLOY_PACKAGES.md
GITHUB_SECRETS_SETUP.md		GITHUB_SECRETS_SETUP.md
LICENSE		LICENSE
README.md		README.md

License

JavianDev/Brevit

Folders and files

Latest commit

History

Repository files navigation

Brevit

Table of Contents

What is Brevit?

Why Brevit?

Cost Savings

Developer Experience

Performance & Reliability

Real-World Cost Example

Key Features

1. JSON Optimization

2. Text Optimization

3. Image Optimization

4. Smart Data Handling

5. Automatic Strategy Selection (.brevity())

When Not to Use Brevit

Benchmarks

Token Reduction Benchmarks

Performance Benchmarks

Installation & Quick Start

Choose Your Language

Quick Start Example

Automatic Optimization with .brevity()

Playgrounds

Local Playground

CLI

Installation

Usage

Examples

Format Overview

Flattened Format (Hybrid Optimization)

Key Features of Flattened Format

API

Core Classes

BrevitClient

BrevitConfig

Enums

JsonOptimizationMode

TextOptimizationMode

ImageOptimizationMode

Interfaces

IJsonOptimizer

ITextOptimizer

IImageOptimizer

Using Brevit in LLM Prompts

Best Practices

Example Prompt Template

Real-World Example

Syntax Cheatsheet

Flattened Format Syntax

Special Cases

Other Implementations

Community Implementations

Full Specification

Format Specification

Grammar

Examples

Libraries

Brevit.NET (C#)

Brevit.js (JavaScript)

Brevit.py (Python)

Use Cases

Performance

Architecture

Contributing

License

Support

Roadmap

Version History

About

Resources

License

Uh oh!

Stars

Watchers

5. Automatic Strategy Selection (`.brevity()`)

Automatic Optimization with `.brevity()`

Packages