This service summarizes textual content and persists the summary in a specified database table. It uses langChain, OpenAI's GPT-3.5 model, Supabase, Sentry and AWS lambda.
- Recursive Chunking: Handles long documents by breaking them into smaller chunks, avoiding token limit failures with OpenAI, ensuring that even extensive documents can be summarized without issues.
- Rate Limiting: Utilizes Bottleneck to manage request rates, preserving system stability and adhering to API usage constraints.
- Integration: Uses Supabase for database operations and Sentry for error tracking, ensuring a robust and reliable service.
- Clone the repository.
- Run
npm install
to install dependencies.
- Copy the example environment file:
cp .env.example .env.
- Update the .env file with your credentials (Supabase URL, Supabase anon key, Sentry DSN).
Use the documentSummarizer function to process and summarize documents. It handles chunking and summarization internally, providing a seamless experience.
The handler function is designed for AWS Lambda deployment. It accepts an event object containing the document text, the target database table name, and the record ID for updates.