An LLM-based administrative assistant that mimics user writing styles and pre-writes email replies for executives, boosting professional and personal productivity by up to 25%
Professionals get hundreds of emails a day:
- Many go unread
- Not organized or prioritized
- Can result in missing important Tax/Federal/Bank Statement Docs
Busy professionals may take a long time to respond to all these emails, and the worry of having to respond can be very stressful. This is further marred by subpar Quality of responses and workstream delays.
Being an LLM-powered administrative assistant, Hedwing handles the following:
- Pre-writes emails for users according to their personal writing styles
- Picks out relevant emails from the past emails of the user
- Gains context around the incoming email from previous responses
- Identifies the relationship between sender and responder to craft responses
- Ensure data security by storing within existing organizational infrastructure
- Concept Overview
- Data Overview
- Personalization Framework
- Prompt Fine-Tuning
- User Interface
- Acknowledgements
The single token of data in our analysis is an email-reply pair. Any email-reply pair can be divided into three elements:
- Global Context
- Local Context
- Pair Style Relationship
Hedwig aims to tap these three concepts using a proprietary personalization framework while using an LLM to reply to any incoming email.
The Enron email dataset was used for project development. This email dataset contains approximately 500,000 emails generated by employees of the Enron Corporation. The Federal Energy Regulatory Commission obtained it while investigating Enron's collapse.
The original data obtained has highly dense JSON files.
For our purpose, we cleaned and transmuted the data to build message-reply pairs using the following rules:
- Emails with the same subject and sender-recipient pair were grouped into threads
- Rather than focussing on an email, an email-reply conversation was emphasized
- Empty emails or emails with just emojis/attachments were removed
~25000 messages reply conversations were identified, which became our modeling dataset
We used close to 250 personal emails to mimic a local testing environment for our tool. This helps refine our prompts further and add human context to our analysis.
For most LLM Models, we suffer from two problems while generating text:
- There's often no single-point source of information
- Information can be out of date
We add Retrieval Augmentation frameworks to our LLM as Personalization elements. That way, the LLM always refers to our primary source data (and does not hallucinate).
The following modular framework embeds personalization elements in each step of your email generation.
Personalization Framework Notebook
Before we even start looking into any incoming email, we can start gaining sufficient knowledge that can help put guardrails for our LLM:
- Who is the sender?
- On who's behalf is the LLM generating the response
- What's the average length of an email that the replier typically sends
These three memory features are re-instated in every step of the Personalization Framework.
To find emails relevant to any incoming query
Every Human is unique
-
Hedwig assigns independent vector universes to each individual user
-
All the emails written by that user in the past are converted to vector embeddings
-
Collections of each user's email embeddings are stored in a vector database (Chroma)
-
Any incoming email is embedded using the same model
-
Maximal Marginal Relevance Search is implemented to find the most relevant emails between the incoming query and the user vector store.
-
MMR Search selects examples based on a combination of which emails are most similar to the sender's email, while also optimizing for diversity.
There can be multi-modal conversations for the same topic
Two components define an email thread:
- Sender-Receiver personnel
- Subject Line
Even with an MMR-backed RAG system, the LLM can often add more information to the email reply than required. Further LLM hallucinations accompany this. To provide guardrails to our output, we update the retriever to emphasize recent emails from the ongoing thread (if any).
Notebook for finding threads.
The latest chronological emails (irrespective of thread) between the user and sender are extracted. An individual LLM Endpoint is utilized to extract writing style adjectives between the two users.
Inspiration was taken from the Chicago Manual of Style The Chicago Manual of Style Online is the venerable, time-tested guide to style, usage, and grammar in an accessible online format
The final text generation chain takes:
- Contextual Email from Level 2
- Writing Style adjectives from Level 3A
- Memory is integrated with each step of the LLM Chain and reinforced in the final prompt
This step gives us our final output.
Rather than fine-tuning our LLM, we emphasized toggling the model parameters and fine-tuning the prompt to extract maximum generative power without GPU cost constraints.
- The base notebook is located here.
- The entire codebase was productized using Streamlit and deployed on HuggingFace Spaces