Sweep: When calling converse(), use the top 5 reflection memories

[ykhli]

Details

the bug may be in conversation.ts

Checklist

• convex/conversation.ts

• Import the filterMemoriesType function from ./lib/memory at the top of the file.
  • Inside the converse() function, after the line where conversationMemories is defined, add a new line to define reflectionMemories. Call filterMemoriesType() with 'reflection' as the memory type and memories as the second argument. Use the slice() method to limit the result to the top 5 memories.
  • Modify the prefixPrompt variable to include the reflectionMemories. After the line where relevantMemories is defined, add a new line to define relevantReflections. Map over the reflectionMemories and join their descriptions with a newline character. Include relevantReflections in the prefixPrompt string.

[sweep-ai]

Here's the PR! #6.

⚡ Sweep Free Trial: I used GPT-4 to create this ticket. You have 3 GPT-4 tickets left for the month and 0 for the day. For more GPT-4 tickets, visit our payment portal. To retrigger Sweep edit the issue.

---

Step 1: 🔍 Code Search

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I looked at (click to expand). If some file is missing from here, you can mention the path in the ticket description.

cat-town/convex/conversation.ts

Lines 1 to 186 in d76e3ca

	import { Id } from './_generated/dataModel';
	import { ActionCtx } from './_generated/server';
	import { fetchEmbeddingWithCache } from './lib/cached_llm';
	import { MemoryDB, filterMemoriesType } from './lib/memory';
	import { LLMMessage, chatCompletion, fetchEmbedding } from './lib/openai';
	import { Message } from './schema';
	type Player = { id: Id<'players'>; name: string; identity: string };
	type Relation = Player & { relationship: string };
	export async function startConversation(
	ctx: ActionCtx,
	audience: Player[],
	memory: MemoryDB,
	player: Player,
	) {
	const newFriendsNames = audience.map((p) => p.name);
	const { embedding } = await fetchEmbeddingWithCache(
	ctx,
	`What do you think about ${newFriendsNames.join(',')}?`,
	{ write: true },
	);
	const memories = await memory.accessMemories(player.id, embedding);
	const convoMemories = filterMemoriesType(['conversation'], memories);
	const animal = player.name === 'Tilly' ? 'dog' : 'cat';
	const prompt: LLMMessage[] = [
	{
	role: 'user',
	content: `You are a ${animal} whose name is ${player.name}. Talk as if you are a ${animal}. Reference information provided to you in the conversation`,
	},
	{
	role: 'user',
	content:
	`You just saw ${newFriendsNames}. You should greet them and start a conversation with them. Below are some of your memories about ${newFriendsNames}:` +
	//relationships.map((r) => r.relationship).join('\n') +
	convoMemories.map((r) => r.memory.description).join('\n') +
	`\n${player.name}:`,
	},
	];
	const stop = newFriendsNames.map((name) => name + ':');
	const { content } = await chatCompletion({ messages: prompt, max_tokens: 300, stop });
	return { content, memoryIds: memories.map((m) => m.memory._id) };
	}
	function messageContent(m: Message): string {
	switch (m.type) {
	case 'started':
	return `${m.fromName} started the conversation.`;
	case 'left':
	return `${m.fromName} left the conversation.`;
	case 'responded':
	return `${m.fromName} to ${m.toNames.join(',')}: ${m.content}\n`;
	}
	}
	export function chatHistoryFromMessages(messages: Message[]): LLMMessage[] {
	return (
	messages
	// For now, just use the message content.
	// However, we could give it context on who started / left the convo
	.filter((m) => m.type === 'responded')
	.map((m) => ({
	role: 'user',
	content: messageContent(m),
	}))
	);
	}
	export async function decideWhoSpeaksNext(
	players: Player[],
	chatHistory: LLMMessage[],
	): Promise<Player> {
	if (players.length === 1) {
	return players[0];
	}
	const promptStr = `[no prose]\n [Output only JSON]
	${JSON.stringify(players)}
	Here is a list of people in the conversation, return BOTH name and ID of the person who should speak next based on the chat history provided below.
	Return in JSON format, example: {"name": "Alex", id: "1234"}
	${chatHistory.map((m) => m.content).join('\n')}`;
	const prompt: LLMMessage[] = [
	{
	role: 'user',
	content: promptStr,
	},
	];
	const { content } = await chatCompletion({ messages: prompt, max_tokens: 300 });
	let speakerId: string;
	try {
	speakerId = JSON.parse(content).id;
	} catch (e) {
	console.error('error parsing speakerId: ', e);
	}
	const randomIdx = Math.floor(Math.random() * players.length);
	return players.find((p) => p.id.toString() === speakerId) || players[randomIdx];
	}
	export async function converse(
	ctx: ActionCtx,
	messages: LLMMessage[],
	player: Player,
	nearbyPlayers: Player[],
	memory: MemoryDB,
	) {
	const nearbyPlayersNames = nearbyPlayers.join(', ');
	const lastMessage: string | null | undefined = messages?.at(-1)?.content;
	const { embedding } = await fetchEmbedding(lastMessage ? lastMessage : '');
	const memories = await memory.accessMemories(player.id, embedding);
	const conversationMemories = filterMemoriesType(['conversation'], memories);
	const reflectionMemories = filterMemoriesType(['reflection'], memories);
	const lastConversationTs = conversationMemories[0]?.memory._creationTime;
	const stop = nearbyPlayers.join(':');
	const relevantReflections: string =
	reflectionMemories.length > 0
	? reflectionMemories
	.slice(0, 2)
	.map((r) => r.memory.description)
	.join('\n')
	: '';
	const relevantMemories: string = conversationMemories
	.slice(0, 2) // only use the first 2 memories
	.map((r) => r.memory.description)
	.join('\n');
	const animal = player.name === 'Tilly' ? 'dog' : 'cat';
	let prefixPrompt = `You are a ${animal} whose name is ${player.name}. Talk as if you are a ${animal}.
	DO NOT use complex phrases humans would use.`;
	prefixPrompt += ` About you: ${player.identity}.\n`;
	if (relevantReflections.length > 0) {
	prefixPrompt += relevantReflections;
	console.log('relevantReflections', relevantReflections);
	}
	prefixPrompt += `\nYou are talking to ${nearbyPlayersNames}, below are something about them: `;
	nearbyPlayers.forEach((p) => {
	prefixPrompt += `\nAbout ${p.name}: ${p.identity}\n`;
	});
	prefixPrompt += `Last time you chatted with some of ${nearbyPlayersNames} it was ${lastConversationTs}. It's now ${Date.now()}. You can cut this conversation short if you talked to this group of people within the last day. \n}`;
	prefixPrompt += `Below are relevant memories to this conversation you are having right now: ${relevantMemories}\n`;
	prefixPrompt +=
	'Below are the current chat history between you and the other folks mentioned above. DO NOT greet the other people more than once. Only greet ONCE. Do not use the word Hey too often. Response should be brief and within 200 characters: \n';
	const prompt: LLMMessage[] = [
	{
	role: 'user',
	content: prefixPrompt,
	},
	...messages,
	{
	role: 'user',
	content: `${player.name}:`,
	},
	];
	const { content } = await chatCompletion({ messages: prompt, max_tokens: 300, stop });
	// console.debug('converse result through chatgpt: ', content);
	return { content, memoryIds: memories.map((m) => m.memory._id) };
	}
	export async function walkAway(messages: LLMMessage[], player: Player): Promise<boolean> {
	const prompt: LLMMessage[] = [
	{
	role: 'user',
	content: `Below is a chat history among a few people who ran into each other. You are ${player.name}. You want to conclude this conversation when you think it's time to go.
	Return 1 if you want to walk away from the conversation and 0 if you want to continue to chat.`,
	},
	...messages,
	];
	const { content: description } = await chatCompletion({
	messages: prompt,
	max_tokens: 1,
	temperature: 0,
	});
	return description === '1';
	}

cat-town/convex/agent.ts

Lines 100 to 283 in d76e3ca

	// playerById.delete(nearbyPlayers[0].id);
	// otherwise, do more than 1:1 conversations by adding them all:
	groups.push([player, ...nearbyPlayers]);
	for (const nearbyPlayer of nearbyPlayers) {
	playerById.delete(nearbyPlayer.id);
	}
	} else {
	solos.push(player);
	}
	}
	return { groups, solos };
	}
	async function handleAgentSolo(ctx: ActionCtx, player: Player, memory: MemoryDB, done: DoneFn) {
	// console.debug('handleAgentSolo: ', player.name, player.id);
	// Handle new observations: it can look at the agent's lastWakeTs for a delta.
	// Calculate scores
	// Run reflection on memories once in a while
	await memory.reflectOnMemories(player.id, player.name);
	// Future: Store observations about seeing players in conversation
	// might include new observations -> add to memory with openai embeddings
	// Later: handle object ownership?
	// Based on plan and observations, determine next action:
	// if so, add new memory for new plan, and return new action
	const walk = player.motion.type === 'stopped' || player.motion.targetEndTs < Date.now();
	// Ignore everyone we last said something to.
	const ignore =
	player.motion.type === 'walking' ? player.motion.ignore : player.lastChat?.message.to ?? [];
	await done(player.agentId, { type: walk ? 'walk' : 'continue', ignore });
	}
	export async function handleAgentInteraction(
	ctx: ActionCtx,
	players: Player[],
	memory: MemoryDB,
	done: DoneFn,
	) {
	// TODO: pick a better conversation starter
	const leader = players[0];
	for (const player of players) {
	const imWalkingHere =
	player.motion.type === 'walking' && player.motion.targetEndTs > Date.now();
	// Get players to walk together and face each other
	if (player.agentId) {
	if (player === leader) {
	if (imWalkingHere) {
	await ctx.runMutation(internal.journal.stop, {
	playerId: player.id,
	});
	}
	} else {
	await ctx.runMutation(internal.journal.walk, {
	agentId: player.agentId,
	target: leader.id,
	ignore: players.map((p) => p.id),
	});
	// TODO: collect collisions and pass them into the engine to wake up
	// other players to avoid these ones in conversation.
	}
	}
	}
	const conversationId = await ctx.runMutation(internal.journal.makeConversation, {
	playerId: leader.id,
	audience: players.slice(1).map((p) => p.id),
	});
	const playerById = new Map(players.map((p) => [p.id, p]));
	const relations = await ctx.runQuery(internal.journal.getRelationships, {
	playerIds: players.map((p) => p.id),
	});
	const relationshipsByPlayerId = new Map(
	relations.map(({ playerId, relations }) => [
	playerId,
	relations.map((r) => ({ ...playerById.get(playerId)!, relationship: r.relationship })),
	]),
	);
	const messages: Message[] = [];
	// TODO: real logic. this just sends one message each!
	const endAfterTs = Date.now() + CONVERSATION_TIME_LIMIT;
	// Choose who should speak next:
	let endConversation = false;
	let lastSpeakerId = leader.id;
	let remainingPlayers = players;
	while (!endConversation) {
	// leader speaks first
	const chatHistory = chatHistoryFromMessages(messages);
	const speaker =
	messages.length === 0
	? leader
	: await decideWhoSpeaksNext(
	remainingPlayers.filter((p) => p.id !== lastSpeakerId),
	chatHistory,
	);
	lastSpeakerId = speaker.id;
	const audiencePlayers = players.filter((p) => p.id !== speaker.id);
	const audience = players.filter((p) => p.id !== speaker.id).map((p) => p.id);
	const shouldWalkAway = audience.length === 0 || (await walkAway(chatHistory, speaker));
	// Decide if we keep talking.
	if (shouldWalkAway || Date.now() > endAfterTs) {
	// It's to chatty here, let's go somewhere else.
	await ctx.runMutation(internal.journal.leaveConversation, {
	playerId: speaker.id,
	audience,
	conversationId,
	});
	// Update remaining players
	remainingPlayers = remainingPlayers.filter((p) => p.id !== speaker.id);
	// End the interaction if there's no one left to talk to.
	endConversation = audience.length === 0;
	// TODO: remove this player from the audience list
	break;
	}
	// TODO - playerRelations is not used today because of https://github.com/a16z-infra/ai-town/issues/56
	const playerRelations = relationshipsByPlayerId.get(speaker.id) ?? [];
	let playerCompletion;
	if (messages.length === 0) {
	playerCompletion = await startConversation(ctx, audiencePlayers, memory, speaker);
	} else {
	// TODO: stream the response and write to the mutation for every sentence.
	playerCompletion = await converse(ctx, chatHistory, speaker, audiencePlayers, memory);
	}
	const message = await ctx.runMutation(internal.journal.talk, {
	playerId: speaker.id,
	audience,
	content: playerCompletion.content,
	relatedMemoryIds: playerCompletion.memoryIds,
	conversationId,
	});
	if (message) {
	messages.push(message);
	}
	}
	if (messages.length > 0) {
	for (const player of players) {
	await memory.rememberConversation(player.name, player.id, player.identity, conversationId);
	await done(player.agentId, { type: 'walk', ignore: players.map((p) => p.id) });
	}
	}
	}
	type DoneFn = (
	agentId: Id<'agents'> | undefined,
	activity:
	| { type: 'walk'; ignore: Id<'players'>[] }
	| { type: 'continue'; ignore: Id<'players'>[] },
	) => Promise<void>;
	function handleDone(ctx: ActionCtx, noSchedule?: boolean): DoneFn {
	const doIt: DoneFn = async (agentId, activity) => {
	// console.debug('handleDone: ', agentId, activity);
	if (!agentId) return;
	let walkResult;
	switch (activity.type) {
	case 'walk':
	walkResult = await ctx.runMutation(internal.journal.walk, {
	agentId,
	ignore: activity.ignore,
	});
	break;
	case 'continue':
	walkResult = await ctx.runQuery(internal.journal.nextCollision, {
	agentId,
	ignore: activity.ignore,
	});
	break;
	default:
	const _exhaustiveCheck: never = activity;
	throw new Error(`Unhandled activity: ${JSON.stringify(activity)}`);
	}
	await ctx.runMutation(internal.engine.agentDone, {
	agentId,
	otherAgentIds: walkResult.nextCollision?.agentIds,
	wakeTs: walkResult.nextCollision?.ts ?? walkResult.targetEndTs,

cat-town/convex/lib/memory.ts

Lines 1 to 314 in d76e3ca

	import { Infer, v } from 'convex/values';
	import { internal } from '../_generated/api.js';
	import { Doc, Id } from '../_generated/dataModel.js';
	import {
	ActionCtx,
	DatabaseReader,
	internalMutation,
	internalQuery,
	} from '../_generated/server.js';
	import { asyncMap } from './utils.js';
	import { EntryOfType, Memories, Memory, MemoryOfType, MemoryType } from '../schema.js';
	import { chatCompletion } from './openai.js';
	import { clientMessageMapper } from '../chat.js';
	import { pineconeAvailable, queryVectors, upsertVectors } from './pinecone.js';
	import { chatHistoryFromMessages } from '../conversation.js';
	import { MEMORY_ACCESS_THROTTLE } from '../config.js';
	import { fetchEmbeddingBatchWithCache } from './cached_llm.js';
	const { embeddingId: _, lastAccess, ...MemoryWithoutEmbeddingId } = Memories.fields;
	const NewMemory = { ...MemoryWithoutEmbeddingId, importance: v.optional(v.number()) };
	const NewMemoryWithEmbedding = { ...MemoryWithoutEmbeddingId, embedding: v.array(v.number()) };
	const NewMemoryObject = v.object(NewMemory);
	type NewMemory = Infer<typeof NewMemoryObject>;
	export interface MemoryDB {
	search(
	playerId: Id<'players'>,
	vector: number[],
	limit?: number,
	): Promise<{ memory: Doc<'memories'>; score: number }[]>;
	accessMemories(
	playerId: Id<'players'>,
	queryEmbedding: number[],
	count?: number,
	): Promise<{ memory: Doc<'memories'>; overallScore: number }[]>;
	addMemories(memories: NewMemory[]): Promise<void>;
	rememberConversation(
	playerName: string,
	playerId: Id<'players'>,
	playerIdentity: string,
	conversationId: Id<'conversations'>,
	): Promise<boolean>;
	reflectOnMemories(playerId: Id<'players'>, name: string): Promise<void>;
	}
	export function MemoryDB(ctx: ActionCtx): MemoryDB {
	if (!pineconeAvailable()) {
	throw new Error('Pinecone environment variables not set. See the README.');
	}
	// If Pinecone env variables are defined, use that.
	const vectorSearch = async (embedding: number[], playerId: Id<'players'>, limit: number) =>
	queryVectors('embeddings', embedding, { playerId }, limit);
	const externalEmbeddingStore = async (
	embeddings: { id: Id<'embeddings'>; values: number[]; metadata: object }[],
	) => upsertVectors('embeddings', embeddings);
	return {
	// Finds memories but doesn't mark them as accessed.
	async search(playerId, queryEmbedding, limit = 100) {
	const results = await vectorSearch(queryEmbedding, playerId, limit);
	const embeddingIds = results.map((r) => r._id);
	const memories = await ctx.runQuery(internal.lib.memory.getMemories, {
	playerId,
	embeddingIds,
	});
	return results.map(({ score }, idx) => ({ memory: memories[idx], score }));
	},
	async accessMemories(playerId, queryEmbedding, count = 10) {
	const results = await vectorSearch(queryEmbedding, playerId, 10 * count);
	return await ctx.runMutation(internal.lib.memory.accessMemories, {
	playerId,
	candidates: results,
	count,
	});
	},
	async addMemories(memoriesWithoutEmbedding) {
	const texts = memoriesWithoutEmbedding.map((memory) => memory.description);
	const { embeddings } = await fetchEmbeddingBatchWithCache(ctx, texts);
	// NB: The cache gets populated by addMemories, so no need to do it here.
	const memories = await asyncMap(memoriesWithoutEmbedding, async (memory, idx) => {
	const embedding = embeddings[idx];
	if (memory.importance === undefined) {
	// TODO: make a better prompt based on the user's memories
	const { content: importanceRaw } = await chatCompletion({
	messages: [
	{ role: 'user', content: memory.description },
	{
	role: 'user',
	content:
	'How important is this? Answer on a scale of 0 to 9. Respond with number only, e.g. "5"',
	},
	],
	max_tokens: 1,
	});
	let importance = NaN;
	for (let i = 0; i < importanceRaw.length; i++) {
	const number = parseInt(importanceRaw[i]);
	if (!isNaN(number)) {
	importance = number;
	break;
	}
	}
	importance = parseFloat(importanceRaw);
	if (isNaN(importance)) {
	console.debug('importance is NaN', importanceRaw);
	importance = 5;
	}
	return { ...memory, embedding, importance };
	} else {
	return { ...memory, embedding, importance: memory.importance };
	}
	});
	const embeddingIds = await ctx.runMutation(internal.lib.memory.addMemories, { memories });
	if (externalEmbeddingStore) {
	await externalEmbeddingStore(
	embeddingIds.map((id, idx) => ({
	id,
	values: embeddings[idx],
	metadata: { playerId: memories[idx].playerId },
	})),
	);
	}
	},
	async rememberConversation(playerName, playerId, playerIdentity, conversationId) {
	const messages = await ctx.runQuery(internal.lib.memory.getRecentMessages, {
	playerId,
	conversationId,
	});
	if (!messages.length) return false;
	const { content: description } = await chatCompletion({
	messages: [
	{
	role: 'user',
	content: `The following are messages. You are ${playerName}, and ${playerIdentity}
	I would like you to summarize the conversation in a paragraph from your perspective. Add if you like or dislike this interaction.`,
	},
	...chatHistoryFromMessages(messages),
	{
	role: 'user',
	content: `Summary:`,
	},
	],
	max_tokens: 500,
	});
	await this.addMemories([
	{
	playerId,
	description,
	data: {
	type: 'conversation',
	conversationId,
	},
	},
	]);
	return true;
	},
	async reflectOnMemories(playerId: Id<'players'>, name: string) {
	const { memories, lastReflectionTs } = await ctx.runQuery(
	internal.lib.memory.getReflectionMemories,
	{
	playerId,
	numberOfItems: 100,
	},
	);
	// should only reflect if lastest 100 items have importance score of >500
	const sumOfImportanceScore = memories
	.filter((m) => m._creationTime > (lastReflectionTs ?? 0))
	.reduce((acc, curr) => acc + curr.importance, 0);
	console.debug('sum of importance score = ', sumOfImportanceScore);
	const shouldReflect = sumOfImportanceScore > 500;
	if (shouldReflect) {
	console.debug('Reflecting...');
	let prompt = `[no prose]\n [Output only JSON] \nYou are ${name}, statements about you:\n`;
	memories.forEach((m, idx) => {
	prompt += `Statement ${idx}: ${m.description}\n`;
	});
	prompt += `What 3 high-level insights can you infer from the above statements?
	Return in JSON format, where the key is a list of input statements that contributed to your insights and value is your insight. Make the response parseable by Typescript JSON.parse() function. DO NOT escape characters or include '\n' or white space in response.
	Example: [{insight: "...", statementIds: [1,2]}, {insight: "...", statementIds: [1]}, ...]`;
	const { content: reflection } = await chatCompletion({
	messages: [
	{
	role: 'user',
	content: prompt,
	},
	],
	});
	try {
	const insights: { insight: string; statementIds: number[] }[] = JSON.parse(reflection);
	let memoriesToSave: MemoryOfType<'reflection'>[] = [];
	insights.forEach((item) => {
	const relatedMemoryIds = item.statementIds.map((idx: number) => memories[idx]._id);
	const reflectionMemory = {
	playerId,
	description: item.insight,
	data: {
	type: 'reflection',
	relatedMemoryIds,
	},
	} as MemoryOfType<'reflection'>;
	memoriesToSave.push(reflectionMemory);
	});
	console.debug('adding reflection memory...', memoriesToSave);
	await this.addMemories(memoriesToSave);
	} catch (e) {
	console.error('error saving or parseing reflection', e);
	console.debug('reflection', reflection);
	return;
	}
	}
	},
	};
	}
	export const filterMemoriesType = <T extends MemoryType>(
	memoryTypes: T[],
	memories: { memory: Doc<'memories'>; overallScore: number }[],
	) => {
	return memories.filter((m) => {
	return memoryTypes.includes(m.memory.data.type as T);
	}) as { memory: MemoryOfType<T>; overallScore: number }[];
	};
	export const getMemories = internalQuery({
	args: { playerId: v.id('players'), embeddingIds: v.array(v.id('embeddings')) },
	handler: async (ctx, args): Promise<Memory[]> => {
	return await asyncMap(args.embeddingIds, (id) =>
	getMemoryByEmbeddingId(ctx.db, args.playerId, id),
	);
	},
	});
	export const accessMemories = internalMutation({
	args: {
	playerId: v.id('players'),
	candidates: v.array(v.object({ _id: v.id('embeddings'), score: v.number() })),
	count: v.number(),
	},
	handler: async (ctx, { playerId, candidates, count }) => {
	const ts = Date.now();
	const relatedMemories = await asyncMap(candidates, ({ _id }) =>
	getMemoryByEmbeddingId(ctx.db, playerId, _id),
	);
	// TODO: fetch <count> recent memories and <count> important memories
	// so we don't miss them in case they were a little less relevant.
	const recencyScore = relatedMemories.map((memory) => {
	return 0.99 ^ Math.floor((ts - memory.lastAccess) / 1000 / 60 / 60);
	});
	const relevanceRange = makeRange(candidates.map((c) => c.score));
	const importanceRange = makeRange(relatedMemories.map((m) => m.importance));
	const recencyRange = makeRange(recencyScore);
	const memoryScores = relatedMemories.map((memory, idx) => ({
	memory,
	overallScore:
	normalize(candidates[idx].score, relevanceRange) +
	normalize(memory.importance, importanceRange) +
	normalize(recencyScore[idx], recencyRange),
	}));
	memoryScores.sort((a, b) => b.overallScore - a.overallScore);
	const accessed = memoryScores.slice(0, count);
	await asyncMap(accessed, async ({ memory }) => {
	if (memory.lastAccess < ts - MEMORY_ACCESS_THROTTLE) {
	await ctx.db.patch(memory._id, { lastAccess: ts });
	}
	});
	return accessed;
	},
	});
	function normalize(value: number, range: readonly [number, number]) {
	const [min, max] = range;
	return (value - min) / (max - min);
	}
	function makeRange(values: number[]) {
	const min = Math.min(...values);
	const max = Math.max(...values);
	return [min, max] as const;
	}
	// Unused, but in case they're helpful later.
	// export const embedMemory = internalAction({
	// args: { memory: v.object(NewMemory) },
	// handler: async (ctx, args): Promise<Id<'memories'>> => {
	// return (await MemoryDB(ctx).addMemories([args.memory]))[0];
	// },
	// });
	// export const embedMemories = internalAction({
	// args: { memories: v.array(v.object(NewMemory)) },
	// handler: async (ctx, args): Promise<Id<'memories'>[]> => {
	// return await MemoryDB(ctx).addMemories(args.memories);
	// },
	// });
	// export const addMemory = internalMutation({
	// args: NewMemoryWithEmbedding,
	// handler: async (ctx, args): Promise<Id<'memories'>> => {
	// const { embedding, ...memory } = args;
	// const { playerId, description: text } = memory;
	// const embeddingId = await ctx.db.insert('embeddings', { playerId, embedding, text });
	// return await ctx.db.insert('memories', { ...memory, embeddingId });
	// },
	// });

cat-town/convex/lib/memory.ts

Lines 1 to 80 in d76e3ca

	import { Infer, v } from 'convex/values';
	import { internal } from '../_generated/api.js';
	import { Doc, Id } from '../_generated/dataModel.js';
	import {
	ActionCtx,
	DatabaseReader,
	internalMutation,
	internalQuery,
	} from '../_generated/server.js';
	import { asyncMap } from './utils.js';
	import { EntryOfType, Memories, Memory, MemoryOfType, MemoryType } from '../schema.js';
	import { chatCompletion } from './openai.js';
	import { clientMessageMapper } from '../chat.js';
	import { pineconeAvailable, queryVectors, upsertVectors } from './pinecone.js';
	import { chatHistoryFromMessages } from '../conversation.js';
	import { MEMORY_ACCESS_THROTTLE } from '../config.js';
	import { fetchEmbeddingBatchWithCache } from './cached_llm.js';
	const { embeddingId: _, lastAccess, ...MemoryWithoutEmbeddingId } = Memories.fields;
	const NewMemory = { ...MemoryWithoutEmbeddingId, importance: v.optional(v.number()) };
	const NewMemoryWithEmbedding = { ...MemoryWithoutEmbeddingId, embedding: v.array(v.number()) };
	const NewMemoryObject = v.object(NewMemory);
	type NewMemory = Infer<typeof NewMemoryObject>;
	export interface MemoryDB {
	search(
	playerId: Id<'players'>,
	vector: number[],
	limit?: number,
	): Promise<{ memory: Doc<'memories'>; score: number }[]>;
	accessMemories(
	playerId: Id<'players'>,
	queryEmbedding: number[],
	count?: number,
	): Promise<{ memory: Doc<'memories'>; overallScore: number }[]>;
	addMemories(memories: NewMemory[]): Promise<void>;
	rememberConversation(
	playerName: string,
	playerId: Id<'players'>,
	playerIdentity: string,
	conversationId: Id<'conversations'>,
	): Promise<boolean>;
	reflectOnMemories(playerId: Id<'players'>, name: string): Promise<void>;
	}
	export function MemoryDB(ctx: ActionCtx): MemoryDB {
	if (!pineconeAvailable()) {
	throw new Error('Pinecone environment variables not set. See the README.');
	}
	// If Pinecone env variables are defined, use that.
	const vectorSearch = async (embedding: number[], playerId: Id<'players'>, limit: number) =>
	queryVectors('embeddings', embedding, { playerId }, limit);
	const externalEmbeddingStore = async (
	embeddings: { id: Id<'embeddings'>; values: number[]; metadata: object }[],
	) => upsertVectors('embeddings', embeddings);
	return {
	// Finds memories but doesn't mark them as accessed.
	async search(playerId, queryEmbedding, limit = 100) {
	const results = await vectorSearch(queryEmbedding, playerId, limit);
	const embeddingIds = results.map((r) => r._id);
	const memories = await ctx.runQuery(internal.lib.memory.getMemories, {
	playerId,
	embeddingIds,
	});
	return results.map(({ score }, idx) => ({ memory: memories[idx], score }));
	},
	async accessMemories(playerId, queryEmbedding, count = 10) {
	const results = await vectorSearch(queryEmbedding, playerId, 10 * count);
	return await ctx.runMutation(internal.lib.memory.accessMemories, {
	playerId,
	candidates: results,
	count,
	});
	},
	async addMemories(memoriesWithoutEmbedding) {
	const texts = memoriesWithoutEmbedding.map((memory) => memory.description);
	const { embeddings } = await fetchEmbeddingBatchWithCache(ctx, texts);

cat-town/convex/lib/memory.ts

Lines 311 to 429 in d76e3ca

	// const embeddingId = await ctx.db.insert('embeddings', { playerId, embedding, text });
	// return await ctx.db.insert('memories', { ...memory, embeddingId });
	// },
	// });
	export const addMemories = internalMutation({
	args: { memories: v.array(v.object(NewMemoryWithEmbedding)) },
	handler: async (ctx, args): Promise<Id<'embeddings'>[]> => {
	return asyncMap(args.memories, async (memoryWithEmbedding) => {
	const { embedding, ...memory } = memoryWithEmbedding;
	const { playerId, description: text } = memory;
	const embeddingId = await ctx.db.insert('embeddings', { playerId, embedding, text });
	await ctx.db.insert('memories', { ...memory, lastAccess: Date.now(), embeddingId });
	return embeddingId;
	});
	},
	});
	// Technically it's redundant to retrieve them by playerId, since the embedding
	// is stored associated with an playerId already.
	async function getMemoryByEmbeddingId(
	db: DatabaseReader,
	playerId: Id<'players'>,
	embeddingId: Id<'embeddings'>,
	) {
	const doc = await db
	.query('memories')
	.withIndex('by_playerId_embeddingId', (q) =>
	q.eq('playerId', playerId).eq('embeddingId', embeddingId),
	)
	.order('desc')
	.first();
	if (!doc) throw new Error(`No memory found for player ${playerId} and embedding ${embeddingId}`);
	return doc;
	}
	export const getReflectionMemories = internalQuery({
	args: { playerId: v.id('players'), numberOfItems: v.number() },
	handler: async (ctx, { playerId, numberOfItems }) => {
	const conversations = await ctx.db
	.query('memories')
	.withIndex('by_playerId_type', (q) =>
	//TODO - we should get memories of other types once we can
	// Probably with an index just on playerId, so we can sort by time
	q.eq('playerId', playerId).eq('data.type', 'conversation'),
	)
	.order('desc')
	.take(numberOfItems);
	console.debug('conversation memories lenth', conversations.length);
	const reflections = await ctx.db
	.query('memories')
	.withIndex('by_playerId_type', (q) =>
	q.eq('playerId', playerId).eq('data.type', 'reflection'),
	)
	.order('desc')
	.take(numberOfItems);
	const lastReflection = await ctx.db
	.query('memories')
	.withIndex('by_playerId_type', (q) =>
	q.eq('playerId', playerId).eq('data.type', 'reflection'),
	)
	.order('desc')
	.first();
	const mergedList = reflections.concat(conversations);
	mergedList.sort((a, b) => b._creationTime - a._creationTime);
	return {
	memories: mergedList.slice(0, numberOfItems),
	lastReflectionTs: lastReflection?._creationTime,
	};
	},
	});
	export const getRecentMessages = internalQuery({
	args: {
	playerId: v.id('players'),
	conversationId: v.id('conversations'),
	},
	handler: async (ctx, { playerId, conversationId }) => {
	// Fetch the first message to bound the search for the last memory.
	// Only a slight optimization for memory search, which might scan to the
	// beginning of time (for this user's conversations).
	const firstMessage = (await ctx.db
	.query('journal')
	.withIndex('by_conversation', (q) => q.eq('data.conversationId', conversationId as any))
	.first()) as EntryOfType<'talking'>;
	// Look for the last conversation memory for this conversation
	// Only need to check from when the first message exists.
	const lastConversationMemory = (await ctx.db
	.query('memories')
	.withIndex('by_playerId_type', (q) =>
	q
	.eq('playerId', playerId)
	.eq('data.type', 'conversation')
	.gt('_creationTime', firstMessage._creationTime),
	)
	.order('desc')
	.filter((q) => q.eq(q.field('data.conversationId'), conversationId))
	.first()) as MemoryOfType<'conversation'> | null;
	const allMessages = (await ctx.db
	.query('journal')
	.withIndex('by_conversation', (q) => {
	const q2 = q.eq('data.conversationId', conversationId as any);
	if (lastConversationMemory) {
	// If we have a memory of this conversation, only look at messages after.
	return q2.gt('_creationTime', lastConversationMemory._creationTime);
	}
	return q2;
	})
	.filter((q) => q.eq(q.field('data.type'), 'talking'))
	.collect()) as EntryOfType<'talking'>[];
	return (await asyncMap(allMessages, clientMessageMapper(ctx.db))).filter(
	(m) => m.from === playerId || m.to.includes(playerId),
	);
	},
	});

---

Step 2: 🧐 Snippet Analysis

From looking through the relevant snippets, I decided to make the following modifications:

File Path

Proposed Changes

convex/conversation.ts

Modify convex/conversation.ts with contents: * Import the filterMemoriesType function from ./lib/memory at the top of the file. • Inside the converse() function, after the line where conversationMemories is defined, add a new line to define reflectionMemories. Call filterMemoriesType() with 'reflection' as the memory type and memories as the second argument. Use the slice() method to limit the result to the top 5 memories. • Modify the prefixPrompt variable to include the reflectionMemories. After the line where relevantMemories is defined, add a new line to define relevantReflections. Map over the reflectionMemories and join their descriptions with a newline character. Include relevantReflections in the prefixPrompt string.

---

Step 3: 📝 Planning

I have created a plan for writing the pull request. I am now working my plan and coding the required changes to address this issue. Here is the planned pull request:

Include top 5 reflection memories in converse() function
sweep/fix-converse-function

Description

This PR addresses the issue #5 by modifying the converse() function in conversation.ts to include the top 5 reflection memories in the chat prompt. Currently, the function only fetches conversation memories and does not consider reflection memories. This update ensures that the chat prompt includes relevant reflection memories, providing a more contextually aware conversation experience.

Summary of Changes

• Imported the filterMemoriesType function from ./lib/memory in conversation.ts.
• Added a new line in the converse() function to fetch the top 5 reflection memories using filterMemoriesType() with 'reflection' as the memory type.
• Modified the prefixPrompt variable to include the reflectionMemories in the chat prompt.
• Updated the prefixPrompt to include the descriptions of the relevant reflection memories.
• Created a new branch fix-converse-function for these changes.

---

Step 4: ⌨️ Coding

File	Instructions	Progress
convex/conversation.ts	Modify convex/conversation.ts with contents: * Import the filterMemoriesType function from ./lib/memory at the top of the file. • Inside the converse() function, after the line where conversationMemories is defined, add a new line to define reflectionMemories. Call filterMemoriesType() with 'reflection' as the memory type and memories as the second argument. Use the slice() method to limit the result to the top 5 memories. • Modify the prefixPrompt variable to include the reflectionMemories. After the line where relevantMemories is defined, add a new line to define relevantReflections. Map over the reflectionMemories and join their descriptions with a newline character. Include relevantReflections in the prefixPrompt string.	✅ Commit d76e3ca

---

Step 5: 🔁 Code Review

Here are my self-reviews of my changes at sweep/fix-converse-function.

Here is the 1st review

Hi there,

Great job on your work so far. I've noticed a few things that need to be addressed:

• In the file convex/conversation.ts, there is a duplicate line of code on lines 122 and 123. You only need to join the reflectionMemories descriptions once, so one of these lines can be removed.

Keep up the good work!

I finished incorporating these changes.

---

To recreate the pull request edit the issue title or description.
^{Join Our Discord}