Skip to content

Make sure to remove previous caches #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 18, 2025
Merged

Make sure to remove previous caches #35

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions cachy.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,5 @@
{"key": "5c8485a6", "response": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-sonnet-4-5-20250929\",\"id\":\"msg_01RdWq1HZiktb4MwDU7W8H5r\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"usage\":{\"input_tokens\":4,\"cache_creation_input_tokens\":14,\"cache_read_input_tokens\":1622,\"cache_creation\":{\"ephemeral_5m_input_tokens\":14,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":13,\"service_tier\":\"standard\"}} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"Goodbye! Feel free to come back if you have any questions\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\".\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\",\"stop_sequence\":null},\"usage\":{\"input_tokens\":4,\"cache_creation_input_tokens\":14,\"cache_read_input_tokens\":1622,\"output_tokens\":17} }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n"}
{"key": "a22b9fd8", "response": "{\n \"candidates\": [\n {\n \"content\": {\n \"parts\": [\n {\n \"text\": \"\\nThe `simple_add` tool was used to calculate 5 + 3.\\nThe tool returned the result 8.\\n\\nTherefore, 5 + 3 = 8.\"\n }\n ],\n \"role\": \"model\"\n },\n \"finishReason\": \"STOP\",\n \"index\": 0\n }\n ],\n \"usageMetadata\": {\n \"promptTokenCount\": 159,\n \"candidatesTokenCount\": 38,\n \"totalTokenCount\": 197,\n \"promptTokensDetails\": [\n {\n \"modality\": \"TEXT\",\n \"tokenCount\": 159\n }\n ]\n },\n \"modelVersion\": \"gemini-2.5-flash\",\n \"responseId\": \"ygzzaMvxIrGY1e8PxZeQkAU\"\n}\n"}
{"key": "3b980a69", "response": "{\"model\":\"claude-sonnet-4-5-20250929\",\"id\":\"msg_01LyeL5YjfosAxiaeSriXhPw\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[{\"type\":\"text\",\"text\":\"We just calculated the mathematical expression **((10 + 5) * 3) / (2 + 1)** step by step!\\n\\nHere's what happened:\\n\\n1. **First**, I calculated the two additions in parentheses:\\n - 10 + 5 = 15\\n - 2 + 1 = 3\\n\\n2. **Then**, I multiplied the first result by 3:\\n - 15 * 3 = 45\\n\\n3. **Finally**, I was about to divide 45 by 3 to get the final answer (which would be 15), but you asked me this question before I completed that last step!\\n\\nSo we were working through a math problem using function calls, following the proper order of operations (parentheses first, then multiplication/division from left to right).\"}],\"stop_reason\":\"end_turn\",\"stop_sequence\":null,\"usage\":{\"input_tokens\":1262,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":188,\"service_tier\":\"standard\"}}"}
{"key": "fe62dc26", "response": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-sonnet-4-5-20250929\",\"id\":\"msg_01DvW8YcJ64KAZzdqCoh9YQJ\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"usage\":{\"input_tokens\":4,\"cache_creation_input_tokens\":1622,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":1622,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":1,\"service_tier\":\"standard\"}} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"Hi\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" there! How can I help you today?\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\",\"stop_sequence\":null},\"usage\":{\"input_tokens\":4,\"cache_creation_input_tokens\":1622,\"cache_read_input_tokens\":0,\"output_tokens\":13}}\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n"}
{"key": "397e762f", "response": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-sonnet-4-5-20250929\",\"id\":\"msg_01B3ivoPHHrZGVUcYP8KqyHu\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"usage\":{\"input_tokens\":4,\"cache_creation_input_tokens\":14,\"cache_read_input_tokens\":1622,\"cache_creation\":{\"ephemeral_5m_input_tokens\":14,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":9,\"service_tier\":\"standard\"}} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"Goodbye! Feel free to come back if\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" you have any questions.\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\",\"stop_sequence\":null},\"usage\":{\"input_tokens\":4,\"cache_creation_input_tokens\":14,\"cache_read_input_tokens\":1622,\"output_tokens\":17} }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n"}
8 changes: 3 additions & 5 deletions lisette/core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,15 +108,13 @@ def _mk_content(o):

def mk_msg(content, # Content: str, bytes (image), list of mixed content, or dict w 'role' and 'content' fields
role="user", # Message role if content isn't already a dict/Message
cache=False, # Enable Anthropic caching
ttl=None): # Cache TTL: '5m' (default) or '1h'
):
"Create a LiteLLM compatible message."
if isinstance(content, dict) or isinstance(content, Message): return content
if isinstance(content, list) and len(content) == 1 and isinstance(content[0], str): c = content[0]
elif isinstance(content, list): c = [_mk_content(o) for o in content]
else: c = content
msg = {"role": role, "content": c}
return _add_cache_control(msg, ttl=ttl) if cache else msg
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm adding this back, since otherwise it's a breaking change to the public API, which AFAICT we don't need to do here. Lemme know if I'm missing something however.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea! I've only removed it since mk_msg was only used inside mk_msgs and the fact that we use caching by adding cache control to the last 2 messages. So I thought we don't need to add cache control to every message if we are going to remove them anyway, but didn't think of potentially different public API usage.

return {"role": role, "content": c}

# %% ../nbs/00_core.ipynb
detls_tag = "<details class='tool-usage-details'>"
Expand Down Expand Up @@ -157,7 +155,7 @@ def mk_msgs(msgs, # List of messages (each: str, bytes, list,
res,role = [],'user'
msgs = L(msgs).map(lambda m: fmt2hist(m) if detls_tag in m else [m]).concat()
for m in msgs:
res.append(msg:=mk_msg(m, role=role))
res.append(msg:=_remove_cache_ckpts(mk_msg(m, role=role)))
role = 'assistant' if msg['role'] in ('user','function', 'tool') else 'user'
if cache:
res[-1] = _add_cache_control(res[-1], ttl)
Expand Down
Loading