You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+23-29Lines changed: 23 additions & 29 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,26 @@
1
1
# ResilientLLM
2
2
3
-
A simple but robust LLM integration layer designed to ensure reliable, seamless interactions across multiple APIs by intelligently handling failures and rate limits.
3
+
A minimalist but robust LLM integration layer designed to ensure reliable, seamless interactions across multiple LLM providers by intelligently handling failures and rate limits.
4
+
5
+
---
6
+
7
+
This library solves challenges in building production-ready AI Agents due to:
8
+
9
+
- ❌ Unstable network conditions
10
+
- ⚠️ Inconsistent error handling
11
+
- ⏳ Unpredictable LLM API rate limit errors
12
+
13
+
### Key Features
14
+
15
+
-**Token estimation**: You don’t need to calculate LLM tokens, they are estimated for each request
16
+
-**Rate limiting**: You don't need to manage the token bucket rate algorithm yourself to follow the rate limits by LLM service providers, it is done for you automatically
17
+
-**Retries, backoff, and circuit breaker**: All are handled internally by the `ResilientOperation`.
18
+
19
+
## Installation
20
+
21
+
```bash
22
+
npm i resilient-llm
23
+
```
4
24
5
25
## Quickstart
6
26
@@ -16,7 +36,7 @@ const llm = new ResilientLLM({
16
36
requestsPerMinute:60, // Limit to 60 requests per minute
17
37
llmTokensPerMinute:90000// Limit to 90,000 LLM tokens per minute
18
38
},
19
-
retries:3, // Number of times to retry if req. fails for reasons possible to fix by retry
39
+
retries:3, // Number of times to retry when req. fails and only if it is possible to fix by retry
20
40
backoffFactor:2// Increase delay between retries by this factor
21
41
});
22
42
@@ -35,32 +55,6 @@ const conversationHistory = [
35
55
})();
36
56
```
37
57
38
-
---
39
-
40
-
### Key Points
41
-
42
-
-**Rate limiting is automatic**: You don’t need to pass token counts or manage rate limits yourself.
43
-
-**Token estimation**: The number of LLM tokens is estimated for each request and enforced.
44
-
-**Retries, backoff, and circuit breaker**: All are handled internally by the `ResilientOperation`.
45
-
46
-
---
47
-
48
-
### Advanced: With Custom Options
49
-
50
-
```js
51
-
constresponse=awaitllm.chat(
52
-
[
53
-
{ role:'user', content:'Summarize the plot of Inception.' }
54
-
],
55
-
{
56
-
maxTokens:512,
57
-
temperature:0.5,
58
-
aiService:'anthropic', // override default
59
-
model:'claude-3-5-sonnet-20240620'
60
-
}
61
-
);
62
-
```
63
-
64
58
## Motivation
65
59
66
60
ResilientLLM is a resilient, unified LLM interface featuring circuit breaker, token bucket rate limiting, caching, and adaptive retry with dynamic backoff support.
@@ -76,7 +70,7 @@ The final solution was to extract tiny LLM orchestration class out of all my AI
76
70
This library solves my challenges in building production-ready AI Agents such as:
77
71
- unstable network conditions
78
72
- inconsistent error handling
79
-
- unpredictable LLM API rate limit errrors
73
+
- unpredictable LLM API rate limit errors
80
74
81
75
This library aims to solve the same challenges for you by providing a resilient layer that intelligently manages failures and rate limits, enabling you (developers) to integrate LLMs confidently and effortlessly at scale.
0 commit comments