+
+ | Category |
+ Function |
+ Status |
+ Description |
+
+
+ | API Service |
+ OpenAI Standard API |
+ ✅ |
+ The service interface complies with OpenAI standards, minimizing integration costs through standardized APIs. It enables users to seamlessly integrate and maintain the system, swiftly respond to business requirements, and concentrate on core development. |
+
+
+ | Blocking access capabilities |
+ ✅ |
+ Suitable for tasks requiring integrity and coherence or for overall verification and processing of results, this approach obtains complete output in a single iteration. Throughout the process, the user must wait until all output content has been fully generated. |
+
+
+ | Streaming access capabilities |
+ ✅ |
+ Suitable for real-time applications with stringent response time requirements, such as code completion, real-time translation, or websites with dynamic content loading. The model transmits content incrementally during generation, enabling users to receive and process partial outputs immediately without waiting for full completion, thereby enhancing interactivity. |
+
+
+ | High-performance gateway |
+ ⬜ |
+ High-performance gateways effectively manage high-concurrency requests, reduce latency, and enhance response times by optimizing data transmission, employing advanced load balancing algorithms, and implementing efficient resource management. |
+
+
+ | Multi-engine Support |
+ Ollama |
+ ✅ |
+ High-performance gateways effectively manage high-concurrency requests, reduce latency, and enhance response times by optimizing data transmission, employing advanced load balancing algorithms, and implementing efficient resource management. |
+
+
+ | vLLM |
+ ✅ |
+ vLLM exhibits significant advantages in memory management and throughput. By optimizing memory usage and parallel computation, it substantially enhances inference speed and resource efficiency, while maintaining compatibility with various hardware environments. vLLM offers a wide range of configuration options, allowing users to adjust inference strategies based on their needs. Its scalable architecture makes it suitable for both research and enterprise-level applications. |
+
+
+ | Tensorrt–LLM |
+ ⬜ |
+ TensorRT-LLM (TensorRT for Large Language Models) is a high-performance, scalable deep learning inference optimization library developed by NVIDIA, specifically designed for large language models (LLMs). |
+
+
+ | Docker Deployment Capability |
+ Docker images built with Python 3.10 |
+ ✅ |
+ TensorRT-LLM is a high-performance, scalable deep learning inference optimization library developed by NVIDIA, specifically designed for large language models (LLMs). |
+
+
+ | Web UI Integration |
+ OpenUI protocol |
+ ⬜ |
+ The comprehensive UI open-source protocol facilitates users in integrating diverse components, enhancing product customizability and extensibility. |
+
+
+ | More Core Features |
+ ModelCache semantic caching |
+ ⬜ |
+ By caching generated QA pairs, similar requests can achieve millisecond-level responses, enhancing the performance and efficiency of model inference. |
+
+
+
+
+## Acknowledgements
+This project draws on the following open-source projects, and we express our gratitude to the relevant projects and researchers for their contributions.
+[Ollama](https://github.com/ollama/ollama)、[vLLM](https://github.com/vllm-project/vllm)
+
+##
+
+ | 分类 |
+ 功能名称 |
+ 状态 |
+ 描述 |
+
+
+ | API Service |
+ 基于Open AI的标准API规范 |
+ ✅ |
+ 服务接口遵循 OpenAI 规范，通过标准化 API 降低接入成本，用户可轻松集成功能，快速响应业务需求，专注于核心开发。 |
+
+
+ | 阻塞式访问能力 |
+ ✅ |
+ 适用于需要完整性和准确性的任务，完成时结果进行整体校验或输出的任务，一次性获取完整输出。在整个过程中，用户需要等待直至所有输出内容完全完成。 |
+
+
+ | 流式访问能力 |
+ ✅ |
+ 适用于对响应时间要求较高的实时应用，如代码补全、实时翻译或动态内容加载的场景。模型在生成过程中分段逐步传输内容，用户可在内容生成后立即接收和处理，无需等待全部完成，从而提升效率。 |
+
+
+ | 高性能网络，提升用户开发能力 |
+ ⬜ |
+ 高性能网络通过优化数据传输、采用先进负载均衡算法及高效资源管理，能有效提升数据来源、降低延迟、提升响应速度。 |
+
+
+ | 多引擎支持 |
+ Ollama |
+ ✅ |
+ Ollama 以易用和轻量著称，专注于高效稳定的大模型推理服务。其友好 API 和简洁流畅流程，使开发者能够轻松将其手作快速部署应用。 |
+
+
+ | vLLM |
+ ✅ |
+ vLLM在内存管理和吞吐量上有显著优势，其通过优化存储和并行计算，显著提升推理速度和资源利用率，兼容多种硬件环境。vLLM提供丰富的配置选项，用户可根据需求调整推理策略，适用于实时和企业级应用。 |
+
+
+ | Tensorrt–LLM |
+ ⬜ |
+ TensorRT–LLM (TensorRT for Large Language Models) 是NVIDIA优化的高性能、大规模推理优化库，专为大型语言模型（LLM）设计。 |
+
+
+ | Docker部署能力 |
+ 基于python3.10构建Docker镜像 |
+ ✅ |
+ 将大型模型及其依赖的镜像，确保版本号一致运行，简化部署与配置。利用Docker的版本构建和自动化部署，提高模型更新与迭代效率，加快从开发到生产落地的转化。 |
+
+
+ | Web UI接入 |
+ OpenUI 协议 |
+ ⬜ |
+ 丰富的UI开源协议便于用户整合多种组件，提升产品的定制性和扩展性。 |
+
+
+ | 更多核心功能 |
+ ModelCache语义缓存 |
+ ⬜ |
+ 通过缓存已有生成的QA Pair，使得请求变更更加细粒度，提高模型推理的性能与效率。 |
+
+
+
+## 致谢
+本项目参考了以下开源项目，在此对相关项目和研究开发人员表示感谢。
+[Ollama](https://github.com/ollama/ollama)、[vLLM](https://github.com/vllm-project/vllm)
+
+## Contributing
+EasyDeploy是一个非常有趣且有用的项目，我们相信这个项目有很大的潜力，无论你是经验丰富的开发者，还是刚刚入门的新手，都欢迎你为这个项目做出一些贡献，包括但不限于：提交问题和建议，参与代码编写，完善文档和示例。
+
diff --git a/docs/easydeploy_modules_20241125.png b/docs/easydeploy_modules_20241125.png
new file mode 100644
index 0000000..68cfe15
Binary files /dev/null and b/docs/easydeploy_modules_20241125.png differ
diff --git a/example/__init__.py b/example/__init__.py
new file mode 100644
index 0000000..40a96af
--- /dev/null
+++ b/example/__init__.py
@@ -0,0 +1 @@
+# -*- coding: utf-8 -*-
diff --git a/example/hello_word.py b/example/hello_word.py
new file mode 100644
index 0000000..22ff936
--- /dev/null
+++ b/example/hello_word.py
@@ -0,0 +1,15 @@
+# -*- coding: utf-8 -*-
+import json
+import requests
+
+
+def run():
+ url = 'http://127.0.0.1:5000/'
+ headers = {"Content-Type": "application/json"}
+ res = requests.post(url, headers=headers)
+ res_text = res.text
+ print('res_text: {}'.format(res_text))
+
+
+if __name__ == '__main__':
+ run()
diff --git a/example/ollama_app.py b/example/ollama_app.py
new file mode 100644
index 0000000..34d0af3
--- /dev/null
+++ b/example/ollama_app.py
@@ -0,0 +1,21 @@
+# -*- coding: utf-8 -*-
+import json
+import requests
+
+
+def run():
+ url = 'http://127.0.0.1:8000/chat/completions'
+ prompt = '你是谁？'
+ model = 'qwen2:0.5b'
+ messages = [{"role": "user", "content": prompt}]
+ data = {'model': model, 'messages': messages}
+ headers = {"Content-Type": "application/json"}
+ response = requests.post(url, headers=headers, data=json.dumps(data))
+
+ if response.status_code == 200:
+ ans_dict = json.loads(response.text)
+ print('data: {}'.format(ans_dict))
+
+
+if __name__ == '__main__':
+ run()
diff --git a/example/ollama_app_stream.py b/example/ollama_app_stream.py
new file mode 100644
index 0000000..4fe661e
--- /dev/null
+++ b/example/ollama_app_stream.py
@@ -0,0 +1,28 @@
+# -*- coding: utf-8 -*-
+import json
+import requests
+
+
+# 发起请求，并将stream参数设置为True以获取流式输出
+url = 'http://127.0.0.1:8000/chat/completions'
+prompt = '你是谁？'
+model = 'qwen2:0.5b'
+messages = [{"role": "user", "content": prompt}]
+data = {'model': model, 'messages': messages, 'stream': True}
+headers = {"Content-Type": "application/json"}
+
+response = requests.post(url, headers=headers, data=json.dumps(data))
+
+resp = ''
+for line in response.iter_lines():
+ print('line: {}'.format(line))
+ print('line: {}'.format(type(line)))
+ data = line.decode('utf-8')
+ print('data: {}'.format(data))
+ print('data: {}'.format(type(data)))
+ data_dict = json.loads(data)
+ print('data_dict: {}'.format(data_dict))
+ print('data_dict: {}'.format(type(data_dict)))
+ text = data_dict['choices'][-1]['delta']['content']
+ resp += text
+print('resp: {}'.format(resp))
diff --git a/server/__init__.py b/server/__init__.py
new file mode 100644
index 0000000..40a96af
--- /dev/null
+++ b/server/__init__.py
@@ -0,0 +1 @@
+# -*- coding: utf-8 -*-
diff --git a/server/ollama_server.py b/server/ollama_server.py
new file mode 100644
index 0000000..2da2a01
--- /dev/null
+++ b/server/ollama_server.py
@@ -0,0 +1,22 @@
+# -*- coding: utf-8 -*-
+import requests
+import json
+
+url_generate = "http://127.0.0.1:11434/api/generate"
+
+
+def get_response(url, data):
+ response = requests.post(url, json=data)
+ response_dict = json.loads(response.text)
+ response_content = response_dict["response"]
+ return response_content
+
+
+data = {
+ "model": "qwen2:0.5b",
+ "prompt": "你是谁?",
+ "stream": False
+}
+
+res = get_response(url_generate,data)
+print(res)
diff --git a/templates/chat_page.html b/templates/chat_page.html
new file mode 100644
index 0000000..6bd7291
--- /dev/null
+++ b/templates/chat_page.html
@@ -0,0 +1,131 @@
+
+
+
+ This is a simple Flask web page.
+
+