Skip to content

Commit cb435ad

Browse files
Harikeshav-RHarikeshav-R
committed
docs: add detailed architecture and build documentation
- Added `architecture.md` to provide an overview of LEAP's system design, including high-level data flow, component relationships, and distributed ring topology. - Documented `build-system.md` to describe CMake configuration, build targets, dependencies, and platform-specific settings. - Introduced `exporter.md` with a detailed explanation of the model export pipeline, CLI usage, and binary formats (FP32 and INT8). - Added `inference-engine.md` detailing inference workflow, SIMD optimizations, distributed mode, and tokenizer/sampler implementations. - Documented `kernel-module.md` to outline the zero-copy networking design, buffer layout, and RX/TX paths for the kernel module.
1 parent d96c8bd commit cb435ad

9 files changed

Lines changed: 1672 additions & 343 deletions

File tree

‎README.md‎

Lines changed: 177 additions & 343 deletions
Large diffs are not rendered by default.

‎docs/architecture.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# Architecture Overview
2+
3+
LEAP is structured as a multi-binary C++20 project built around a single CMake configuration. The system compiles into **three executables** (`export`, `tokenizer`, `inference`) and **one static library** (`model`), with an optional Linux kernel module (`leap_kmod`).
4+
5+
## High-Level Data Flow
6+
7+
The end-to-end pipeline follows four stages:
8+
9+
```mermaid
10+
graph LR
11+
A["HuggingFace<br/>Safetensors"] -->|"export"| B["LEAP .bin<br/>(FP32 or INT8)"]
12+
C["tokenizer.model<br/>(Tiktoken/BPE)"] -->|"tokenizer"| D["tokenizer.bin"]
13+
B --> E["inference"]
14+
D --> E
15+
E -->|"Ring Topology"| F["Worker Nodes"]
16+
F -->|"Activation Tensors"| E
17+
```
18+
19+
1. **Export** — Loads PyTorch/Safetensors weights via LibTorch, optionally quantizes to INT8, and writes a LEAP binary file.
20+
2. **Tokenize** — Converts a Tiktoken BPE model into a compact binary vocabulary file.
21+
3. **Inference** — Loads the binary model via `mmap`, runs the transformer forward pass with SIMD-optimized kernels, and optionally distributes layers across a ring of networked nodes.
22+
23+
---
24+
25+
## Source Layout
26+
27+
```
28+
src/
29+
├── export/ # Model conversion & quantization (7 files)
30+
│ ├── Loader.cpp # Safetensors → LibTorch tensor loading
31+
│ ├── Export.cpp # FP32/INT8 binary serialization
32+
│ └── main.cpp # CLI entry point
33+
│
34+
├── model/ # LibTorch-based model definitions (13 files)
35+
│ ├── Transformer # Full Llama architecture (training-capable)
36+
│ ├── Attention # Multi-Head / Grouped Query Attention + KV Cache
37+
│ ├── FeedForward # SwiGLU FFN
38+
│ └── RMSNorm # Root Mean Square Layer Normalization
39+
│
40+
├── inference/ # High-performance inference runtime (20 files)
41+
│ ├── FloatTransformer # FP32 forward pass with SIMD
42+
│ ├── QuantizedTransformer # W8A8 INT8 forward pass with SIMD
43+
│ ├── TransformerFactory # Model loading via mmap + version dispatch
44+
│ ├── Tokenizer # Pure C++ BPE encoder/decoder
45+
│ ├── Sampler # Temperature/Top-P/Argmax sampling
46+
│ ├── TcpTransport # TCP transport (ring topology)
47+
│ ├── UdpTransport # UDP transport (chunked datagrams)
48+
│ ├── KernelTransport # Zero-copy kernel module transport
49+
│ ├── Transport.h # Abstract base + control message protocol
50+
│ └── main.cpp # CLI + chat/generate loops
51+
│
52+
├── kernel/ # Linux kernel module (3 files)
53+
│ ├── leap_module.c # Netfilter hook, mmap, ioctl handlers
54+
│ ├── leap_protocol.h # Shared protocol constants (kernel/userspace)
55+
│ └── Makefile
56+
│
57+
└── tokenizer/ # Tokenizer export tool (3 files)
58+
├── Tokenizer.cpp # Tiktoken wrapper + binary export
59+
└── main.cpp # CLI entry point
60+
```
61+
62+
---
63+
64+
## Component Relationships
65+
66+
```mermaid
67+
graph TD
68+
subgraph "Build-Time (Export Pipeline)"
69+
EXP["export binary"]
70+
MOD["model library<br/>(LibTorch)"]
71+
EXP --> MOD
72+
end
73+
74+
subgraph "Runtime (Inference Pipeline)"
75+
INF["inference binary"]
76+
TRANS["Transport Layer"]
77+
INF --> TRANS
78+
end
79+
80+
subgraph "Kernel Space (Linux Only)"
81+
KMOD["leap_kmod"]
82+
end
83+
84+
TRANS -.->|"ioctl + mmap"| KMOD
85+
TOK["tokenizer binary"] --> MOD
86+
```
87+
88+
| Component | Depends On | Output |
89+
|-----------|-----------|--------|
90+
| `model` (lib) | LibTorch | Static library |
91+
| `export` | `model`, LibTorch, nlohmann_json, safetensors-cpp, CLI11 | `model.bin` |
92+
| `tokenizer` | LibTorch, tiktoken-cpp, CLI11 | `tokenizer.bin` |
93+
| `inference` | CLI11, OpenMP | Text generation |
94+
| `leap_kmod` | Linux kernel headers | `leap_transport.ko` |
95+
96+
> **Key Design Decision:** The `inference` binary has **zero dependency** on LibTorch or Python. It operates entirely on raw memory-mapped binary files, using hand-written SIMD kernels. This makes it deployable on any machine with a C++20 compiler and OpenMP.
97+
98+
---
99+
100+
## Distributed Ring Topology
101+
102+
In distributed mode, LEAP splits model layers across nodes connected in a **unidirectional ring**:
103+
104+
```mermaid
105+
graph LR
106+
M["Master<br/>Layers 0..S-1"] -->|"send_next()"| W1["Worker 0<br/>Layers S..E1-1"]
107+
W1 -->|"send_next()"| W2["Worker 1<br/>Layers E1..N-1"]
108+
W2 -->|"send_prev()"| M
109+
```
110+
111+
Each forward pass sends the activation tensor (`dim × float32`) through the ring. The **master** node handles tokenization, sampling, and the generation loop. **Workers** run in a blocking `worker_loop()`, receiving activations, computing their assigned layers, and forwarding the result.
112+
113+
### Control Plane
114+
115+
Layer assignments can be changed at runtime via `/resize` commands. The master sends a `ControlMessage` (padded to full packet size) through the ring, and waits for an `ACK` to propagate back from the tail worker. After resizing, the KV cache is cleared and conversation state is reset.
116+
117+
---
118+
119+
## Precision Modes
120+
121+
| Mode | File Version | Memory per Param | SIMD Kernels | Use Case |
122+
|------|-------------|-----------------|-------------|----------|
123+
| FP32 | v1 | 4 bytes | AVX2 / NEON | Maximum accuracy |
124+
| INT8 (W8A8) | v2 | ~1.25 bytes | `vdotq_s32` (NEON), `_mm256_madd_epi16` (AVX2) | 4× memory reduction, higher throughput |
125+
126+
The `TransformerFactory` reads the binary header to auto-detect precision and instantiate the correct class (`FloatTransformer` or `QuantizedTransformer`).
127+
128+
---
129+
130+
## Further Reading
131+
132+
| Document | Description |
133+
|----------|-------------|
134+
| [Exporter](exporter.md) | Model conversion pipeline and binary formats |
135+
| [Inference Engine](inference-engine.md) | SIMD kernels, forward pass, and distributed execution |
136+
| [Transport Layer](transport.md) | TCP, UDP, and Kernel transport implementations |
137+
| [Kernel Module](kernel-module.md) | Zero-copy Linux kernel module internals |
138+
| [Model Library](model-library.md) | LibTorch-based Llama architecture |
139+
| [Tokenizer](tokenizer.md) | BPE tokenizer export and inference |
140+
| [Build System](build-system.md) | CMake configuration and dependencies |

‎docs/build-system.md‎

Lines changed: 202 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,202 @@
1+
# Build System (`CMakeLists.txt`)
2+
3+
LEAP uses **CMake 4.1+** with C++20, targeting multi-platform builds (Linux, macOS, potentially Windows). The build produces three executables, one static library, and an optional Linux kernel module.
4+
5+
## Build Targets
6+
7+
| Target | Type | Description |
8+
|--------|------|-------------|
9+
| `export` | Executable | Model conversion & quantization tool |
10+
| `inference` | Executable | High-performance inference runtime |
11+
| `tokenizer` | Executable | Tokenizer model exporter |
12+
| `model` | Static Library | LibTorch-based Llama model definitions |
13+
| `leap_kmod` | Kernel Module | Zero-copy networking (Linux only, optional) |
14+
15+
---
16+
17+
## Dependencies
18+
19+
| Dependency | Version | Targets | Source |
20+
|-----------|---------|---------|--------|
21+
| **LibTorch** | 2.4+ | `export`, `model`, `tokenizer` | Manual download or pip |
22+
| **nlohmann/json** | 3.12+ | `export` | FetchContent (auto-downloaded) |
23+
| **safetensors-cpp** | latest | `export` | FetchContent (Git) |
24+
| **CLI11** | 2.6.1 | `export`, `inference`, `tokenizer` | FetchContent (Git) |
25+
| **tokenizers (tiktoken-cpp)** | — | `tokenizer` | Git submodule (`third-party/tokenizers`) |
26+
| **OpenMP** | — | All targets | System (Linux) or Homebrew (macOS) |
27+
28+
### LibTorch
29+
30+
The most significant dependency. Required for the export pipeline but **not** for inference.
31+
32+
**Option A: Direct Download (x86\_64)**
33+
```bash
34+
wget https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-2.4.0%2Bcpu.zip
35+
unzip libtorch-*.zip -d third-party/
36+
```
37+
38+
**Option B: Via Python pip**
39+
```bash
40+
pip install torch --index-url https://download.pytorch.org/whl/cpu
41+
TORCH_PATH=$(python3 -c 'import torch; print(torch.utils.cmake_prefix_path)')
42+
cmake -DCMAKE_PREFIX_PATH=$TORCH_PATH ...
43+
```
44+
45+
> **Note:** Option B is recommended for ARM platforms (Raspberry Pi, Jetson) where pre-built LibTorch binaries aren't available.
46+
47+
---
48+
49+
## Compiler Flags
50+
51+
### Release Optimization Flags
52+
53+
| Platform | Flags |
54+
|----------|-------|
55+
| GCC/Clang | `-O3 -march=native -mtune=native -ffast-math -funroll-loops -fomit-frame-pointer` |
56+
| MSVC | `/O2 /arch:AVX2 /fp:fast` |
57+
58+
Key flags:
59+
- **`-march=native`**: Enables all SIMD instructions supported by the build machine (AVX2, NEON, etc.).
60+
- **`-ffast-math`**: Allows aggressive FP optimizations (breaks strict IEEE compliance — acceptable for inference workloads).
61+
- **`-funroll-loops`**: Aggressive loop unrolling for better pipeline utilization.
62+
63+
### Link-Time Optimization (LTO)
64+
65+
LTO/IPO is automatically detected and enabled for Release builds via `check_ipo_supported()`. This allows cross-translation-unit inlining and dead code elimination.
66+
67+
### Exception Handling
68+
69+
LibTorch's default build flags disable exceptions (`-fno-exceptions`). LEAP's CMake configuration explicitly strips these flags and forces:
70+
```
71+
-fexceptions -frtti -fvisibility=default
72+
```
73+
74+
This ensures C++ exception handling works correctly throughout all targets.
75+
76+
---
77+
78+
## OpenMP Configuration
79+
80+
### Linux
81+
Standard `find_package(OpenMP)` — links `OpenMP::OpenMP_CXX`.
82+
83+
### macOS (Homebrew)
84+
macOS requires special handling because Apple Clang doesn't ship OpenMP:
85+
1. Detects Homebrew's `libomp` prefix via `brew --prefix libomp`.
86+
2. Falls back to hardcoded paths (`/opt/homebrew/opt/libomp` or `/usr/local/opt/libomp`).
87+
3. Adds compile flags: `-Xpreprocessor -fopenmp`.
88+
4. Links the `omp` library directly.
89+
90+
```bash
91+
# Required on macOS
92+
brew install libomp
93+
```
94+
95+
---
96+
97+
## Kernel Module Build
98+
99+
Enabled via the `BUILD_KERNEL_MODULE` CMake option (Linux only):
100+
101+
```bash
102+
cmake -DBUILD_KERNEL_MODULE=ON ...
103+
```
104+
105+
This creates a custom target that invokes the kernel's build system:
106+
```cmake
107+
add_custom_target(leap_kmod ALL
108+
COMMAND make -C ${CMAKE_CURRENT_SOURCE_DIR}/src/kernel
109+
)
110+
```
111+
112+
**Requirements:**
113+
- Linux kernel headers: `sudo apt install linux-headers-$(uname -r)`
114+
115+
---
116+
117+
## Build Instructions
118+
119+
### Quick Start
120+
121+
```bash
122+
# Clone with submodules
123+
git clone --recursive https://github.com/Harikeshav-R/LEAP.git
124+
cd LEAP
125+
126+
# Configure (choose one LibTorch method)
127+
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
128+
-DCMAKE_PREFIX_PATH=$(pwd)/third-party/libtorch
129+
130+
# Build all targets
131+
cmake --build build --config Release -- -j$(nproc)
132+
```
133+
134+
### Build with Kernel Module (Linux)
135+
136+
```bash
137+
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
138+
-DBUILD_KERNEL_MODULE=ON \
139+
-DCMAKE_PREFIX_PATH=$(pwd)/third-party/libtorch
140+
141+
cmake --build build --config Release -- -j$(nproc)
142+
```
143+
144+
### macOS
145+
146+
```bash
147+
brew install libomp cmake
148+
149+
# Using pip-installed PyTorch
150+
pip install torch --index-url https://download.pytorch.org/whl/cpu
151+
TORCH_PATH=$(python3 -c 'import torch; print(torch.utils.cmake_prefix_path)')
152+
153+
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
154+
-DCMAKE_PREFIX_PATH=$TORCH_PATH
155+
156+
cmake --build build --config Release -- -j$(sysctl -n hw.ncpu)
157+
```
158+
159+
---
160+
161+
## Dependency Graph
162+
163+
```mermaid
164+
graph TD
165+
EXPORT["export"] --> MODEL["model (lib)"]
166+
EXPORT --> TORCH["LibTorch"]
167+
EXPORT --> JSON["nlohmann/json"]
168+
EXPORT --> SAFETENSORS["safetensors-cpp"]
169+
EXPORT --> CLI11["CLI11"]
170+
EXPORT --> OMP["OpenMP"]
171+
172+
MODEL --> TORCH
173+
174+
INF["inference"] --> CLI11
175+
INF --> OMP
176+
177+
TOK["tokenizer"] --> TORCH
178+
TOK --> TIKTOK["tiktoken-cpp"]
179+
TOK --> CLI11
180+
TOK --> OMP
181+
182+
KMOD["leap_kmod"] --> KHEADERS["Linux Kernel Headers"]
183+
```
184+
185+
---
186+
187+
## Troubleshooting
188+
189+
### LibTorch ABI Mismatch
190+
**Symptom:** Linker errors with `std::__cxx11::basic_string`
191+
**Fix:** Use the **cxx11 ABI** version of LibTorch. If using pip, ensure GCC versions match.
192+
193+
### Kernel Module Load Failure
194+
**Symptom:** `Operation not permitted` on `insmod`
195+
**Fix:** Disable Secure Boot or sign the module.
196+
197+
**Symptom:** `Exec format error`
198+
**Fix:** Module compiled for a different kernel. Rebuild after `sudo apt install linux-headers-$(uname -r)`.
199+
200+
### OpenMP Not Found (macOS)
201+
**Symptom:** CMake error about libomp
202+
**Fix:** `brew install libomp`

0 commit comments

Comments
 (0)