From 454d5ed516f84deb8cc2a3934cae0d1dabd1408f Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 18 Mar 2026 15:17:24 +0000 Subject: [PATCH 1/2] Initial plan From a87e8cdc78f21e1ab9480ee4fcfa838ad7d9a992 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 18 Mar 2026 15:33:00 +0000 Subject: [PATCH 2/2] docs: add MkDocs documentation with Read the Docs theme Co-authored-by: Basigli <84378604+Basigli@users.noreply.github.com> --- .github/workflows/docs.yml | 30 ++++++++ .gitignore | 5 +- docs/architecture.md | 109 ++++++++++++++++++++++++++++ docs/components/client.md | 65 +++++++++++++++++ docs/components/datanode.md | 49 +++++++++++++ docs/components/server.md | 98 ++++++++++++++++++++++++++ docs/configuration.md | 58 +++++++++++++++ docs/contributing.md | 135 +++++++++++++++++++++++++++++++++++ docs/getting-started.md | 137 ++++++++++++++++++++++++++++++++++++ docs/index.md | 33 +++++++++ docs/protocol.md | 96 +++++++++++++++++++++++++ docs/requirements.txt | 1 + mkdocs.yml | 20 ++++++ 13 files changed, 835 insertions(+), 1 deletion(-) create mode 100644 .github/workflows/docs.yml create mode 100644 docs/architecture.md create mode 100644 docs/components/client.md create mode 100644 docs/components/datanode.md create mode 100644 docs/components/server.md create mode 100644 docs/configuration.md create mode 100644 docs/contributing.md create mode 100644 docs/getting-started.md create mode 100644 docs/index.md create mode 100644 docs/protocol.md create mode 100644 docs/requirements.txt create mode 100644 mkdocs.yml diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml new file mode 100644 index 0000000..787667c --- /dev/null +++ b/.github/workflows/docs.yml @@ -0,0 +1,30 @@ +name: Deploy Documentation + +on: + push: + branches: + - main + paths: + - 'docs/**' + - 'mkdocs.yml' + workflow_dispatch: + +permissions: + contents: write + +jobs: + deploy-docs: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.x' + + - name: Install MkDocs + run: pip install -r docs/requirements.txt + + - name: Build and deploy documentation + run: mkdocs gh-deploy --force diff --git a/.gitignore b/.gitignore index 734d2cc..b144571 100644 --- a/.gitignore +++ b/.gitignore @@ -8,4 +8,7 @@ CMakeFiles/* !build/buildGnome.sh !build/buildTestMultiClient.sh !build/buildTestMultiDataNode.sh -!build/buildTestMultiple.sh \ No newline at end of file +!build/buildTestMultiple.sh + +# MkDocs generated site (build artifact) +site/ \ No newline at end of file diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..a6ed41b --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,109 @@ +# Architecture + +Squid Storage is composed of three independent components that communicate over TCP using the [Squid Protocol](protocol.md). + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ CLIENT (GUI) │ +│ SquidStorage ←──────────────────────────────→ SquidStorage │ +│ (primary socket) (secondary socket) │ +└────────────────────────────┬────────────────────────────────────┘ + │ Squid Protocol (TCP) + ▼ +┌────────────────────────────────────────────────────────────────┐ +│ SERVER │ +│ - Metadata (dataNodeReplicationMap) │ +│ - File lock management │ +│ - Heartbeat monitoring │ +│ - select()-based async I/O │ +└───────────┬────────────────────────────────────┬───────────────┘ + │ Squid Protocol (TCP) │ Squid Protocol (TCP) + ▼ ▼ +┌───────────────────────┐ ┌───────────────────────┐ +│ DATANODE 1 │ │ DATANODE 2 │ +│ (file replicas) │ │ (file replicas) │ +└───────────────────────┘ └───────────────────────┘ +``` + +## Components + +### Server (Central Coordinator) + +The **Server** is the single source of truth for metadata. It: + +- Accepts connections from Clients and DataNodes. +- Maintains the `dataNodeReplicationMap` — a mapping of each file to the DataNodes that hold a replica. +- Manages distributed **file locks** so that at most one client writes a file at a time. +- Forwards file operations (create, update, delete) to the relevant DataNodes and broadcasts changes to all connected Clients. +- Sends periodic **heartbeats** to DataNodes to detect failures. +- Calls `rebalanceFileReplication()` when a DataNode goes offline to restore the desired replication level. +- Uses the `select()` system call so it can monitor all sockets concurrently in a single thread. + +More details: [Server Component](components/server.md) + +### DataNode (Storage Node) + +Each **DataNode** is a lightweight storage process. It: + +- Stores file replicas in its working directory. +- Responds to heartbeat pings from the Server. +- Receives file content from the Server when a replication or update is needed. +- Sends file content to the Server when a Client requests a read. + +More details: [DataNode Component](components/datanode.md) + +### Client (GUI Application) + +The **Client** provides a graphical interface backed by Dear ImGui + SDL2 + OpenGL. It: + +- Opens **two TCP connections** to the Server at startup — a *primary* socket for sending commands and a *secondary* socket for receiving asynchronous updates pushed by the Server. +- Lets users create, open (read), edit (update), and delete files. +- Acquires a distributed lock before writing and releases it when done. +- Switches to **read-only mode** when disconnected to prevent inconsistencies. + +More details: [Client Component](components/client.md) + +## Common Shared Code + +The `common/` module contains code shared by all three components: + +| Module | Path | Description | +|--------|------|-------------| +| `SquidProtocol` | `common/src/squidprotocol/` | Message serialisation/deserialisation and send/receive helpers | +| `FileManager` | `common/src/filesystem/` | File CRUD operations and version tracking | +| `FileLock` | `common/src/filesystem/` | Distributed lock data structure | +| `FileTransfer` | `common/src/filesystem/` | Chunked file transfer over a socket | +| `Peer` | `common/src/peer/` | Base class with socket management and reconnection logic | + +## Data Flow — File Update + +``` +Client Server DataNodes + │ │ │ + │── AcquireLock(file) ───────▶│ │ + │◀─ Response(isLocked) ───────│ │ + │ │ │ + │── UpdateFile(file) ────────▶│ │ + │ │── UpdateFile(file) ────────▶│ (replica 1) + │ │── UpdateFile(file) ────────▶│ (replica 2) + │◀─ ACK ──────────────────────│ │ + │ │ │ + │── ReleaseLock(file) ───────▶│ │ + │◀─ ACK ──────────────────────│ │ +``` + +## Replication and Fault Tolerance + +- Each file is replicated across `replicationFactor` DataNodes (default: 2). +- The Server tracks which DataNodes hold each file in `dataNodeReplicationMap`. +- When a DataNode goes offline (missed heartbeat), the Server removes it from the map and calls `rebalanceFileReplication()` to assign new replicas. +- Clients automatically attempt to reconnect in a loop when they lose the server connection. +- While disconnected, files are shown in **read-only mode** to preserve consistency. + +## Consistency Model + +Squid Storage prioritises **consistency over availability**: + +- A Client can only write when it holds a distributed lock **and** is connected to the Server. +- All writes are propagated synchronously to all DataNodes before the Server acknowledges the Client. +- If a component is partitioned, it cannot modify shared state; it can only read its local copy. diff --git a/docs/components/client.md b/docs/components/client.md new file mode 100644 index 0000000..750dc1e --- /dev/null +++ b/docs/components/client.md @@ -0,0 +1,65 @@ +# Client + +The **Client** (`SquidStorage` executable) provides a graphical interface for users to interact with the distributed storage system. It is built with **Dear ImGui**, **SDL2**, and **OpenGL 2**. + +## Starting the Client + +```bash +./SquidStorage [serverIP] [serverPort] +``` + +| Argument | Default | Description | +|----------|---------|-------------| +| `serverIP` | `127.0.0.1` | IP address of the running Server | +| `serverPort` | `12345` | TCP port the Server is listening on | + +## GUI Overview + +The client window shows: + +- **File list panel** — displays all files currently tracked by the distributed storage. +- **New File** button — creates a new empty file. +- **Open** button — reads and displays the selected file's content in an editable text area. +- **Delete** button — deletes the selected file from the distributed storage. + +Editing a file automatically acquires a distributed lock before the write and releases it after the update is committed. + +## Dual-Socket Connection + +When the Client starts it establishes **two TCP connections** to the Server: + +| Socket | Purpose | +|--------|---------| +| **Primary** | Sending commands (CreateFile, UpdateFile, DeleteFile, AcquireLock, …) | +| **Secondary** | Receiving asynchronous push notifications (file updates from other clients) | + +This separation prevents command responses and asynchronous updates from interleaving on the same socket. + +## Supported Operations + +| Operation | Protocol Message | Description | +|-----------|-----------------|-------------| +| Create file | `CreateFile` | Sends the file content to the Server for replication | +| Read file | `ReadFile` | Retrieves a file from the Server (which fetches it from a DataNode) | +| Update file | `UpdateFile` | Sends updated content; Server propagates to DataNodes | +| Delete file | `DeleteFile` | Removes the file from all DataNodes | +| Acquire lock | `AcquireLock` | Requests an exclusive write lock from the Server | +| Release lock | `ReleaseLock` | Frees the write lock after writing | +| Sync status | `SyncStatus` | Retrieves the full list of files and their versions | + +## Disconnected (Read-Only) Mode + +If the Client loses its connection to the Server, it: + +1. Attempts to reconnect in a loop. +2. Switches all files to **read-only** mode until reconnection succeeds. + +This ensures that a partitioned client cannot write stale data and create inconsistencies. + +## Technology Stack + +| Library | Version | Role | +|---------|---------|------| +| **Dear ImGui** | bundled | Immediate-mode GUI rendering | +| **SDL2** | system | Window creation, input handling | +| **OpenGL 2** | system | GPU-accelerated rendering backend | diff --git a/docs/components/datanode.md b/docs/components/datanode.md new file mode 100644 index 0000000..e3de077 --- /dev/null +++ b/docs/components/datanode.md @@ -0,0 +1,49 @@ +# DataNode + +A **DataNode** (`DataNode` executable) is a storage process that holds file replicas and serves them to the Server on request. You can run as many DataNode instances as you like; the Server distributes replicas across all available nodes. + +## Responsibilities + +- Store file replicas in its working directory. +- Receive and persist files sent by the Server (`CreateFile`, `UpdateFile`). +- Delete files on `DeleteFile` requests from the Server. +- Respond to `Heartbeat` pings to signal availability. +- Serve file content to the Server when a Client reads a file (`ReadFile`). + +## Starting a DataNode + +```bash +./DataNode [serverIP] [serverPort] +``` + +Both arguments are optional: + +| Argument | Default | Description | +|----------|---------|-------------| +| `serverIP` | `127.0.0.1` | IP address of the running Server | +| `serverPort` | `12345` | TCP port the Server is listening on | + +The DataNode uses its **current working directory** as the storage root. Run each instance from a different directory so that their file stores do not overlap: + +```bash +mkdir -p /var/storage/dn1 && (cd /var/storage/dn1 && /path/to/DataNode 192.168.1.10 12345) +mkdir -p /var/storage/dn2 && (cd /var/storage/dn2 && /path/to/DataNode 192.168.1.10 12345) +``` + +## Lifecycle + +1. **Connect** — the DataNode establishes a TCP connection to the Server. +2. **Identify** — the Server sends an `Identify` request; the DataNode responds with `nodeType=DATANODE` and its `processName`. +3. **Receive files** — the Server pushes file replicas as `CreateFile` / `UpdateFile` / `DeleteFile` messages. +4. **Heartbeat** — the Server periodically sends `Heartbeat` messages; the DataNode replies with `ACK`. +5. **Reconnect** — if the connection drops, the DataNode continuously retries the connection in a loop. + +## Storage Layout + +Files are stored relative to the DataNode's working directory, mirroring the path used in the Squid Protocol messages. For example, a file with `filePath:/squidstorage/report.txt` is stored as `./squidstorage/report.txt` inside the DataNode's directory. + +The `FileManager` component (from `common/`) tracks file versions using a `.fileVersion.txt` metadata file so that stale replicas can be detected and updated. + +## Version Tracking + +Each file has an integer version number. When the Server rebalances replicas, it includes the current version so the receiving DataNode can verify it has the latest content. The `SyncStatus` message allows the Server to compare file version maps across DataNodes. diff --git a/docs/components/server.md b/docs/components/server.md new file mode 100644 index 0000000..fa165b6 --- /dev/null +++ b/docs/components/server.md @@ -0,0 +1,98 @@ +# Server + +The **Server** (`SquidStorageServer`) is the central coordinator of Squid Storage. It manages all metadata, routes file operations between Clients and DataNodes, and monitors the health of every connected node. + +## Responsibilities + +- Accept incoming TCP connections from Clients and DataNodes. +- Route and propagate file operations (create, read, update, delete). +- Maintain the `dataNodeReplicationMap` — which DataNodes hold replicas of each file. +- Manage distributed **file locks** and check for lock expiry. +- Send periodic **heartbeats** to DataNodes; remove unresponsive nodes from the replication map. +- Rebalance file replicas when the actual replication count falls below the configured factor. +- Use `select()` to handle all sockets concurrently without spawning per-connection threads. + +## Starting the Server + +```bash +./SquidStorageServer [port] [replicationFactor] [timeoutSeconds] +``` + +All arguments are optional and fall back to defaults: + +| Argument | Default | Description | +|----------|---------|-------------| +| `port` | `12345` | TCP port to listen on | +| `replicationFactor` | `2` | Number of DataNode replicas per file | +| `timeoutSeconds` | `60` | Socket operation timeout in seconds | + +## Internal State + +### `dataNodeReplicationMap` + +``` +map> +``` + +Tracks, for every file, which DataNodes currently hold a replica and the corresponding protocol endpoint. When a DataNode disconnects, it is erased from this map and `rebalanceFileReplication()` is triggered. + +### `clientEndpointMap` + +``` +map> +``` + +Each Client opens two connections to the Server. The Server stores both so it can push asynchronous updates through the secondary socket. + +### `dataNodeEndpointMap` + +``` +map +``` + +Active DataNode connections indexed by name. + +### `fileLockMap` + +``` +map +``` + +Stores the current lock holder and lock timestamp for every file. `checkFileLockExpiration()` periodically releases stale locks (default interval: 5 minutes). + +## Key Methods + +| Method | Description | +|--------|-------------| +| `run()` | Main event loop — calls `select()` and dispatches incoming messages | +| `handleAccept()` | Handles a new incoming connection; performs the handshake | +| `handleConnection()` | Processes a received message from a Client or DataNode | +| `propagateCreateFile()` | Sends a `CreateFile` message to the selected DataNodes | +| `propagateUpdateFile()` | Sends an `UpdateFile` message to all DataNodes holding the file | +| `propagateDeleteFile()` | Sends a `DeleteFile` message to all DataNodes holding the file | +| `getFileFromDataNode()` | Retrieves a file from a DataNode and forwards it to a Client | +| `sendHeartbeats()` | Pings all DataNodes; removes those that do not respond | +| `rebalanceFileReplication()` | Assigns new DataNodes to a file whose replica count is below the threshold | +| `acquireLock()` / `releaseLock()` | Grants or releases a distributed write lock for a file | +| `checkFileLockExpiration()` | Periodically releases locks that have exceeded the timeout | + +## Connection Handshake + +When a new socket connects, the Server sends an `Identify` request. The peer responds with its `nodeType` (`CLIENT` or `DATANODE`) and its `processName`. Based on the type the Server: + +- **Client**: records both the primary socket (for commands) and waits for a second connection to be registered as the secondary (update) socket. +- **DataNode**: registers the endpoint in `dataNodeEndpointMap` and starts including it in replication decisions. + +## Replication Flow + +1. A Client calls `CreateFile` or `UpdateFile`. +2. The Server selects `replicationFactor` DataNodes from `dataNodeEndpointMap` using round-robin. +3. The Server forwards the file to each selected DataNode via `propagateCreateFile()` / `propagateUpdateFile()`. +4. The Server updates `dataNodeReplicationMap` to record the new assignments. +5. The Server broadcasts the change to all other connected Clients. + +## Fault Tolerance + +- The `sendHeartbeats()` loop runs in a background thread. +- Any DataNode that fails to respond is removed from `dataNodeEndpointMap` and `dataNodeReplicationMap`. +- `rebalanceFileReplication()` is immediately invoked for every file that lost a replica, pushing a fresh copy to a healthy DataNode. diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..11d35a2 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,58 @@ +# Configuration + +All three Squid Storage executables accept runtime configuration via command-line arguments. No configuration files are required. + +## Server + +```bash +./SquidStorageServer [port] [replicationFactor] [timeoutSeconds] +``` + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `port` | integer | `12345` | TCP port the server listens on | +| `replicationFactor` | integer | `2` | Number of DataNode replicas maintained per file. A higher value improves fault tolerance but increases storage usage. | +| `timeoutSeconds` | integer | `60` | Socket operation timeout in seconds. Connections that do not respond within this window are considered failed. | + +There are also two compile-time constants in `server/src/server.hpp`: + +| Constant | Default | Description | +|----------|---------|-------------| +| `DEFAULT_LOCK_INTERVAL` | `5` (minutes) | How frequently stale file locks are checked and released | +| `DEFAULT_PATH` | `./test_txt/test_server` | Default working directory for the server's own file store | +| `BUFFER_SIZE` | `1024` bytes | Read buffer size for socket I/O | + +## DataNode + +```bash +./DataNode [serverIP] [serverPort] +``` + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `serverIP` | string | `127.0.0.1` | IP address of the Server to connect to | +| `serverPort` | integer | `12345` | TCP port of the Server | + +The DataNode uses its **current working directory** as the storage root. Set this by running the process from the desired directory. + +## Client + +```bash +./SquidStorage [serverIP] [serverPort] +``` + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `serverIP` | string | `127.0.0.1` | IP address of the Server to connect to | +| `serverPort` | integer | `12345` | TCP port of the Server | + +## Choosing the Replication Factor + +| Replication Factor | Meaning | Minimum DataNodes Required | +|--------------------|---------|---------------------------| +| `1` | No redundancy — file exists on exactly one DataNode | 1 | +| `2` | Can survive the loss of one DataNode (default) | 2 | +| `3` | Can survive the simultaneous loss of two DataNodes | 3 | +| `N` | Can survive the loss of N−1 DataNodes | N | + +If the number of available DataNodes is less than the replication factor, the Server replicates to as many nodes as are available and will automatically increase replicas as new nodes join. diff --git a/docs/contributing.md b/docs/contributing.md new file mode 100644 index 0000000..78d8d02 --- /dev/null +++ b/docs/contributing.md @@ -0,0 +1,135 @@ +# Contributing + +Thank you for your interest in contributing to Squid Storage! Please follow the conventions below to keep the project consistent and maintainable. + +## Commit Conventions + +Follow the **Conventional Commits** format: + +``` +(type)scope: message +``` + +**Example:** + +```bash +git commit -m "(feat)server: add round-robin DataNode selection" +git commit -m "(fix)protocol: handle empty file path argument" +git commit -m "(docs)readme: add architecture diagram" +``` + +### Commit Types + +| Type | When to use | +|------|-------------| +| `feat` | New feature | +| `chore` | Maintenance, dependency updates | +| `refactor` | Code restructuring without behaviour change | +| `fix` | Bug fix | +| `docs` | Documentation only | +| `test` | Adding or updating tests | +| `misc` | Anything that does not fit another type | + +Keep each commit focused on **one logical change**. Do not mix feature work and refactoring in the same commit. + +## Branching Strategy + +Create a **feature branch** for every new development phase: + +```bash +git checkout -b feature/my-feature +``` + +When a milestone is reached, create a **release branch**: + +```bash +git checkout -b release/v1.2.0 +``` + +## Versioning + +Squid Storage follows [Semantic Versioning](https://semver.org/): + +``` +MAJOR.MINOR.PATCH +``` + +| Segment | Increment when… | +|---------|----------------| +| `MAJOR` | Breaking changes to the API or protocol | +| `MINOR` | New backwards-compatible features | +| `PATCH` | Backwards-compatible bug fixes | + +### Creating a Tag and Release + +```bash +# Create an annotated tag +git tag -a v1.2.0 -m "Add replication rebalancing improvements" +git push origin v1.2.0 + +# Delete a tag (if needed) +git tag -d v1.2.0 +git push origin --delete v1.2.0 +``` + +Then go to **GitHub → Releases → New Release** and attach the tag. + +## Changelog + +Consider writing a `CHANGELOG.md` for every new release. You can auto-generate one from commit messages: + +```bash +git log --pretty=format:"- %s" v1.1.0..v1.2.0 >> CHANGELOG.md +``` + +Or write it manually using this structure: + +```markdown +# Changelog + +## [1.2.0] - 2025-06-01 +### Added +- Replication rebalancing on DataNode reconnect. + +### Fixed +- Race condition in file lock expiration check. + +## [1.1.0] - 2025-04-15 +### Added +- Secondary socket for asynchronous client updates. +``` + +## Building and Testing + +Before submitting a pull request, make sure the project builds cleanly on your platform: + +```bash +cmake -B build +cmake --build build +``` + +Run the integration tests using the scripts in `test/`: + +```bash +# Requires tmux +test/testMultiple.sh +``` + +## Documentation + +Documentation is built with [MkDocs](https://www.mkdocs.org/) using the **readthedocs** theme. To preview changes locally: + +```bash +pip install mkdocs +mkdocs serve +``` + +Then open [http://127.0.0.1:8000](http://127.0.0.1:8000) in your browser. + +To build the static site: + +```bash +mkdocs build +``` + +The generated site is placed in the `site/` directory. diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..2d10b25 --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,137 @@ +# Getting Started + +This page explains how to build Squid Storage from source and run the three components — Server, DataNode, and Client. + +## Prerequisites + +| Dependency | Version | Notes | +|------------|---------|-------| +| C++ compiler | C++17 or later | GCC or Clang | +| CMake | 3.10+ | Build system | +| SDL2 | any recent | GUI windowing | +| OpenGL | system | GUI rendering | + +### Install dependencies on Ubuntu + +```bash +sudo apt-get update +sudo apt-get install -y cmake libsdl2-dev libglu1-mesa-dev freeglut3-dev mesa-common-dev +``` + +### Install dependencies on macOS + +```bash +brew install cmake sdl2 +``` + +OpenGL is provided by the system framework on macOS and does not require a separate install. + +## Building + +Clone the repository and build with CMake: + +```bash +git clone https://github.com/Basigli/Squid-Storage.git +cd Squid-Storage + +cmake -B build +cmake --build build +``` + +Three executables will be produced inside the `build/` directory: + +| Executable | Description | +|------------|-------------| +| `SquidStorageServer` | Central coordinator server | +| `DataNode` | Storage node | +| `SquidStorage` | GUI client | + +## Running + +### 1. Start the Server + +The server must be started first. All other components connect to it. + +```bash +./build/SquidStorageServer [port] [replicationFactor] [timeoutSeconds] +``` + +**Example — default settings:** + +```bash +./build/SquidStorageServer +# Starting server on port: 12345, replication factor: 2, timeout: 60s +``` + +**Example — custom settings:** + +```bash +./build/SquidStorageServer 8080 3 120 +# Starting server on port: 8080, replication factor: 3, timeout: 120s +``` + +See [Configuration](configuration.md) for a full description of each parameter. + +### 2. Start one or more DataNodes + +Each DataNode connects to the running server. Start each instance from its own directory so that files are stored separately. + +```bash +mkdir -p datanode1 && cd datanode1 +../build/DataNode [serverIP] [serverPort] +``` + +**Example:** + +```bash +mkdir -p datanode1 && (cd datanode1 && ../build/DataNode 127.0.0.1 12345) +mkdir -p datanode2 && (cd datanode2 && ../build/DataNode 127.0.0.1 12345) +``` + +The DataNode uses the current working directory as its storage root. + +### 3. Start the Client + +```bash +./build/SquidStorage [serverIP] [serverPort] +``` + +**Example:** + +```bash +./build/SquidStorage 127.0.0.1 12345 +``` + +The GUI window will open. You can create, open, edit, and delete files that are automatically replicated across all connected DataNodes. + +## Full Local Cluster Example + +The following sequence starts a complete local cluster with one server, two DataNodes, and one client: + +```bash +# Terminal 1 — server +./build/SquidStorageServer 12345 2 60 + +# Terminal 2 — datanode 1 +mkdir -p /tmp/dn1 && (cd /tmp/dn1 && /path/to/build/DataNode 127.0.0.1 12345) + +# Terminal 3 — datanode 2 +mkdir -p /tmp/dn2 && (cd /tmp/dn2 && /path/to/build/DataNode 127.0.0.1 12345) + +# Terminal 4 — client +./build/SquidStorage 127.0.0.1 12345 +``` + +The replication factor is set to `2`, so each file will be stored on both DataNodes. + +## Test Scripts + +The `test/` directory contains shell scripts that automate a multi-component test using **tmux**: + +```bash +test/testMultiple.sh # server + datanodes + clients +test/testMultiClient.sh # multiple clients only +test/testMultiDataNode.sh # multiple datanodes only +``` + +These scripts require `tmux` to be installed. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..c306e43 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,33 @@ +# Squid Storage + +[![CMake Build (Ubuntu + macOS)](https://github.com/MHS-20/Squid-Storage/actions/workflows/cmake-multi-platform.yml/badge.svg)](https://github.com/MHS-20/Squid-Storage/actions/workflows/cmake-multi-platform.yml) + +**Squid Storage** is a distributed storage system designed to manage file replication and consistency of a specific folder across multiple nodes. + +It provides: + +- **Automatic replication** — files are replicated across multiple DataNodes for fault tolerance. +- **Distributed file locking** — prevents concurrent writes and data corruption. +- **Graphical client interface** — users can create, read, update, and delete files through a GUI. +- **Partition tolerance** — every component keeps running and attempts reconnection if disconnected. +- **Real-time propagation** — changes are immediately broadcast to all connected clients and DataNodes. + +## Key Components + +| Component | Role | +|-----------|------| +| **Server** | Central coordinator — manages file metadata, client/DataNode connections, and replication | +| **DataNode** | Storage node — stores file replicas and responds to server requests | +| **Client** | GUI application — lets users interact with the distributed storage system | + +## Quick Links + +- [Getting Started](getting-started.md) — build and run Squid Storage +- [Architecture](architecture.md) — how the system is designed +- [Squid Protocol](protocol.md) — the custom wire protocol +- [Configuration](configuration.md) — runtime parameters +- [Contributing](contributing.md) — how to contribute + +## Platform Support + +Squid Storage is continuously built and tested on **Ubuntu** and **macOS** via GitHub Actions. diff --git a/docs/protocol.md b/docs/protocol.md new file mode 100644 index 0000000..ff8f951 --- /dev/null +++ b/docs/protocol.md @@ -0,0 +1,96 @@ +# Squid Protocol + +The **Squid Protocol** is a lightweight, text-based application protocol used for all communication between Clients, the Server, and DataNodes. + +## Message Format + +Every message follows the pattern: + +``` +Keyword +``` + +Arguments are comma-separated key-value pairs enclosed in angle brackets. Messages with no arguments still include the angle brackets: + +``` +Heartbeat<> +``` + +### Example Messages + +``` +CreateFile +UpdateFile +AcquireLock +Response +Response +Identify<> +Response +``` + +## Message Reference + +| Keyword | Direction | Arguments | Response | +|---------|-----------|-----------|----------| +| `CreateFile` | Client → Server, Server → DataNode | `filePath` | `ACK` | +| `TransferFile` | Server → DataNode | `filePath` | `ACK` | +| `ReadFile` | Client → Server | `filePath` | `ACK` + file content | +| `UpdateFile` | Client → Server, Server → DataNode | `filePath` | `ACK` | +| `DeleteFile` | Client → Server, Server → DataNode | `filePath` | `ACK` | +| `AcquireLock` | Client → Server | `filePath` | `isLocked` (true/false) | +| `ReleaseLock` | Client → Server | `filePath` | `ACK` | +| `Heartbeat` | Server → DataNode | — | `ACK` | +| `SyncStatus` | Client/DataNode → Server | — | `ACK` + file version map | +| `Identify` | Server → peer | — | `nodeType`, `processName` | +| `Response` | any | `ACK` / `nodeType,processName` / `isLocked` | — | +| `Close` | any | — | `ACK` | + +## Argument Details + +### `filePath` + +The path of the file inside the shared storage namespace. Always an absolute-style path beginning with `/`, for example `/squidstorage/report.txt`. + +### `isLocked` + +Boolean string returned by `AcquireLock`. `true` means the lock was successfully granted; `false` means another client currently holds it. + +### `nodeType` + +Returned in an `Identify` response. One of: + +- `CLIENT` +- `DATANODE` + +### `processName` + +A human-readable identifier for the connecting process (e.g. a DataNode instance name or client window title). Used by the Server to populate its endpoint maps. + +## File Transfer + +File content is transmitted immediately after the protocol message using a **length-prefixed** binary transfer: + +1. The sender transmits the message (e.g. `CreateFile`). +2. The sender then transmits the file length as a 4-byte big-endian integer. +3. The sender transmits the raw file bytes. +4. The receiver sends a `Response`. + +The `FileTransfer` class in `common/src/filesystem/filetransfer.cpp` implements both sides. + +## Versioned Operations + +`CreateFile` and `UpdateFile` can optionally carry a `version` argument when the Server needs to push a specific version to a DataNode during rebalancing: + +``` +CreateFile +``` + +## Implementation + +The protocol is implemented in: + +| File | Description | +|------|-------------| +| `common/src/squidprotocol/squidprotocol.hpp/.cpp` | `SquidProtocol` class — send/receive helpers for every message type | +| `common/src/squidprotocol/squidProtocolFormatter.hpp/.cpp` | Serialises and deserialises `Message` objects to/from the text wire format | +| `server/src/squidProtocolServer.cpp` | Server-side request dispatcher | diff --git a/docs/requirements.txt b/docs/requirements.txt new file mode 100644 index 0000000..f6549e1 --- /dev/null +++ b/docs/requirements.txt @@ -0,0 +1 @@ +mkdocs>=1.6.0 diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..51212a9 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,20 @@ +site_name: Squid Storage +site_description: A distributed storage system for managing file replication and consistency across multiple nodes. +site_author: Squid Storage Contributors +repo_url: https://github.com/Basigli/Squid-Storage +repo_name: Basigli/Squid-Storage + +theme: + name: readthedocs + +nav: + - Home: index.md + - Getting Started: getting-started.md + - Architecture: architecture.md + - Components: + - Server: components/server.md + - DataNode: components/datanode.md + - Client: components/client.md + - Squid Protocol: protocol.md + - Configuration: configuration.md + - Contributing: contributing.md