Skip to content

Nirvanjha2004/Distributed-key-value-pair

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

This is a high-quality project for an undergraduate portfolio. To make it stand out, the README needs to explain the why (architectural decisions) as much as the how (how to run it).

Here is a professional README.md template tailored specifically to your code and the C++ systems programming concepts you've used.


Distributed KV Store & L4 Load Balancer

A high-performance, distributed Key-Value database built from scratch in C++. This system features Leader-Follower replication, a custom Layer 4 Load Balancer that acts as an HTTP-to-TCP Gateway, and crash consistency via file-based persistence.

🚀 Overview

This project demonstrates core backend engineering principles:

  • Networking: Socket programming using TCP/IP.
  • Concurrency: Multi-threaded request handling with std::thread and std::mutex.
  • Consistency: Asynchronous replication from Leader to Follower.
  • DevOps: Full containerization using Docker and Docker Compose.

🏗️ Architecture

1. The Load Balancer (Port 5000)

Acting as the entry point, the LB performs Request Routing based on the operation type:

  • Writes (SET): Forwarded to the Leader (Port 8080).
  • Reads (GET): Forwarded to the Follower (Port 8081) to scale read heavy workloads.
  • Protocol Translation: It accepts HTTP requests (e.g., via a browser) and translates them into a custom Raw TCP protocol for the backend nodes.

2. The Storage Nodes (Leader & Follower)

  • Memory-First: Uses an std::map for $O(\log n)$ lookup speeds.
  • Persistence: Every SET operation is appended to a port-specific .txt file to ensure data isn't lost on restart.
  • Replication Loop: When the Leader receives a write, it successfully updates its own state and then opens a socket to the Follower to propagate the change.

🛠️ Technical Stack

  • Language: C++17
  • Platform: Linux / WSL (Uses sys/socket.h)
  • Concurrency: Pthreads/C++ Threads
  • Tools: GNU Make, Docker, Docker Compose

🚦 Getting Started

Prerequisites

  • Docker & Docker Compose (Recommended)
  • OR GCC/G++ and Make (if running locally on Linux/WSL)

Option A: The "One-Click" Run (Docker)

This is the easiest way to see the distributed system in action.

# Clone the repo
git clone <your-repo-link>
cd RedisClone

# Start the cluster
docker-compose up --build

Option B: Local Manual Build

mkdir bin data
make
./bin/myserver 8080 &  # Start Leader
./bin/myserver 8081 &  # Start Follower
./bin/mylb             # Start Load Balancer

🧪 Testing the System

1. Perform a Write (To Leader)

Open your browser or use curl:

curl "http://localhost:5000/set?key=username&value=nirvan"

*Response: OK*

2. Perform a Read (From Follower)

The LB will automatically route this to the follower node.

curl "http://localhost:5000/get?key=username"

*Response: nirvan*

3. Verify Persistence

Check the data/ directory. You will see db_8080.txt and db_8081.txt containing the synced key-value pairs.


📈 Future Roadmap

  • Consistent Hashing: To support $N$ number of follower nodes.
  • WAL (Write-Ahead Logging): For better crash recovery.
  • Heartbeat Mechanism: To allow the Load Balancer to detect node failures in real-time.

📄 License

This project is open-source and available under the MIT License.


How to use this:

  1. Copy the text above into a file named README.md in your project root.
  2. Screenshot: Take a screenshot of your terminal running the project and replace the [Image of...] tag with your actual image.
  3. Upload: Push it to GitHub. GitHub will automatically render this beautifully.

Next Step: Would you like me to help you create a LinkedIn post draft to share this project with your network?

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors