The core of a high-load P2P Message broker

I am sharing with you a draft of writing a core for a p2p messenger. An unsuccessful attempt to write a p2p messenger. Initially, I quickly implemented all the logic: stun, hole punching, p2p messaging, and messaging in case any of the users are offline. The code was clean and worked. I started writing on windows because I didn't have the opportunity to test it on android devices. It seemed like I could keep writing the messenger, add profile changes, improve the user experience, add languages, themes, and so on, but I started porting this core to android, which turned out to be a very difficult challenge. As a result, I spent more time porting than implementing it on Windows, all the logic broke down and I had to fix it, rewriting literally every function. I came to the conclusion that it's easier to leave the UI ready and write the logic from scratch, because due to the rush, the code began to be duplicated, turned into mush and became indistinct.

Note: If you want to see the working code, please roll back to the commit before Android porting started. The documentation below reflects the intended architecture of the project.

---

🛡️ Core Philosophy: Zero-Trust & Privacy

The main idea of the project is ** the server is only a temporary intermediary**.

• The server does not store the correspondence: All messages and profile data live only on users' devices.
• Security in case of hacking: Even if the server is completely compromised, the attacker will not gain access to the message history, as they are encrypted end-to-end (E2EE) and are deleted from the server immediately after delivery.
• P2P Priority: The system always tries to establish a direct connection. The server is used only for exchanging IP addresses (Signal Server) and temporarily storing encrypted "blobs" if the recipient is offline.

🏗 Architecture & Modules

The project is divided into logical blocks that ensure operation in difficult network conditions (NAT, mobile networks):

1. Network & P2P (The "Core")

• UnifiedP2PManager: The brain of the system. Manages the message lifecycle, encrypts the data, and decides whether to send it via UDP directly or to look for other ways.
• StunManager: Responsible for detecting the user's public IP and port (STUN protocol), which is critical for NAT traversal.
• HolePunchManager: Implements the UDP Hole Punching technique to punch a connection between two devices behind a NAT.

2. Message Delivery

• OfflineMessageManager: If a P2P connection is not possible, the message is encrypted on the sender's side and transmitted to the server as an "encrypted packet".
• MessageDeliveryManager: Monitors delivery confirmations (ACKs) and retries (retries).

3. Data & Crypto

• OfflineMessageManager: If a P2P connection is not possible, the message is encrypted on the sender's side and transmitted to the server as an "encrypted packet".
• MessageDeliveryManager: Monitors delivery confirmations (ACKs) and retries (retries).

Data Storage

• SQLite: A local database on the client side for storing history and contacts.
• PostgreSQL: A server-side database for managing offline message queues and user registration.

Development Environment (IDE)

• Visual Studio 2022: The main Windows development environment.
• Android Studio: Used for debugging and building for mobile platforms.
• Qt Creator: A tool for working with the Qt UI and resources.

Core & Frameworks

• C++ 17 / ** CMake** (Build System)
• Qt 5 & Qt 6: Cross-platform framework (both versions were used during migration).
• libsodium 1.0.20: A modern cryptographic library for implementing E2EE (end-to-end encryption).

📂 Project Structure

project_root/
├── client/
│   ├── network/            # P2P, STUN, Hole Punching logic
│   ├── cryptoAlgorithms/   # E2E Encryption wrappers
│   ├── dbManager/          # Local SQLite storage logic
│   └── ui/                 # Qt-based interface
├── server/
│   ├── ServerManager       # UDP Signal Server
│   └── DbManager           # Temporary offline storage (encrypted)
└── configs/                # Network & port configurations

Preview

⚠️ Known Issues (The "Android Chaos")

Porting to Android has made destructive changes:

• Socket conflicts on mobile interfaces.
• The lifecycle of Android background processes terminates P2P sessions.
• Code duplication due to attempts to quickly adapt the logic to mobile specifics.

I want to write a clean working project right for android, and I'm leaving this failed option to you for inspiration.

The most important thing is that P2P correspondence works