diff --git a/.gitignore b/.gitignore index 31e2bf0a..a798d092 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,72 @@ +### Linux template +*~ + +# temporary files which can be created if a process still has a handle open of a deleted file +.fuse_hidden* + +# KDE directory preferences +.directory + +# Linux trash folder which might appear on any partition or disk +.Trash-* + +# .nfs files are created when an open file is removed but is still being accessed +.nfs* + +### Windows template +# Windows thumbnail cache files +Thumbs.db +Thumbs.db:encryptable +ehthumbs.db +ehthumbs_vista.db + +# Dump file +*.stackdump + +# Folder config file +[Dd]esktop.ini + +# Recycle Bin used on file shares +$RECYCLE.BIN/ + +# Windows Installer files +*.cab +*.msi +*.msix +*.msm +*.msp + +# Windows shortcuts +*.lnk + +### macOS template +# General +.DS_Store +.AppleDouble +.LSOverride + +# Icon must end with two \r +Icon + +# Thumbnails +._* + +# Files that might appear in the root of a volume +.DocumentRevisions-V100 +.fseventsd +.Spotlight-V100 +.TemporaryItems +.Trashes +.VolumeIcon.icns +.com.apple.timemachine.donotpresent + +# Directories potentially created on remote AFP share +.AppleDB +.AppleDesktop +Network Trash Folder +Temporary Items +.apdisk + crps_*.csv test_miner_data.json @@ -169,8 +238,10 @@ testing/ .vscode/settings.json .idea/ -# Ignore macOS system files -.DS_Store - # History files prediction*.json + +### Misc + +miner.local.config.js +validator.local.config.js diff --git a/README.md b/README.md index fcea0706..f2f71c27 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,37 @@ - -# Synth Subnet - -## Intro +

+ + Rounded Mode icon + +

+ +

+ Synth Subnet +

+ +

+ + GitHub License + +

+ +### Table Of Contents + +* [1. Overview](#-1-overview) + - [1.1. Introduction](#11-introduction) + - [1.2. Task Presented to the Miners](#12-task-presented-to-the-miners) + - [1.3. Validator's Scoring Methodology](#13-validators-scoring-methodology) + - [1.4. Calculation of Leaderboard Score](#14-calculation-of-leaderboard-score) + - [1.5. Overall Purpose](#15-overall-purpose) +* [2. Usage](#-2-usage) + - [2.1. Requirements](#21-requirements) + - [2.2. Setup](#22-setup) + - [2.3. Miners](#23-miners) + - [2.4. Validators](#24-validators) +* [3. License](#-3-license) + +## 🔭 1. Overview + +### 1.1. Introduction The Synth Subnet leverages Bittensor’s decentralized intelligence network to create the world's most powerful synthetic data for price forecasting. Unlike traditional price prediction systems that focus on single-point forecasts, Synth specializes in capturing the full distribution of possible price movements and their associated probabilities, to build the most accurate synthetic data in the world. @@ -11,8 +41,9 @@ Validators score miners on short-term and long-term prediction accuracy, with re The Synth Subnet aims to become a key source of synthetic price data for AI Agents and the go-to resource for options trading and portfolio management, offering valuable insights into price probability distributions. +[Back to top ^][table-of-contents] -## 1. Task Presented to the Miners +### 1.2. Task Presented to the Miners Miners are tasked with providing probabilistic forecasts of a cryptocurrency's future price movements. Specifically, each miner is required to generate multiple simulated price paths for an asset, from the current time over specified time increments and time horizon. Initially all checking prompts will be to produce 100 simulated paths for the future price of bitcoin at 5-minute time increments for the next 24 hours. @@ -41,18 +72,17 @@ where: - $N = \dfrac{T}{\Delta t}$ is the total number of increments. - We recommend the miner sends a request to the Pyth Oracle to acquire the price of the asset at the start_time. If they fail to return predictions by the start_time or the predictions are in the wrong format, they will be scored 0 for that prompt. +[Back to top ^][table-of-contents] -## 2. Validator's Scoring Methodology +### 1.3. Validator's Scoring Methodology The role of the validators is, after the time horizon as passed, to judge the accuracy of each miner’s predicted paths compared to how the price moved in reality. The validator evaluates the miners' probabilistic forecasts using the Continuous Ranked Probability Score (CRPS). The CRPS is a proper scoring rule that measures the accuracy of probabilistic forecasts for continuous variables, considering both the calibration and sharpness of the predicted distribution. The lower the CRPS, the better the forecasted distribution predicted the observed value. - -### Application of CRPS to Ensemble Forecasts +#### Application of CRPS to Ensemble Forecasts In our setup, miners produce ensemble forecasts by generating a finite number of simulated price paths rather than providing an explicit continuous distribution. The CRPS can be calculated directly from these ensemble forecasts using an empirical formula suitable for finite samples. @@ -70,8 +100,7 @@ where: This formulation allows us to assess the miners' forecasts directly from their simulated paths without the need to construct an explicit probability distribution. - -### Application to Multiple Time Increments +#### Application to Multiple Time Increments To comprehensively assess the miners' forecasts, the CRPS is applied to sets of price changes over different time increments. These increments include short-term and long-term intervals (in the case of the initial checking prompt parameters, these will be 5 minutes, 30 minutes, 3 hours, 24 hours). @@ -82,10 +111,11 @@ For each time increment: The final score for a miner for a single checking prompt is the sum of these CRPS values over all the time increments. +[Back to top ^][table-of-contents] -## 3. Calculation of Leaderboard Score +### 1.4. Calculation of Leaderboard Score -### Normalization Using Softmax Function +#### Normalization Using Softmax Function After calculating the sum of the CRPS values, the validator normalizes these scores across all miners who submitted correctly formatted forecasts prior to the start time. The normalized score $S_i$ for miner $i$ is calculated as: @@ -100,8 +130,7 @@ where: Any miners who didn’t submit a correct prediction are allocated a normalised score of 0 for that prompt. - -### Exponentially Decaying Time-Weighted Average (Leaderboard Score) +#### Exponentially Decaying Time-Weighted Average (Leaderboard Score) The validator is required to store the historic request scores for each miner. After each new request is scored, the validator recalculates the ‘leaderboard score’ for each miner, using an exponentially decaying time-weighted average over their past **per request** scores, up to a threshold of 30 days in the past. @@ -120,8 +149,7 @@ where: - $\lambda = \dfrac{\ln 2}{h}$ is the decay constant, with half-life $h = 10$ days. - The sum runs over all requests $j$ such that $t - t_j \leq T$, where $T = 30$ days is the threshold time. - -### Allocation of Emissions +#### Allocation of Emissions At the end of each day, the leaderboard scores are then raised to the power of an exponent $\alpha$ (e.g., $\alpha = 2$) to amplify performance differences. The adjusted scores determine each miner's share of the total emissions for that day Adjusted Scores: @@ -136,9 +164,9 @@ $$ P_{i,t} = \frac{AdjScore_{i,t}}{\sum_j AdjScore_{j,t}} \times Total $$ +[Back to top ^][table-of-contents] - -## Overall Purpose +### 1.5. Overall Purpose The system creates a competitive environment through: @@ -157,191 +185,91 @@ The system creates a competitive environment through: 5. **Calculating Leaderboard Scores and Allocating Emissions** - Rewards consistent performance and encourages competition +[Back to top ^][table-of-contents] -## Environment Setup Instructions +## 🪄 2. Usage -### Step 1: Create a Wallet -You can create a single wallet or multiple wallets, depending on how many miners or validators you intend to run. -There is no functional difference between wallets for miners or validators, and you can name them as you prefer. - -#### Create a Miner Wallet -- create a coldkey -``` -btcli wallet new_coldkey --wallet.name miner -``` - -- create a hotkey -``` -btcli wallet new_hotkey --wallet.name miner --wallet.hotkey default -``` - -#### Create a Validator Wallet -- create a coldkey -``` -btcli wallet new_coldkey --wallet.name validator -``` - -- create a hotkey -``` -btcli wallet new_hotkey --wallet.name validator --wallet.hotkey default -``` +### 2.1. Requirements ---- +* [Git](https://github.com/git-guides/install-git) +* [Ubuntu v20.04+](https://ubuntu.com/download) -### Step 2: Register Wallet in the Subnet -Before starting a miner or validator, you must acquire a slot in the subnet. This connects your application to the Bittensor subnet. +[Back to top ^][table-of-contents] -Register a miner wallet: -``` -btcli subnet register --wallet.name miner --wallet.hotkey default --subtensor.network test -``` +### 2.2. Setup -Register a validator wallet: -``` -btcli subnet register --wallet.name validator --wallet.hotkey default --subtensor.network test +1. Clone the repository: +```shell +git clone https://github.com/mode-network/synth-subnet.git ``` ---- - -### Step 3: Verify Registration -You can check wallet details and verify successful registration in the subnet. - -Miner wallet details: -``` -btcli wallet overview --wallet.name miner --subtensor.network test +2. Change directory to the project root: +```shell +cd ./synth-subnet ``` -Validator wallet details: +3. Add the required repositories: +```shell +sudo add-apt-repository ppa:deadsnakes/ppa ``` -btcli wallet overview --wallet.name validator --subtensor.network test -``` - ---- +> ⚠️ **NOTE:** The [deadsnakes](https://github.com/deadsnakes) repository, while unofficial, it is hugely popular and used by many Python projects. -### Step 4: Check Network Metagraph -You can explore the metagraph of the network for additional insights: +4. Install Rust: +```shell +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` -btcli subnet metagraph --netuid 247 --subtensor.network test -``` - ---- -### Step 5: Install PM2 -PM2 is recommended for running miner and validator processes. +5. Install the dependencies: -#### Update and install Node.js and npm: -``` -sudo apt update -sudo apt install nodejs npm +```shell +sudo apt update && \ + sudo apt install nodejs npm python3.10 pkg-config ``` -#### Install PM2 globally: -``` +6. Install [PM2](https://pm2.io/): +```shell sudo npm install pm2 -g ``` -#### Verify the installation: -``` -pm2 --version -``` - ---- - -### Step 6: Clone the Synth Repository -Clone the repository containing the example miner and validator code. This repository also provides a base template you can customize. -``` -git clone https://github.com/mode-network/synth-subnet.git -``` - ---- - -### Step 7: Set Up Python Virtual Environment -Install Python 3.9 (ensure no conflicts with existing Python versions, do not uninstall or upgrade if you have the default 3.8 or any other in the system): -``` -sudo apt install python3.9 +7. Install the Python virtual environment: +```shell +sudo apt install python3.10-venv ``` -Install the venv package: -``` -sudo apt install python3.9-venv +8. Create a new virtual environment: +```shell +python3.10 -m venv bt_venv ``` -Create and activate the virtual environment: -``` -python3.9 -m venv bt_venv +9. Activate and switch to the newly created Python virtual environment: +```shell source bt_venv/bin/activate ``` -Install dependencies: -``` +10. Install the Python dependencies: +```shell pip install -r requirements.txt ``` -Add the current directory to PYTHONPATH to resolve module issues: -``` -export PYTHONPATH="/home/{your-user}/synth-subnet:$PYTHONPATH" -``` - ---- - -### Step 8: Run a Miner -Configuration files (`*.config.js`) in the repository define how to start and manage applications with PM2. They allow you to specify application details, environment variables, and runtime configurations. - -#### IMPORTANT: Activate the virtual environment before running: -``` -source bt_venv/bin/activate -``` - -Start a miner using the example configuration file: -``` -pm2 start miner.config.js -``` - -Alternatively, use the dummy miner for testing: -``` -pm2 start miner-dummy.local.config.js -``` +[Back to top ^][table-of-contents] -You can create and run your custom configuration file similarly. +### 2.3. Miners -#### PM2 Commands: +Please refer to this [guide](./docs/miner_guide.md) for detailed instructions on getting a miner up and running. -To return a list of currently running applications: -``` -pm2 list -``` +[Back to top ^][table-of-contents] -Stop a specific application: -``` -pm2 stop miner -``` +### 2.4. Validators -Restart an application after code changes: -``` -pm2 start miner -``` +Please refer to this [guide](./docs/validator_guide.md) for detailed instructions on getting a validator up and running. -View logs: -``` -pm2 logs miner -``` +[Back to top ^][table-of-contents] -View detailed logs: -``` -pm2 logs miner --lines 100 -``` +## 📄 3. License ---- +Please refer to the [LICENSE](./LICENSE) file. -### Step 9: Run a Validator -Similar to miners, validators also use `*.config.js` files for configuration. +[Back to top ^][table-of-contents] -#### IMPORTANT: Activate the virtual environment before running: -``` -source bt_venv/bin/activate -``` - -Start a validator using the example configuration file: -``` -pm2 start validator.config.js -``` + +[table-of-contents]: #table-of-contents diff --git a/docs/images/mode_icon@128x128.png b/docs/images/mode_icon@128x128.png new file mode 100644 index 00000000..b645b26d Binary files /dev/null and b/docs/images/mode_icon@128x128.png differ diff --git a/docs/miner_guide.md b/docs/miner_guide.md new file mode 100644 index 00000000..d4df4753 --- /dev/null +++ b/docs/miner_guide.md @@ -0,0 +1,112 @@ +# Miner Guide + +### Table Of Contents + +* [1. Create a Wallet](#1-create-a-wallet) +* [2. Run the Miner](#2-run-the-miner) +* [3. Appendix](#3-appendix) + - [3.1. Useful Commands](#31-useful-commands) + - [3.2. Troubleshooting](#32-troubleshooting) + +## 1. Create a Wallet + +> 💡 **TIP:** For a more extensive list of the Bittensor CLI commands see [here](https://docs.bittensor.com/btcli). + +1. (Optional) If you haven't already, ensure you are running from the Python virtual environment: +```shell +source bt_venv/bin/activate +``` + +2. Create the cold/hot wallets: +```shell +btcli wallet create \ + --wallet.name miner \ + --wallet.hotkey default +``` + +> 🚨 **WARNING:** You must ensure your wallets have enough TAO (0.1 should be sufficient) to be start mining. For testnet, you can use the [`btcli wallet faucet`](https://docs.bittensor.com/btcli#btcli-wallet-faucet). + +3. Acquire a slot on the Bittensor subnet by registering the wallet: +```shell +btcli subnet register \ + --wallet.name miner \ + --subtensor.network test \ + --netuid 247 +``` + +4. (Optional) Check the wallet has been registered: +```shell +btcli wallet overview \ + --wallet.name miner \ + --wallet.hotkey default \ + --subtensor.network test +``` + +5. (Optional) Check the network metagraph: +```shell +btcli subnet metagraph \ + --subtensor.network test \ + --netuid 247 +``` + +[Back to top ^][table-of-contents] + +## 2. Run the Miner + +1. (Optional) If you haven't already, ensure you are running from the Python virtual environment: +```shell +source bt_venv/bin/activate +``` + +2. Start PM2 with the miner config: +```shell +pm2 start miner.config.js +``` + +3. (Optional) Check the miner is running: +```shell +pm2 list +``` + +[Back to top ^][table-of-contents] + +## 3. Appendix + +### 3.1. Useful Commands + +| Command | Description | +|------------------------------|-----------------------------| +| `pm2 stop miner` | Stops the miner. | +| `pm2 logs miner --lines 100` | View the logs of the miner. | + +[Back to top ^][table-of-contents] + +### 3.2. Troubleshooting + +#### `ModuleNotFoundError: No module named 'simulation'` + +This error is due to Python unable to locate the local Python modules. To avoid this error, ensure you have created and activate the Python virtual environment from the project root and ensure the `PYTHONPATH` is present in the `.config.js` file and is pointing to the project root. + +An example of a config file should be: +```js +// miner.config.js +module.exports = { + apps: [ + { + name: 'miner', + script: 'python3', + args: './neurons/miner.py --netuid 247 --logging.debug --logging.trace --subtensor.network test --wallet.name miner --wallet.hotkey default --axon.port 8091', + env: { + PYTHONPATH: '.' + }, + }, + ], +}; +``` + +As you can see, we are setting the `PYTHONPATH` environment variable that will be injected when we run `pm2 start miner`. + +[Back to top ^][table-of-contents] + + +[table-of-contents]: #table-of-contents diff --git a/docs/validator_guide.md b/docs/validator_guide.md new file mode 100644 index 00000000..3b1dc0c3 --- /dev/null +++ b/docs/validator_guide.md @@ -0,0 +1,84 @@ +# Validator + +### Table Of Contents + +* [1. Create a Wallet](#1-create-a-wallet) +* [2. Run the Validator](#2-run-the-validator) +* [3. Appendix](#3-appendix) + - [3.1. Useful Commands](#31-useful-commands) + +## 1. Create a Wallet + +> 💡 **TIP:** For a more extensive list of the Bittensor CLI commands see [here](https://docs.bittensor.com/btcli). + +1. (Optional) If you haven't already, ensure you are running from the Python virtual environment: +```shell +source bt_venv/bin/activate +``` + +2. Create the cold/hot wallets: +```shell +btcli wallet create \ + --wallet.name validator \ + --wallet.hotkey default +``` + +> 🚨 **WARNING:** You must ensure your wallets have enough TAO (0.1 should be sufficient) to be start mining. For testnet, you can use the [`btcli wallet faucet`](https://docs.bittensor.com/btcli#btcli-wallet-faucet). + +3. Acquire a slot on the Bittensor subnet by registering the wallet: +```shell +btcli subnet register \ + --wallet.name validator \ + --wallet.hotkey default \ + --subtensor.network test \ + --netuid 247 +``` + +4. (Optional) Check the wallet has been registered: +```shell +btcli wallet overview \ + --wallet.name validator \ + --subtensor.network test +``` + +5. (Optional) Check the network metagraph: +```shell +btcli subnet metagraph \ + --subtensor.network test \ + --netuid 247 +``` + +[Back to top ^][table-of-contents] + +## 2. Run the Validator + +1. (Optional) If you haven't already, ensure you are running from the Python virtual environment: +```shell +source bt_venv/bin/activate +``` + +2. Start PM2 with the validator config: +```shell +pm2 start validator.config.js +``` + +3. (Optional) Check the validator is running: +```shell +pm2 list +``` + +[Back to top ^][table-of-contents] + +## 3. Appendix + +### 3.1. Useful Commands + +| Command | Description | +|----------------------------------|---------------------------------| +| `pm2 stop validator` | Stops the validator. | +| `pm2 logs validator --lines 100` | View the logs of the validator. | + +[Back to top ^][table-of-contents] + + +[table-of-contents]: #table-of-contents diff --git a/miner-dummy.local.config.js b/miner.dev.config.js similarity index 61% rename from miner-dummy.local.config.js rename to miner.dev.config.js index 106c99a1..ce0a7194 100644 --- a/miner-dummy.local.config.js +++ b/miner.dev.config.js @@ -1,9 +1,12 @@ module.exports = { apps: [ { - name: 'miner-dummy-local', + name: 'miner-dev', script: 'python3', - args: './neurons/miner.py --netuid 1 --logging.debug --logging.trace --subtensor.chain_endpoint ws://127.0.0.1:9946 --wallet.name miner-dummy --wallet.hotkey default --axon.port 10000 --miner_type dummy' + args: './neurons/miner.py --netuid 1 --logging.debug --logging.trace --subtensor.chain_endpoint ws://127.0.0.1:9946 --wallet.name miner --wallet.hotkey default --axon.port 8091', + env: { + PYTHONPATH: '.' + }, }, ], }; diff --git a/miner.local.config.js b/miner.local.config.js deleted file mode 100644 index 3b7af3c4..00000000 --- a/miner.local.config.js +++ /dev/null @@ -1,9 +0,0 @@ -module.exports = { - apps: [ - { - name: 'miner-local', - script: 'python3', - args: './neurons/miner.py --netuid 1 --logging.debug --logging.trace --subtensor.chain_endpoint ws://127.0.0.1:9946 --wallet.name miner --wallet.hotkey default --axon.port 8091' - }, - ], -}; diff --git a/validator.config.js b/validator.config.js index d32a8302..e54e68f0 100644 --- a/validator.config.js +++ b/validator.config.js @@ -3,7 +3,10 @@ module.exports = { { name: 'validator', script: 'python3', - args: './neurons/validator.py --netuid 247 --logging.debug --logging.trace --subtensor.network test --wallet.name validator --wallet.hotkey default --neuron.axon_off true' + args: './neurons/validator.py --netuid 247 --logging.debug --logging.trace --subtensor.network test --wallet.name validator --wallet.hotkey default --neuron.axon_off true', + env: { + PYTHONPATH: '.', + }, }, ], }; diff --git a/validator.local.config.js b/validator.dev.config.js similarity index 76% rename from validator.local.config.js rename to validator.dev.config.js index 6577fcc5..dfa11f92 100644 --- a/validator.local.config.js +++ b/validator.dev.config.js @@ -1,9 +1,12 @@ module.exports = { apps: [ { - name: 'validator-local', + name: 'validator-dev', script: 'python3', - args: './neurons/validator.py --netuid 1 --logging.debug --logging.trace --subtensor.chain_endpoint ws://127.0.0.1:9946 --wallet.name validator --wallet.hotkey default --neuron.axon_off true' + args: './neurons/validator.py --netuid 1 --logging.debug --logging.trace --subtensor.chain_endpoint ws://127.0.0.1:9946 --wallet.name validator --wallet.hotkey default --neuron.axon_off true', + env: { + PYTHONPATH: '.', + }, }, ], };