This repository is under active development and is not expected to work in a general setting. It should be considered view only.
This respository optimizes the filecoin sealing process for throughput per dollar. It is a companion to https://github.com/supranational/filecoin_pc1 and will read sealed PC1 column data from NVMe devices under SPDK to generate tree-r and tree-c files that are compatible with the existing sealing software. The current software is configured to seal 64 sectors in parallel.
Like filecoin_pc1 this is an initial prototype release.
What's coming:
- Integrate filecoin_pc1, filecoin_pc2, c1 into a single repository
- Support for pipelined and simultaneous jobs
- More seamless support for sealing a variety of parallel sector counts
- Simplified configuration
Using the reference platform outlined below this software generates PC2 tree-r and tree-c files for 64 sectors in approximately 70 minutes.
We will specify a reference configuration with the final release of the software. Our current testing configuration consists of:
- Threadripper PRO 5975WX
- ASUS WRX80E SAGE Motherboard
- 256GB Memory
- 15 Samsung 7.68GB U.2 Drives
- 4 Supermicro AOC-SLG4-4E4T NVMe HBA
- 2 NVIDIA GeForce RTX 3080
Though huge pages aren't strictly necessary for PC2, the software is configured to use them since it is expected to run alongside PC1.
sudo vi /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=128"
GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=128"
sudo update-grub
sudo reboot
You can confirm huge pages are enabled with:
grep Huge /proc/meminfo
# Look for:
HugePages_Total: 128
HugePages_Free: 128
Additionally you may need to enable huge pages after boot using:
sudo sysctl -w vm.nr_hugepages=128
Build SPDK in the Parent Directory:
git clone --branch v22.09 https://github.com/spdk/spdk --recursive spdk-v22.09
cd spdk-v22.09/
sudo scripts/pkgdep.sh
./configure --with-virtio --with-vhost
make
sudo env NRHUGE=128 ./scripts/setup.sh
SPDK is used for directly controlling the NVMe drives.
Update NVMe Controller Addresses in src/column_reader.cpp:
SPDK can be used to identify attached NVMe devices and their addresses with the following command:
sudo ./scripts/setup.sh status
For more extensive information about attached devices:
sudo ./build/examples/identify
This will show the NVMe disks (controllers) along with their addresses, which will resemble 0000:2c:00.0. The address list in src/column_reader.cpp must be updated to reflect the addresses that will be used.
In addition, if you have a different number of drives than the reference configuration, then you must update NUM_CONTROLLERS in /filecoin_pc1/src/sealing/constants.hpp
The drive configuration for PC2 must match PC1 for correct functionality.
./build.sh
PC1 should be run prior to running PC2 to populate the DRG data on the NVMe drives. See https://github.com/supranational/filecoin_pc1 for related instructions.
Once this is completed, run:
sudo rm results/* # Delete any existing output files
sudo ./pc2
This will create a "results" directly that stores the PC2 output files.
