Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -517,21 +517,21 @@ You can also run the generator standalone:

Using the above docker tool the following files are generated (unless already generated or forced via `--forceKeyGen` flag):

**Generated files (dual-key manifest — two roles per validator index):**
**Generated files (dual-key manifest — two roles per validator index, SSZ only):**
```
local-devnet/genesis/hash-sig-keys/
├── validator-keys-manifest.yaml # Metadata (attester + proposer pubkeys per index)
├── validator_0_proposer_key_pk.json # Proposer public key (validator 0)
├── validator_0_proposer_key_sk.json # Proposer secret key
├── validator_0_attester_key_pk.json # Attester (attestation) public key
├── validator_0_attester_key_sk.json # Attester secret key
├── validator_1_proposer_key_pk.json
├── validator_1_proposer_key_sk.json
├── validator_1_attester_key_pk.json
├── validator_1_attester_key_sk.json
├── validator_0_proposer_key_pk.ssz # Proposer public key (validator 0)
├── validator_0_proposer_key_sk.ssz # Proposer secret key
├── validator_0_attester_key_pk.ssz # Attester (attestation) public key
├── validator_0_attester_key_sk.ssz # Attester secret key
├── validator_1_proposer_key_pk.ssz
├── validator_1_proposer_key_sk.ssz
├── validator_1_attester_key_pk.ssz
├── validator_1_attester_key_sk.ssz
└── ... # Same pattern for additional validators
```
Older **single-key** layouts (`validator_N_pk.json` / `validator_N_sk.json` only) are still recognized when regenerating from an existing manifest.
Older **single-key** layouts (`validator_N_pk.ssz` / `validator_N_sk.ssz` only) are still recognized when regenerating from an existing manifest.

**Signature Scheme:**
The system uses the **SIGTopLevelTargetSumLifetime32Dim64Base8** hash-based signature scheme, which provides:
Expand Down Expand Up @@ -614,7 +614,7 @@ zeam_0:
Post genesis generation, the quickstarts loads and calls the appropriate node's client cmd from `client-cmds` folder where either `docker` or `binary` cmd is picked as per the `node_setup` mode. (Generally `binary` mode is handy for local interop debugging for a client).

**Client Integration:**
Your client implementation should read these environment variables and use the hash-sig keys for validator operations. After `parse-vc.sh` runs, **`$HASH_SIG_PK_PATH` / `$HASH_SIG_SK_PATH`** point at the **proposer** JSON keys when using dual-key manifest files; **`$HASH_SIG_ATTESTER_PK_PATH`** / **`$HASH_SIG_ATTESTER_SK_PATH`** (and proposer-specific `HASH_SIG_PROPOSER_*`) are set when those files exist.
Your client implementation should read these environment variables and use the hash-sig keys for validator operations. After `parse-vc.sh` runs, **`$HASH_SIG_PK_PATH` / `$HASH_SIG_SK_PATH`** point at the **proposer** SSZ keys when using dual-key manifest files; **`$HASH_SIG_ATTESTER_PK_PATH`** / **`$HASH_SIG_ATTESTER_SK_PATH`** (and proposer-specific `HASH_SIG_PROPOSER_*`) are set when those files exist.

- `$item` - the node name for which this cmd is being executed, index into `validator-config.yaml` for its configuration
- `$configDir` - the abs folder housing `genesis` configuration (same as `NETWORK_DIR` env variable provided while executing shell command), already mapped to `/config` in the docker mode
Expand Down Expand Up @@ -692,7 +692,7 @@ NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis --forceKeyG
### Key Security

**Secret keys are highly sensitive:**
- ⚠️ **Never commit** `validator_*_*_sk.json` or `validator_*_sk.json` secret key files to version control
- ⚠️ **Never commit** `validator_*_*_sk.ssz` or `validator_*_sk.ssz` secret key files to version control
- ⚠️ **Never share** secret keys
- ✅ **Backup** secret keys in secure, encrypted storage
- ✅ **Restrict permissions** on key files (e.g., `chmod 600`)
Expand Down Expand Up @@ -738,9 +738,9 @@ validators:

**Problem**: Hash-sig keys not loading during node startup
```
Warning: Hash-sig public key not found at genesis/hash-sig-keys/validator_0_proposer_key_pk.json
Warning: Hash-sig public key not found at genesis/hash-sig-keys/validator_0_proposer_key_pk.ssz
```
(or `validator_0_pk.json` when using a legacy single-key tree)
(or `validator_0_pk.ssz` when using a legacy single-key tree)

**Solution**: Run the genesis generator to create keys:
```sh
Expand All @@ -755,9 +755,9 @@ NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis

**Problem**: Hash-sig key file not found
```
Warning: Hash-sig secret key not found at genesis/hash-sig-keys/validator_5_proposer_key_sk.json
Warning: Hash-sig secret key not found at genesis/hash-sig-keys/validator_5_proposer_key_sk.ssz
```
(or `validator_5_sk.json` in legacy layouts)
(or `validator_5_sk.ssz` in legacy layouts)

**Solution**: This usually means you have more validators configured than hash-sig keys generated. Regenerate genesis files:
```sh
Expand Down
20 changes: 20 additions & 0 deletions ansible-deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,11 @@ Examples:

Environment Variables:
NETWORK_DIR Network directory (overrides --network-dir)
ANSIBLE_FORKS Passed as ansible-playbook -f (disables IP-based default)
LEAN_VALIDATOR_CONFIG_PATH Validator YAML used only to compute -f in ansible-deploy
(default: \$NETWORK_DIR/genesis/validator-config.yaml)
LEAN_ANSIBLE_FORKS_MIN Floor for computed forks (default: 5)
LEAN_ANSIBLE_FORKS_MAX Ceiling for computed forks (default: 128)

EOF
}
Expand Down Expand Up @@ -173,6 +178,21 @@ ANSIBLE_CMD="$ANSIBLE_CMD -i $ANSIBLE_DIR/inventory/hosts.yml"
ANSIBLE_CMD="$ANSIBLE_CMD $ANSIBLE_DIR/playbooks/$PLAYBOOK"
ANSIBLE_CMD="$ANSIBLE_CMD -e \"$EXTRA_VARS\""

# Forks: ANSIBLE_FORKS wins; else count unique enrFields.ip in genesis validator-config.yaml
_vc_for_forks="${LEAN_VALIDATOR_CONFIG_PATH:-$NETWORK_DIR_ABS/genesis/validator-config.yaml}"
if [[ "${_vc_for_forks}" != /* ]]; then
_vc_for_forks="$SCRIPT_DIR/${_vc_for_forks}"
fi
if [[ -n "${ANSIBLE_FORKS:-}" ]]; then
ANSIBLE_CMD="$ANSIBLE_CMD -f ${ANSIBLE_FORKS}"
elif [[ -f "${_vc_for_forks}" ]] && command -v yq &>/dev/null; then
_ansible_forks="$("$ANSIBLE_DIR/compute-forks-from-validator-config.sh" "${_vc_for_forks}")"
ANSIBLE_CMD="$ANSIBLE_CMD -f ${_ansible_forks}"
echo -e "${GREEN}Ansible forks:${NC} ${_ansible_forks} (unique IPs in ${_vc_for_forks}; set ANSIBLE_FORKS or LEAN_VALIDATOR_CONFIG_PATH to override)"
elif [[ -f "${_vc_for_forks}" ]]; then
echo -e "${YELLOW}Warning:${NC} yq not found; omitting -f (using ansible.cfg forks default)"
fi

if [ -n "$TAGS" ]; then
ANSIBLE_CMD="$ANSIBLE_CMD --tags $TAGS"
fi
Expand Down
20 changes: 18 additions & 2 deletions ansible/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,20 @@ docker ps | grep zeam_0
- `roles/` - Reusable role modules (zeam, ream, qlean, lantern, lighthouse, grandine, ethlambda, genesis, common)
- `requirements.yml` - Ansible Galaxy dependencies

## Parallelism (large devnets)

`ansible.cfg` sets **`strategy = free`** and a **`forks`** fallback for bare `ansible-playbook` invocations.

**`run-ansible.sh`** and **`ansible-deploy.sh`** pass **`-f N`** where `N` comes from counting **unique `enrFields.ip`** values in the active `validator-config.yaml` (via `ansible/compute-forks-from-validator-config.sh`), clamped between **`LEAN_ANSIBLE_FORKS_MIN`** (default `5`) and **`LEAN_ANSIBLE_FORKS_MAX`** (default `128`). The minimum avoids fully serializing deploys when every validator row shares one IP (many nodes, one machine) while still reflecting larger devnets by unique address.

- Override forks: `ANSIBLE_FORKS=100` or `ansible-playbook -f 100 ...`
- Override validator config path for fork counting only (ansible-deploy): `LEAN_VALIDATOR_CONFIG_PATH=/path/to/validator-config.yaml`
- Tune clamp: `LEAN_ANSIBLE_FORKS_MIN=1 LEAN_ANSIBLE_FORKS_MAX=256`

If `yq` is missing, the wrappers omit `-f` and Ansible uses `ansible.cfg` **`forks`**.

Very high forks can stress SSH, the control node, or image registries; back off if you see timeouts or rate limits.

## Configuration Source

Ansible roles automatically extract Docker images and deployment modes from `client-cmds/*-cmd.sh` files:
Expand Down Expand Up @@ -205,8 +219,10 @@ Test copying genesis files to remote hosts (genesis files must be generated loca

**Verify copied files on remote host:**
```sh
ls -la local-devnet/genesis/
# Should see: config.yaml, validators.yaml, nodes.yaml, genesis.json, genesis.ssz, *.key files
# Local genesis dir still holds every *.key after generate-genesis.
# On each remote host you get shared yamls/ssz/json plus *.key only for validators
# whose enrFields.ip matches that host (collocated nodes on one machine share the same set).
ls -la /opt/lean-quickstart/genesis/
```

### Phase 4: Test Docker Image Extraction (Latest Changes)
Expand Down
9 changes: 9 additions & 0 deletions ansible/ansible.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,15 @@ result_format = yaml
display_skipped_hosts = False
display_ok_hosts = True
deprecation_warnings = False

# Large devnets: use strategy=free so hosts do not wait on each other at every task.
# Concurrent sessions: run-ansible.sh / ansible-deploy.sh pass -f from unique
# enrFields.ip in validator-config (see compute-forks-from-validator-config.sh).
# This value applies when ansible-playbook is run without -f and ANSIBLE_FORKS is unset.
# Override: ANSIBLE_FORKS=50 ansible-playbook ... or ansible-playbook -f 50 ...
# For strict lockstep: ANSIBLE_STRATEGY=linear
forks = 25
strategy = free
# SSH private key file (can be overridden per-host in inventory or via environment variable)
# Defaults to ~/.ssh/id_rsa if not specified
# To use a different key, either:
Expand Down
60 changes: 60 additions & 0 deletions ansible/compute-forks-from-validator-config.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/usr/bin/env bash
# Emit an integer for ansible-playbook --forks based on unique enrFields.ip
# values in validator-config.yaml (one row per logical machine in typical layouts).
#
# Usage: compute-forks-from-validator-config.sh /abs/or/rel/path/to/validator-config.yaml
#
# Environment (optional):
# LEAN_ANSIBLE_FORKS_MIN floor after counting unique IPs (default: 5). Helps when
# many inventory hosts share one IP so work is not fully serialized.
# LEAN_ANSIBLE_FORKS_MAX ceiling (default: 128)
# LEAN_ANSIBLE_FORKS_FALLBACK if the file is missing, yq fails, or count is 0 (default: 25)

set -euo pipefail

vc_path="${1:-}"
_min="${LEAN_ANSIBLE_FORKS_MIN:-5}"
_max="${LEAN_ANSIBLE_FORKS_MAX:-128}"
_fallback="${LEAN_ANSIBLE_FORKS_FALLBACK:-25}"

if [[ -z "$vc_path" ]]; then
echo "$_fallback"
exit 0
fi

if [[ ! -f "$vc_path" ]]; then
echo "$_fallback"
exit 0
fi

if ! command -v yq &>/dev/null; then
echo "$_fallback"
exit 0
fi

unique_ips=0
if ! unique_ips=$(yq eval '[.validators[]? | .enrFields.ip // "" | select(. != "")] | unique | length' "$vc_path" 2>/dev/null); then
echo "$_fallback"
exit 0
fi

# yq may print nothing on some failures
if [[ -z "${unique_ips// /}" ]] || ! [[ "$unique_ips" =~ ^[0-9]+$ ]]; then
echo "$_fallback"
exit 0
fi

if (( unique_ips < 1 )); then
echo "$_fallback"
exit 0
fi

forks=$unique_ips
if (( forks < _min )); then
forks=$_min
fi
if (( forks > _max )); then
forks=$_max
fi

echo "$forks"
5 changes: 5 additions & 0 deletions ansible/inventory/group_vars/all.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
---
# Global variables for all nodes
#
# Parallel runs: ansible/ansible.cfg sets strategy=free and forks=25 as fallback when
# no -f is passed. run-ansible.sh and ansible-deploy.sh set -f from unique enrFields.ip
# in validator-config (clamped by LEAN_ANSIBLE_FORKS_MIN / LEAN_ANSIBLE_FORKS_MAX).
# Override forks: ANSIBLE_FORKS=100 or ansible-playbook -f 100 ...

# Paths (relative to playbook execution directory or absolute)
network_dir: "{{ playbook_dir }}/../local-devnet"
Expand Down
22 changes: 17 additions & 5 deletions ansible/playbooks/copy-genesis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@
# - ansible-playbook playbooks/site.yml (runs generate-genesis automatically)
#
# This playbook validates all required files exist locally before copying.
#
# Shared genesis artifacts (ssz, json, yamls) go to every host. Node *.key files
# are private libp2p identities — only validators collocated on this machine
# (same enrFields.ip as ansible_host) are copied here; peers' *public* keys
# come from validators.yaml / genesis, not from other nodes' .key files.

- name: Copy Genesis Files to Remote Hosts
hosts: all:!local
Expand Down Expand Up @@ -115,9 +120,9 @@
delegate_to: localhost
run_once: true

- name: Confirm all key files are present
- name: Confirm all key files are present locally
debug:
msg: "✅ All {{ all_node_names | length }} key file(s) are present and ready to copy: {{ all_node_names | join(', ') }}"
msg: "✅ All {{ all_node_names | length }} libp2p .key file(s) exist locally (each host receives only keys for its IP)"
delegate_to: localhost
run_once: true

Expand All @@ -143,13 +148,20 @@
- genesis.ssz
- validator-config.yaml

- name: Copy node private key files to remote host
- name: Resolve collocated validator names (same IP as this inventory host)
vars:
_vc: "{{ lookup('file', validator_config_file) | from_yaml }}"
_by_ip: "{{ _vc.validators | default([]) | selectattr('enrFields.ip', 'defined') | selectattr('enrFields.ip', 'equalto', ansible_host | string) | map(attribute='name') | list }}"
set_fact:
node_libp2p_keys_to_copy: "{{ _by_ip if (_by_ip | length) > 0 else [inventory_hostname] }}"

- name: Copy libp2p private key files for collocated validators only
copy:
src: "{{ genesis_dir }}/{{ item }}.key"
dest: "{{ actual_remote_genesis_dir }}/{{ item }}.key"
mode: '0600'
force: yes
loop: "{{ all_node_names }}"
loop: "{{ node_libp2p_keys_to_copy }}"
loop_control:
label: "{{ item }}.key"

Expand All @@ -159,7 +171,7 @@
_assignments: "{{ _av[inventory_hostname] | default([]) }}"
_base: "{{ _assignments | map(attribute='privkey_file') | map('regex_replace', '_sk\\.ssz$', '') | list }}"
set_fact:
node_hash_sig_files: "{{ (_base | product(['_sk.ssz', '_pk.ssz', '_sk.json', '_pk.json']) | map('join') | list) + ['validator-keys-manifest.yaml'] }}"
node_hash_sig_files: "{{ (_base | product(['_sk.ssz', '_pk.ssz']) | map('join') | list) + ['validator-keys-manifest.yaml'] }}"
when: hash_sig_keys_stat.stat.exists

- name: Create hash-sig-keys directory on remote
Expand Down
2 changes: 1 addition & 1 deletion ansible/playbooks/deploy-nodes.yml
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@
_assignments: "{{ _av[node_name] | default([]) }}"
_base: "{{ _assignments | map(attribute='privkey_file') | map('regex_replace', '_sk\\.ssz$', '') | list }}"
set_fact:
node_hash_sig_files: "{{ (_base | product(['_sk.ssz', '_pk.ssz', '_sk.json', '_pk.json']) | map('join') | list) + ['validator-keys-manifest.yaml'] }}"
node_hash_sig_files: "{{ (_base | product(['_sk.ssz', '_pk.ssz']) | map('join') | list) + ['validator-keys-manifest.yaml'] }}"
when: hash_sig_keys_local.stat.exists
tags:
- deploy
Expand Down
3 changes: 3 additions & 0 deletions ansible/playbooks/site.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@
# clean_data - Set to true to clean data directories before deployment (default: false)
# skip_genesis - Set to true to skip genesis generation (default: false)
# genesis_offset - Seconds to add to current time for genesis (default: 360 for ansible mode)
#
# Parallelism: ansible.cfg uses strategy=free; run-ansible / ansible-deploy pass -f from
# unique enrFields.ip in validator-config unless ANSIBLE_FORKS is set.

- name: Clean Data Directories
import_playbook: clean-node-data.yml
Expand Down
28 changes: 14 additions & 14 deletions generate-genesis.sh
Original file line number Diff line number Diff line change
Expand Up @@ -212,15 +212,15 @@ if [ ! -f "$MANIFEST_FILE" ]; then
else
for ((i=0; i<VALIDATOR_COUNT; i++)); do
# devnet4+: proposer + attester key files per index
if [ -f "$HASH_SIG_KEYS_DIR/validator_${i}_proposer_key_pk.json" ] && \
[ -f "$HASH_SIG_KEYS_DIR/validator_${i}_attester_key_pk.json" ]; then
if [ ! -f "$HASH_SIG_KEYS_DIR/validator_${i}_proposer_key_sk.json" ] || \
[ ! -f "$HASH_SIG_KEYS_DIR/validator_${i}_attester_key_sk.json" ]; then
if [ -f "$HASH_SIG_KEYS_DIR/validator_${i}_proposer_key_pk.ssz" ] && \
[ -f "$HASH_SIG_KEYS_DIR/validator_${i}_attester_key_pk.ssz" ]; then
if [ ! -f "$HASH_SIG_KEYS_DIR/validator_${i}_proposer_key_sk.ssz" ] || \
[ ! -f "$HASH_SIG_KEYS_DIR/validator_${i}_attester_key_sk.ssz" ]; then
KEYS_EXIST=false
break
fi
elif [ -f "$HASH_SIG_KEYS_DIR/validator_${i}_pk.json" ] && \
[ -f "$HASH_SIG_KEYS_DIR/validator_${i}_sk.json" ]; then
elif [ -f "$HASH_SIG_KEYS_DIR/validator_${i}_pk.ssz" ] && \
[ -f "$HASH_SIG_KEYS_DIR/validator_${i}_sk.ssz" ]; then
:
else
KEYS_EXIST=false
Expand Down Expand Up @@ -284,17 +284,17 @@ else
--num-validators "$VALIDATOR_COUNT" \
--log-num-active-epochs "$ACTIVE_EPOCH" \
--output-dir "/genesis/hash-sig-keys" \
--export-format both
--export-format ssz

if [ $? -ne 0 ]; then
echo " ❌ Failed to generate hash-sig keys"
exit 1
fi

echo " ✅ Generated keys for $VALIDATOR_COUNT validators"
echo " ✅ Files created (per validator: proposer + attester pk/sk):"
echo " ✅ Files created (per validator: proposer + attester pk/sk SSZ):"
for i in $(seq 0 $((VALIDATOR_COUNT - 1))); do
echo " - validator_${i}_proposer_key_{pk,sk}.json validator_${i}_attester_key_{pk,sk}.json"
echo " - validator_${i}_proposer_key_{pk,sk}.ssz validator_${i}_attester_key_{pk,sk}.ssz"
done

echo ""
Expand Down Expand Up @@ -778,12 +778,12 @@ done
echo ""
echo "🔐 Hash-Sig Validator Keys:"
for i in $(seq 0 $((VALIDATOR_COUNT - 1))); do
if [ -f "$GENESIS_DIR/hash-sig-keys/validator_${i}_proposer_key_pk.json" ]; then
echo " $GENESIS_DIR/hash-sig-keys/validator_${i}_proposer_key_{pk,sk}.json"
echo " $GENESIS_DIR/hash-sig-keys/validator_${i}_attester_key_{pk,sk}.json"
if [ -f "$GENESIS_DIR/hash-sig-keys/validator_${i}_proposer_key_pk.ssz" ]; then
echo " $GENESIS_DIR/hash-sig-keys/validator_${i}_proposer_key_{pk,sk}.ssz"
echo " $GENESIS_DIR/hash-sig-keys/validator_${i}_attester_key_{pk,sk}.ssz"
else
echo " $GENESIS_DIR/hash-sig-keys/validator_${i}_pk.json"
echo " $GENESIS_DIR/hash-sig-keys/validator_${i}_sk.json"
echo " $GENESIS_DIR/hash-sig-keys/validator_${i}_pk.ssz"
echo " $GENESIS_DIR/hash-sig-keys/validator_${i}_sk.ssz"
fi
done
echo ""
Expand Down
14 changes: 7 additions & 7 deletions parse-vc.sh
Original file line number Diff line number Diff line change
Expand Up @@ -110,13 +110,13 @@ hashSigKeyIndex=$(yq eval ".validators | to_entries | .[] | select(.value.name =

# Load hash-sig keys if configured
if [ "$keyType" == "hash-sig" ] && [ "$hashSigKeyIndex" != "null" ] && [ -n "$hashSigKeyIndex" ]; then
# devnet4+: separate proposer + attester keys (hash-sig-cli); legacy: single pk/sk per index
_proposer_pk="$configDir/hash-sig-keys/validator_${hashSigKeyIndex}_proposer_key_pk.json"
_proposer_sk="$configDir/hash-sig-keys/validator_${hashSigKeyIndex}_proposer_key_sk.json"
_attester_pk="$configDir/hash-sig-keys/validator_${hashSigKeyIndex}_attester_key_pk.json"
_attester_sk="$configDir/hash-sig-keys/validator_${hashSigKeyIndex}_attester_key_sk.json"
_legacy_pk="$configDir/hash-sig-keys/validator_${hashSigKeyIndex}_pk.json"
_legacy_sk="$configDir/hash-sig-keys/validator_${hashSigKeyIndex}_sk.json"
# devnet4+: separate proposer + attester keys (hash-sig-cli); legacy: single pk/sk per index (SSZ only)
_proposer_pk="$configDir/hash-sig-keys/validator_${hashSigKeyIndex}_proposer_key_pk.ssz"
_proposer_sk="$configDir/hash-sig-keys/validator_${hashSigKeyIndex}_proposer_key_sk.ssz"
_attester_pk="$configDir/hash-sig-keys/validator_${hashSigKeyIndex}_attester_key_pk.ssz"
_attester_sk="$configDir/hash-sig-keys/validator_${hashSigKeyIndex}_attester_key_sk.ssz"
_legacy_pk="$configDir/hash-sig-keys/validator_${hashSigKeyIndex}_pk.ssz"
_legacy_sk="$configDir/hash-sig-keys/validator_${hashSigKeyIndex}_sk.ssz"

if [ -f "$_proposer_pk" ] && [ -f "$_attester_pk" ]; then
hashSigPkPath="$_proposer_pk"
Expand Down
Loading
Loading