feat: integrate spo indexer #526

caparrosm · 2025-11-18T19:28:57Z

No description provided.

gilescope · 2025-11-19T11:42:53Z

Cargo.toml

 midnight-transient-crypto_v6 = { git = "https://github.com/midnightntwrk/midnight-ledger", package = "midnight-transient-crypto", tag = "ledger-6.1.0-alpha.5" }
 midnight-zswap_v6            = { git = "https://github.com/midnightntwrk/midnight-ledger", package = "midnight-zswap", tag = "ledger-6.1.0-alpha.5" }
 nix                          = { version = "0.30" }
+once_cell                    = { version = "1.19" }


Is once_cell really required? I thought most of the functionality was merged into std now.
(E.g. rust-lang/rust#105587 )

gilescope · 2025-11-19T11:50:12Z

MIGRATION.md

+{"timestamp":"2025-11-17T23:05:51.417863+00:00","level":"ERROR","target":"spo_indexer","file":"spo-indexer/src/main.rs","line":31,"message":"process exited with ERROR","kvs":{"backtrace":"disabled backtrace","error":"cannot make rpc call: sidechain_getEpochCommittee"}}
+```
+
+**Root Cause**: The `sidechain_getEpochCommittee` RPC method does not exist in the Midnight Preview network API (likely removed or renamed between alpha.2 and alpha.5).


sidechain_getEpochCommittee does still exist. https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frpc.preview.midnight.network#/rpc rpc / methods() call returns it in the list.

It's possibly not shown on the polkadot.js UI due to a bug in that but it's there if you call it.

hseeberger · 2025-11-19T16:55:55Z

indexer-common/migrations/postgres/002_spo_initial.sql

@@ -0,0 +1,84 @@
+CREATE TABLE epochs (


Please use a single sql file called "002_spo.sql"; no further history yet.

cosmir17

Hi @caparrosm, thanks for the good work on the SPO integration! The full integration approach with path dependencies is excellent, and I can see you've followed several midnight-indexer patterns correctly.

I can't approve this PR as I defer to Giles and Heiko for decisions, but I wanted to provide these review comments to help get this ready for them. The core integration work seems solid.

cosmir17 · 2025-11-21T00:20:17Z

spo-indexer/config.yaml

+    genesis_protocol_version: 16000
+    reconnect_max_delay: "10s" # 10ms, 100ms, 1s, 10s
+    reconnect_max_attempts: 30 # Roughly 5m
+    blockfrost_id: "previewukkFxumNW31cXmsBtKI1JTnbxvcVCbCj"


is this sensitive information?
if so, please let us know 🙏

This is sensitive information as anybody could reuse and spend your blockfrost tokens on your behalf. If you are using a free plan it would prevent you from executing more calls if a malicious party reuses this and reaches your threshold. On other plans, this could have a financial impact.
Secrets should be managed at deployment.

cosmir17 · 2025-11-21T00:32:53Z

spo-api/README.md

@@ -0,0 +1,163 @@
+# SPO API


it's not about this readme.md file.

midnight-indexer/justfile's generate-indexer-api-schema command can also generate a graphql schema file for your component and we can include the generated file in this pr as well?

cosmir17 · 2025-11-21T00:50:18Z

spo-api/src/application.rs

@@ -0,0 +1,49 @@
+// This file is part of midnight-indexer.


it's not about this file.

automated tests are needed.

Unit tests :

Unit tests following midnight-indexer conventions (inline #[cfg(test)] modules)

test functions across:

Helper functions (rpc.rs)

Config deserialization (config.rs)
...

Integration tests (can be follow-up PR):

main branch example, indexer-tests native_e2e.rs, e2e.rs

cosmir17 · 2025-11-21T01:08:10Z

spo-indexer/src/main.rs

@@ -0,0 +1,88 @@
+// This file is part of midnight-indexer.


It seems, sigterm codes are missing in spo-indexer codes (spo-indexer/src/main.rs and spo-indexer/src/application.rs).

they are needed in k8s envs.

cosmir17 · 2025-11-21T01:15:54Z

spo-indexer/config.yaml

+telemetry:
+  tracing:
+    enabled: false
+    service_name: "chain-indexer"


this seems wrong.
you want service_name: "spo-indexer" instead?

cosmir17 · 2025-11-21T01:25:55Z

spo-indexer/src/application.rs

+        loop {
+            ticker.tick().await;
+            if let Err(e) = refresh_stake_snapshots(&client_bg, &storage_bg, &st_cfg).await {
+                eprintln!("stake refresh failed: {e:?}");


I see :

println!("latest epoch reached"); println!("processing epoch {}", epoch.epoch_no); println!("\tcommittee size: {}", committee.len()); eprintln!("stake refresh failed: {e:?}"); eprintln!("stake refresh for {pid} failed: {err:?}");

the main branch's production code uses log::{debug, error, info, warn}; or context(...) instead..?

cosmir17 · 2025-11-21T01:34:29Z

spo-indexer/src/application.rs

+            continue;
+        }
+
+        let epoch = cur_epoch.unwrap();


I see :

let epoch = cur_epoch.unwrap(); // <- Could panic if None let raw_spo = val_to_registration.get(&spo_sk).unwrap(); // <- Could panic if missing produced_blocks: *produced.unwrap() as u64, // <- Could panic if None + if leftover > index.try_into().unwrap() { // <- Could panic on conversion let mut hasher = Blake2bVar::new(28).unwrap(); // <- Could panic (unlikely) hasher.finalize_variable(&mut buffer).unwrap(); // <- Could panic (unlikely)

indexer main branch code only uses unwrap for intended very few fatal error cases with comment. most of fatal cases, we use .expect(...)

recently, this has become famous : https://blog.cloudflare.com/18-november-2025-outage/#memory-preallocation

CLAassistant · 2025-11-26T16:24:38Z

All committers have signed the CLA.

- Replace println!/eprintln! with log::{debug, error, info, warn} - Replace .unwrap() with .expect() with meaningful messages - Add SIGTERM handling to spo-indexer (graceful shutdown) - Fix telemetry service_name: "chain-indexer" → "spo-indexer" - Remove Blockfrost ID from config.yaml (use env var instead) - Consolidate SPO SQL migrations into 003_spo.sql - Remove unused imports in spo-indexer

#661) * fix: update reqwest feature from rustls-tls to rustls for reqwest 0.13 compatibility * refactor(spo): apply PR #526 review comments - Replace println!/eprintln! with log::{debug, error, info, warn} - Replace .unwrap() with .expect() with meaningful messages - Add SIGTERM handling to spo-indexer (graceful shutdown) - Fix telemetry service_name: "chain-indexer" → "spo-indexer" - Remove Blockfrost ID from config.yaml (use env var instead) - Consolidate SPO SQL migrations into 003_spo.sql - Remove unused imports in spo-indexer * style(spo): apply indexer code style patterns Apply coding conventions: lowercase logs, inline format args, turbofish, to_owned(), doc comment periods, and imports at file top. * feat(spo-api): add GraphQL schema generation CLI Add spo-api-cli binary with print-api-schema-v1 command and justfile target to generate spo-api/graphql/schema-v1.graphql. * refactor(spo): address PR #661 review comments Apply coding style fixes including SIGTERM casing, typed APIs, proper error handling, and import patterns. Consolidate SPO migrations into 001_initial.sql. * refactor(spo): merge spo-api into indexer-api (#674) * refactor(spo): merge spo-api into indexer-api Move SPO GraphQL queries from standalone spo-api crate into indexer-api, following the existing storage trait abstraction pattern. This consolidates the API surface and eliminates the need for a separate SPO API service. Changes: - Add SPO domain types (SpoIdentity, PoolMetadata, EpochPerf, etc.) - Add SpoStorage trait with 18 storage methods - Implement SpoStorage for PostgreSQL (cloud) and SQLite (standalone) - Add SpoQuery GraphQL resolver with 20 query methods - Use MergedObject to combine Query and SpoQuery - Remove spo-api from workspace members - Update GraphQL schema with SPO queries * refactor(spo): fold SpoQuery into Query Move all SPO query methods from separate SpoQuery struct into the main Query struct, eliminating the need for MergedObject. This simplifies the GraphQL schema structure as suggested in PR review. * refactor(spo): address PR #661 review comments (2nd round) * refactor(spo): delete spo-api crate and add SQLite SPO tables - Delete spo-api/ directory (merged into indexer-api in PR #674) - Delete outdated MIGRATION.md (ProtoFire migration docs) - Remove spo-api service from docker-compose.yaml - Add SPO tables to SQLite migration for standalone mode

cosmir17 · 2026-01-31T18:06:09Z

Closing this PR — the work has been migrated to #735 since we don't have push access to the original forked branch. PR #735 includes all the original SPO indexer code plus insourcing cleanup (#661, #725, #727, #731) and a merge with current main (ledger 7.0.0, node 0.20.0).

#661) * fix: update reqwest feature from rustls-tls to rustls for reqwest 0.13 compatibility * refactor(spo): apply PR #526 review comments - Replace println!/eprintln! with log::{debug, error, info, warn} - Replace .unwrap() with .expect() with meaningful messages - Add SIGTERM handling to spo-indexer (graceful shutdown) - Fix telemetry service_name: "chain-indexer" → "spo-indexer" - Remove Blockfrost ID from config.yaml (use env var instead) - Consolidate SPO SQL migrations into 003_spo.sql - Remove unused imports in spo-indexer * style(spo): apply indexer code style patterns Apply coding conventions: lowercase logs, inline format args, turbofish, to_owned(), doc comment periods, and imports at file top. * feat(spo-api): add GraphQL schema generation CLI Add spo-api-cli binary with print-api-schema-v1 command and justfile target to generate spo-api/graphql/schema-v1.graphql. * refactor(spo): address PR #661 review comments Apply coding style fixes including SIGTERM casing, typed APIs, proper error handling, and import patterns. Consolidate SPO migrations into 001_initial.sql. * refactor(spo): merge spo-api into indexer-api (#674) * refactor(spo): merge spo-api into indexer-api Move SPO GraphQL queries from standalone spo-api crate into indexer-api, following the existing storage trait abstraction pattern. This consolidates the API surface and eliminates the need for a separate SPO API service. Changes: - Add SPO domain types (SpoIdentity, PoolMetadata, EpochPerf, etc.) - Add SpoStorage trait with 18 storage methods - Implement SpoStorage for PostgreSQL (cloud) and SQLite (standalone) - Add SpoQuery GraphQL resolver with 20 query methods - Use MergedObject to combine Query and SpoQuery - Remove spo-api from workspace members - Update GraphQL schema with SPO queries * refactor(spo): fold SpoQuery into Query Move all SPO query methods from separate SpoQuery struct into the main Query struct, eliminating the need for MergedObject. This simplifies the GraphQL schema structure as suggested in PR review. * refactor(spo): address PR #661 review comments (2nd round) * refactor(spo): delete spo-api crate and add SQLite SPO tables - Delete spo-api/ directory (merged into indexer-api in PR #674) - Delete outdated MIGRATION.md (ProtoFire migration docs) - Remove spo-api service from docker-compose.yaml - Add SPO tables to SQLite migration for standalone mode

* feat(spo): integrate spo indexer Originally authored by caparrosm (ProtoFire). Commit was unsigned and blocked PR merge due to repo requiring verified signatures. Amended author and re-signed to unblock. * fix(spo): apply review comments and coding style to ProtoFire SPO code (#661) * fix: update reqwest feature from rustls-tls to rustls for reqwest 0.13 compatibility * refactor(spo): apply PR #526 review comments - Replace println!/eprintln! with log::{debug, error, info, warn} - Replace .unwrap() with .expect() with meaningful messages - Add SIGTERM handling to spo-indexer (graceful shutdown) - Fix telemetry service_name: "chain-indexer" → "spo-indexer" - Remove Blockfrost ID from config.yaml (use env var instead) - Consolidate SPO SQL migrations into 003_spo.sql - Remove unused imports in spo-indexer * style(spo): apply indexer code style patterns Apply coding conventions: lowercase logs, inline format args, turbofish, to_owned(), doc comment periods, and imports at file top. * feat(spo-api): add GraphQL schema generation CLI Add spo-api-cli binary with print-api-schema-v1 command and justfile target to generate spo-api/graphql/schema-v1.graphql. * refactor(spo): address PR #661 review comments Apply coding style fixes including SIGTERM casing, typed APIs, proper error handling, and import patterns. Consolidate SPO migrations into 001_initial.sql. * refactor(spo): merge spo-api into indexer-api (#674) * refactor(spo): merge spo-api into indexer-api Move SPO GraphQL queries from standalone spo-api crate into indexer-api, following the existing storage trait abstraction pattern. This consolidates the API surface and eliminates the need for a separate SPO API service. Changes: - Add SPO domain types (SpoIdentity, PoolMetadata, EpochPerf, etc.) - Add SpoStorage trait with 18 storage methods - Implement SpoStorage for PostgreSQL (cloud) and SQLite (standalone) - Add SpoQuery GraphQL resolver with 20 query methods - Use MergedObject to combine Query and SpoQuery - Remove spo-api from workspace members - Update GraphQL schema with SPO queries * refactor(spo): fold SpoQuery into Query Move all SPO query methods from separate SpoQuery struct into the main Query struct, eliminating the need for MergedObject. This simplifies the GraphQL schema structure as suggested in PR review. * refactor(spo): address PR #661 review comments (2nd round) * refactor(spo): delete spo-api crate and add SQLite SPO tables - Delete spo-api/ directory (merged into indexer-api in PR #674) - Delete outdated MIGRATION.md (ProtoFire migration docs) - Remove spo-api service from docker-compose.yaml - Add SPO tables to SQLite migration for standalone mode * refactor(spo): move updated_at logic from DB trigger to application layer (PM-21550) (#725) * fix(spo): replace deprecated sidechain_getAriadneParameters RPC (#727) Use systemParameters_getAriadneParameters which sources the D Parameter from the on-chain pallet instead of Cardano. * refactor(spo): remove static metadata and OnlineClient from spo-indexer (#731) * refactor(spo): remove static metadata and OnlineClient from spo-indexer Delete ProtoFire's polkadot_metadata.scale, subxt macro, and build.rs. Replace get_block_timestamp with a DB query on the blocks table (populated by chain-indexer) and rewrite get_epoch_duration to use LegacyRpcMethods instead of OnlineClient. * feat(spo): add run-spo-indexer command to justfile * style(spo): apply rustfmt formatting * refactor: apply review feedback on get_block_timestamp * fix(spo): remove MIN_COMMITTEE_SIZE retry loop from get_committee The infinite retry loop waited for >= 300 committee members before proceeding, causing the spo-indexer to hang on all non-mainnet environments where committee sizes are much smaller (e.g. 6-10). * fix(spo): align Dockerfile with other indexer components * ci: add spo-indexer to Docker image build matrix (#744) * update Cargo.lock

integrate spo indexer

a37e167

caparrosm requested a review from a team as a code owner November 18, 2025 19:28

gilescope reviewed Nov 19, 2025

View reviewed changes

hseeberger changed the title ~~integrate spo indexer~~ feat: integrate spo indexer Nov 19, 2025

hseeberger requested changes Nov 19, 2025

View reviewed changes

cosmir17 reviewed Nov 21, 2025

View reviewed changes

cosmir17 mentioned this pull request Jan 13, 2026

fix(spo): apply review comments and coding style to ProtoFire SPO code #661

Merged

cosmir17 mentioned this pull request Jan 31, 2026

feat(spo): integrate SPO indexer into midnight-indexer #735

Merged

cosmir17 closed this Jan 31, 2026

cosmir17 mentioned this pull request Feb 5, 2026

chore: add spo-indexer docs, clean up unused NATS config and misleading blockfrost placeholder #754

Merged

feat: integrate spo indexer #526

feat: integrate spo indexer #526

Uh oh!

Conversation

caparrosm commented Nov 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cosmir17 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CLAassistant commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cosmir17 commented Jan 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

CLAassistant commented Nov 26, 2025 •

edited

Loading