Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add network tag to metrics #12733

Merged
merged 21 commits into from
Dec 18, 2024
Merged

feat: add network tag to metrics #12733

merged 21 commits into from
Dec 18, 2024

Conversation

virajbhartiya
Copy link
Member

Closes #12715

Add network tag to metrics

@virajbhartiya
Copy link
Member Author

@rvagg I am trying to setup a Grafana dashboard on my device to test this out. Can you please just let me know if I am going in the right direction with this PR

@rvagg
Copy link
Member

rvagg commented Nov 28, 2024

Yep, I think this is the right direction. It's unfortunate that you need to add it to all of the views but that looks like the trickiest bit here.

You should be able to register the value at startup, like in here:

lotus/cmd/lotus/daemon.go

Lines 211 to 216 in 07f2f69

ctx, _ := tag.New(context.Background(),
tag.Insert(metrics.Version, build.NodeBuildVersion),
tag.Insert(metrics.Commit, build.CurrentCommit),
tag.Insert(metrics.NodeType, "chain"),
)
// Register all metric views

And in here:

lotus/cmd/lotus-miner/run.go

Lines 58 to 63 in 07f2f69

ctx, _ := tag.New(lcli.DaemonContext(cctx),
tag.Insert(metrics.Version, build.MinerBuildVersion),
tag.Insert(metrics.Commit, build.CurrentCommit),
tag.Insert(metrics.NodeType, "miner"),
)
// Register all metric views

Currently I have grafana setup for my node and I'm scraping with prometheus, in my prometheus.yml scape_configs I have:

  - job_name: lotus-mainnet
    scrape_interval: 10s
    metrics_path: '/debug/metrics'
    static_configs:
      - targets: ['localhost:1234']
        labels:
          network: 'mainnet'
  - job_name: lotus-calibnet
    scrape_interval: 10s
    metrics_path: '/debug/metrics'
    static_configs:
      - targets: ['localhost:1235']
        labels:
          network: 'calibnet'

you can see my mainnet and calibnet tags being inserted there so I can do what you're doing here, but after the fact.

Then in grafana, when I explore metrics I can see the labels showing up that get reported for that metric type. All of mine have network but some have additional tags. In this one, network is there, but prometheus has also added instance and job, but the metric itself (block/failure) also has its own custom tag, from here:

lotus/chain/sub/incoming.go

Lines 466 to 473 in 07f2f69

func recordFailure(ctx context.Context, metric *stats.Int64Measure, failureType string) {
ctx, _ = tag.New(
ctx,
tag.Upsert(metrics.FailureType, failureType),
)
stats.Record(ctx, metric.M(1))
}

Screenshot 2024-11-28 at 4 57 22 pm

@virajbhartiya
Copy link
Member Author

We could get the network name from build/buildconstants/params.go we'll have to split the + to use it

@BigLep BigLep requested a review from rvagg December 3, 2024 08:41
rvagg
rvagg reviewed Dec 4, 2024
View reviewed changes
cmd/lotus-miner/run.go Outdated Show resolved Hide resolved
rvagg
rvagg reviewed Dec 4, 2024
View reviewed changes
cmd/lotus/daemon.go Outdated Show resolved Hide resolved
@Kubuxu
Copy link
Contributor

Kubuxu commented Dec 4, 2024

@rvagg I am trying to setup a Grafana dashboard on my device to test this out. Can you please just let me know if I am going in the right direction with this PR

You can do "curl localhost:1234/debug/metrics` (IIRC) to fetch these metrics to check

@rvagg
Copy link
Member

rvagg commented Dec 5, 2024

good hint 👌

$ curl -s http://localhost:1234/debug/metrics | grep block_failure
# HELP lotus_block_failure Counter for block validation failures
# TYPE lotus_block_failure counter
lotus_block_failure{failure_type=""} 193
lotus_block_failure{failure_type="unknown_miner"} 3557

so @virajbhartiya you should be able to compile a node, run it, do this curl and confirm that all of the metrics that you're editing have the network tag on them.

rvagg
rvagg reviewed Dec 5, 2024
View reviewed changes
cmd/lotus-miner/run.go Outdated Show resolved Hide resolved
@rvagg
Copy link
Member

rvagg commented Dec 5, 2024

Code seems good to go other than the stray "strings" in a file that looks like it was manually edited but not run through go fmt, just accepting my PR suggestions I guess? Well now there's a new suggestion that should do it.

If you could compile and run this and confirm that the tags appear on that /debug/metrics output that would be helpful @virajbhartiya . Otherwise I think this is good to go.

@BigLep BigLep mentioned this pull request Dec 6, 2024
Merged
@virajbhartiya
Copy link
Member Author

virajbhartiya commented Dec 10, 2024 •
edited
Loading 

curl -s http://localhost:1234/debug/metrics | grep network
lotus_info{commit="+git.1ab14eae2.dirty",network="calibrationnet",node_type="chain",version="1.31.1-dev"} 1
lotus_peer_count{network=""} 13
lotus_pubsub_prune{network=""} 80
lotus_pubsub_recv_rpc{network=""} 60
lotus_pubsub_rejected{network=""} 1
lotus_pubsub_send_rpc{network=""} 28
lotus_rcmgr_allow_conn{direction="inbound",network="",use_fd="true"} 1
lotus_rcmgr_allow_conn{direction="outbound",network="",use_fd="false"} 191
lotus_rcmgr_allow_conn{direction="outbound",network="",use_fd="true"} 165
lotus_rcmgr_allow_mem{network=""} 385
lotus_rcmgr_allow_peer{network="",peer_id=""} 357
lotus_rcmgr_allow_proto{network="",proto="/fil/chain/xchg/0.0.1"} 20
lotus_rcmgr_allow_proto{network="",proto="/fil/hello/1.0.0"} 76
lotus_rcmgr_allow_proto{network="",proto="/fil/kad/calibrationnet/kad/1.0.0"} 47
lotus_rcmgr_allow_proto{network="",proto="/ipfs/id/1.0.0"} 58
lotus_rcmgr_allow_proto{network="",proto="/ipfs/id/push/1.0.0"} 27
lotus_rcmgr_allow_proto{network="",proto="/ipfs/ping/1.0.0"} 4
lotus_rcmgr_allow_proto{network="",proto="/libp2p/autonat/1.0.0"} 2
lotus_rcmgr_allow_proto{network="",proto="/meshsub/1.1.0"} 58
lotus_rcmgr_allow_stream{direction="inbound",network="",peer_id=""} 139
lotus_rcmgr_allow_stream{direction="outbound",network="",peer_id=""} 163
lotus_rcmgr_allow_svc{network="",svc="libp2p.autonat"} 2
lotus_rcmgr_allow_svc{network="",svc="libp2p.identify"} 85
lotus_rcmgr_allow_svc{network="",svc="libp2p.ping"} 4
lotus_splitstore_miss{network=""} 926055

@rvagg
Copy link
Member

rvagg commented Dec 11, 2024 

lotus_pubsub_send_rpc{network=""} 28
lotus_rcmgr_allow_conn{direction="inbound",network="",use_fd="true"} 1
lotus_rcmgr_allow_conn{direction="outbound",network="",use_fd="false"} 191
lotus_rcmgr_allow_conn{direction="outbound",network="",use_fd="true"} 165
lotus_rcmgr_allow_mem{network=""} 385
lotus_rcmgr_allow_peer{network="",peer_id=""} 357

they're empty! something's going wrong for them not to be set in the main context .. I wonder if that context isn't being propagated through these subcomponents, or maybe set too late

@rvagg
Copy link
Member

rvagg commented Dec 11, 2024

@virajbhartiya can you dump a full output from that curl, I assume this is just partial? We only have the one correct one for lotus_info but that's reported not long after the ctx is initialised. Perhaps we have context breaks for some of these and I don't know if that's intentional or not, or how hard it is to find them. If they're all missing except for lotus_info then maybe something else is going on in the fx direct-injection tooling not to be passing on the context.

@rvagg
Copy link
Member

rvagg commented Dec 11, 2024

This might not be too hard to solve actually - for all of the metrics that don't fill out network, see if you can trace their contexts. It looks like we don't have too many in this boat:

$ git grep context.Background | grep -v '_test.*.go' | wc -l 
140

Even just hunting them down individually might not be hard. e.g. SplitstoreMiss is here:

lotus/blockstore/splitstore/splitstore.go

Line 415 in 12d76bd

stats.Record(s.ctx, metrics.SplitstoreMiss.M(1))

A quick search for s.ctx.*= shows it up here:

lotus/blockstore/splitstore/splitstore.go

Line 269 in 12d76bd

ss.ctx, ss.cancel = context.WithCancel(context.Background())

(Why it doesn't take a context argument is another question! But we'll leave that aside).

So in this case, you can put your tag on that context right where it's initialised.

The rcmgr metrics might not be hard either: if you look in node/modules/lp2p/rcmgr.go you'll find blocks like this:

lotus/node/modules/lp2p/rcmgr.go

Lines 195 to 209 in 12d76bd

func (r rcmgrMetrics) BlockConn(dir network.Direction, usefd bool) {
ctx := context.Background()
if dir == network.DirInbound {
ctx, _ = tag.New(ctx, tag.Upsert(metrics.Direction, "inbound"))
} else {
ctx, _ = tag.New(ctx, tag.Upsert(metrics.Direction, "outbound"))
}
if usefd {
ctx, _ = tag.New(ctx, tag.Upsert(metrics.UseFD, "true"))
} else {
ctx, _ = tag.New(ctx, tag.Upsert(metrics.UseFD, "false"))
}
stats.Record(ctx, metrics.RcmgrBlockConn.M(1))
}

Already setting up tags, just put the network tag value there too.

@virajbhartiya
Copy link
Member Author

mainnet

curl -s http://localhost:1234/debug/metrics | grep network
lotus_info{commit="+git.78e0bf5d6.dirty",network="mainnet",node_type="chain",version="1.32.1-dev"} 1
lotus_peer_count{network=""} 0
lotus_pubsub_prune{network=""} 2
lotus_pubsub_recv_rpc{network=""} 126
lotus_pubsub_send_rpc{network=""} 6
lotus_rcmgr_allow_conn{direction="outbound",network="mainnet",use_fd="false"} 299
lotus_rcmgr_allow_conn{direction="outbound",network="mainnet",use_fd="true"} 226
lotus_rcmgr_allow_mem{network=""} 2349
lotus_rcmgr_allow_peer{network="mainnet",peer_id=""} 524
lotus_rcmgr_allow_proto{network="mainnet",proto="/chain/ipfs/bitswap/1.2.0"} 388
lotus_rcmgr_allow_proto{network="mainnet",proto="/fil/hello/1.0.0"} 249
lotus_rcmgr_allow_proto{network="mainnet",proto="/fil/kad/testnetnet/kad/1.0.0"} 153
lotus_rcmgr_allow_proto{network="mainnet",proto="/ipfs/id/1.0.0"} 301
lotus_rcmgr_allow_proto{network="mainnet",proto="/ipfs/id/push/1.0.0"} 25
lotus_rcmgr_allow_proto{network="mainnet",proto="/ipfs/ping/1.0.0"} 1
lotus_rcmgr_allow_proto{network="mainnet",proto="/libp2p/autonat/1.0.0"} 10
lotus_rcmgr_allow_proto{network="mainnet",proto="/meshsub/1.1.0"} 288
lotus_rcmgr_allow_stream{direction="inbound",network="mainnet",peer_id=""} 813
lotus_rcmgr_allow_stream{direction="outbound",network="mainnet",peer_id=""} 622
lotus_rcmgr_allow_svc{network="mainnet",svc="libp2p.autonat"} 10
lotus_rcmgr_allow_svc{network="mainnet",svc="libp2p.identify"} 321
lotus_rcmgr_allow_svc{network="mainnet",svc="libp2p.ping"} 1
lotus_rcmgr_block_peer{network="mainnet",peer_id=""} 1
lotus_splitstore_miss{network="mainnet"} 2.280991e+06

@virajbhartiya
Copy link
Member Author

Calibration net (ran this a bit longer)

curl -s http://localhost:1234/debug/metrics | grep network
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="0.01"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="0.05"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="0.1"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="0.3"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="0.6"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="0.8"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="1"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="2"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="3"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="4"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="5"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="6"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="8"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="10"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="13"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="16"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="20"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="25"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="30"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="40"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="50"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="65"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="80"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="100"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="130"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="160"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="200"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="250"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="300"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="400"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="500"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="650"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="800"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="1000"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="2000"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="3000"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="4000"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="5000"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="7500"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="10000"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="20000"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="50000"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="100000"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="250000"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="500000"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="1e+06"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet",le="+Inf"} 1
lotus_api_request_duration_ms_sum{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet"} 0
lotus_api_request_duration_ms_count{api="lotus-daemon",endpoint="ChainHead",network="calibrationnet"} 1
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="0.01"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="0.05"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="0.1"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="0.3"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="0.6"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="0.8"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="1"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="2"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="3"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="4"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="5"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="6"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="8"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="10"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="13"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="16"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="20"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="25"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="30"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="40"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="50"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="65"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="80"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="100"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="130"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="160"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="200"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="250"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="300"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="400"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="500"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="650"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="800"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="1000"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="2000"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="3000"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="4000"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="5000"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="7500"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="10000"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="20000"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="50000"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="100000"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="250000"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="500000"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="1e+06"} 2
lotus_api_request_duration_ms_bucket{api="lotus-daemon",endpoint="SyncState",network="calibrationnet",le="+Inf"} 2
lotus_api_request_duration_ms_sum{api="lotus-daemon",endpoint="SyncState",network="calibrationnet"} 0
lotus_api_request_duration_ms_count{api="lotus-daemon",endpoint="SyncState",network="calibrationnet"} 2
lotus_block_failure{failure_type="",network=""} 4
lotus_block_received{network=""} 17
lotus_block_success{network=""} 13
lotus_block_validation_ms_bucket{network="",le="0.01"} 0
lotus_block_validation_ms_bucket{network="",le="0.05"} 0
lotus_block_validation_ms_bucket{network="",le="0.1"} 0
lotus_block_validation_ms_bucket{network="",le="0.3"} 0
lotus_block_validation_ms_bucket{network="",le="0.6"} 0
lotus_block_validation_ms_bucket{network="",le="0.8"} 0
lotus_block_validation_ms_bucket{network="",le="1"} 0
lotus_block_validation_ms_bucket{network="",le="2"} 0
lotus_block_validation_ms_bucket{network="",le="3"} 0
lotus_block_validation_ms_bucket{network="",le="4"} 0
lotus_block_validation_ms_bucket{network="",le="5"} 0
lotus_block_validation_ms_bucket{network="",le="6"} 3
lotus_block_validation_ms_bucket{network="",le="8"} 7
lotus_block_validation_ms_bucket{network="",le="10"} 9
lotus_block_validation_ms_bucket{network="",le="13"} 9
lotus_block_validation_ms_bucket{network="",le="16"} 10
lotus_block_validation_ms_bucket{network="",le="20"} 20
lotus_block_validation_ms_bucket{network="",le="25"} 24
lotus_block_validation_ms_bucket{network="",le="30"} 28
lotus_block_validation_ms_bucket{network="",le="40"} 31
lotus_block_validation_ms_bucket{network="",le="50"} 32
lotus_block_validation_ms_bucket{network="",le="65"} 32
lotus_block_validation_ms_bucket{network="",le="80"} 32
lotus_block_validation_ms_bucket{network="",le="100"} 32
lotus_block_validation_ms_bucket{network="",le="130"} 32
lotus_block_validation_ms_bucket{network="",le="160"} 32
lotus_block_validation_ms_bucket{network="",le="200"} 32
lotus_block_validation_ms_bucket{network="",le="250"} 32
lotus_block_validation_ms_bucket{network="",le="300"} 32
lotus_block_validation_ms_bucket{network="",le="400"} 32
lotus_block_validation_ms_bucket{network="",le="500"} 32
lotus_block_validation_ms_bucket{network="",le="650"} 32
lotus_block_validation_ms_bucket{network="",le="800"} 32
lotus_block_validation_ms_bucket{network="",le="1000"} 32
lotus_block_validation_ms_bucket{network="",le="2000"} 32
lotus_block_validation_ms_bucket{network="",le="3000"} 36
lotus_block_validation_ms_bucket{network="",le="4000"} 36
lotus_block_validation_ms_bucket{network="",le="5000"} 36
lotus_block_validation_ms_bucket{network="",le="7500"} 36
lotus_block_validation_ms_bucket{network="",le="10000"} 36
lotus_block_validation_ms_bucket{network="",le="20000"} 36
lotus_block_validation_ms_bucket{network="",le="50000"} 36
lotus_block_validation_ms_bucket{network="",le="100000"} 36
lotus_block_validation_ms_bucket{network="",le="250000"} 36
lotus_block_validation_ms_bucket{network="",le="500000"} 36
lotus_block_validation_ms_bucket{network="",le="1e+06"} 36
lotus_block_validation_ms_bucket{network="",le="+Inf"} 36
lotus_block_validation_ms_sum{network=""} 11559.000000000004
lotus_block_validation_ms_count{network=""} 36
lotus_chain_node_height{network=""} 2.219501e+06
lotus_chain_node_height_expected{network=""} 2.2195e+06
lotus_chain_node_worker_height{network=""} 2.219501e+06
lotus_info{commit="+git.be5d79a88",network="calibrationnet",node_type="chain",version="1.32.1-dev"} 1
lotus_message_received{network=""} 2
lotus_message_success{network=""} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="0.01"} 0
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="0.05"} 0
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="0.1"} 0
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="0.3"} 0
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="0.6"} 0
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="0.8"} 0
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="1"} 0
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="2"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="3"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="4"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="5"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="6"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="8"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="10"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="13"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="16"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="20"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="25"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="30"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="40"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="50"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="65"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="80"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="100"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="130"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="160"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="200"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="250"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="300"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="400"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="500"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="650"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="800"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="1000"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="2000"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="3000"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="4000"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="5000"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="7500"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="10000"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="20000"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="50000"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="100000"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="250000"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="500000"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="1e+06"} 2
lotus_message_validation_ms_bucket{local="",message_valid="true",network="",le="+Inf"} 2
lotus_message_validation_ms_sum{local="",message_valid="true",network=""} 2.694
lotus_message_validation_ms_count{local="",message_valid="true",network=""} 2
lotus_mpool_add_ms_bucket{network="",le="0.01"} 0
lotus_mpool_add_ms_bucket{network="",le="0.05"} 0
lotus_mpool_add_ms_bucket{network="",le="0.1"} 0
lotus_mpool_add_ms_bucket{network="",le="0.3"} 0
lotus_mpool_add_ms_bucket{network="",le="0.6"} 0
lotus_mpool_add_ms_bucket{network="",le="0.8"} 0
lotus_mpool_add_ms_bucket{network="",le="1"} 0
lotus_mpool_add_ms_bucket{network="",le="2"} 2
lotus_mpool_add_ms_bucket{network="",le="3"} 2
lotus_mpool_add_ms_bucket{network="",le="4"} 2
lotus_mpool_add_ms_bucket{network="",le="5"} 2
lotus_mpool_add_ms_bucket{network="",le="6"} 2
lotus_mpool_add_ms_bucket{network="",le="8"} 2
lotus_mpool_add_ms_bucket{network="",le="10"} 2
lotus_mpool_add_ms_bucket{network="",le="13"} 2
lotus_mpool_add_ms_bucket{network="",le="16"} 2
lotus_mpool_add_ms_bucket{network="",le="20"} 2
lotus_mpool_add_ms_bucket{network="",le="25"} 2
lotus_mpool_add_ms_bucket{network="",le="30"} 2
lotus_mpool_add_ms_bucket{network="",le="40"} 2
lotus_mpool_add_ms_bucket{network="",le="50"} 2
lotus_mpool_add_ms_bucket{network="",le="65"} 2
lotus_mpool_add_ms_bucket{network="",le="80"} 2
lotus_mpool_add_ms_bucket{network="",le="100"} 2
lotus_mpool_add_ms_bucket{network="",le="130"} 2
lotus_mpool_add_ms_bucket{network="",le="160"} 2
lotus_mpool_add_ms_bucket{network="",le="200"} 2
lotus_mpool_add_ms_bucket{network="",le="250"} 2
lotus_mpool_add_ms_bucket{network="",le="300"} 2
lotus_mpool_add_ms_bucket{network="",le="400"} 2
lotus_mpool_add_ms_bucket{network="",le="500"} 2
lotus_mpool_add_ms_bucket{network="",le="650"} 2
lotus_mpool_add_ms_bucket{network="",le="800"} 2
lotus_mpool_add_ms_bucket{network="",le="1000"} 2
lotus_mpool_add_ms_bucket{network="",le="2000"} 2
lotus_mpool_add_ms_bucket{network="",le="3000"} 2
lotus_mpool_add_ms_bucket{network="",le="4000"} 2
lotus_mpool_add_ms_bucket{network="",le="5000"} 2
lotus_mpool_add_ms_bucket{network="",le="7500"} 2
lotus_mpool_add_ms_bucket{network="",le="10000"} 2
lotus_mpool_add_ms_bucket{network="",le="20000"} 2
lotus_mpool_add_ms_bucket{network="",le="50000"} 2
lotus_mpool_add_ms_bucket{network="",le="100000"} 2
lotus_mpool_add_ms_bucket{network="",le="250000"} 2
lotus_mpool_add_ms_bucket{network="",le="500000"} 2
lotus_mpool_add_ms_bucket{network="",le="1e+06"} 2
lotus_mpool_add_ms_bucket{network="",le="+Inf"} 2
lotus_mpool_add_ms_sum{network=""} 2
lotus_mpool_add_ms_count{network=""} 2
lotus_mpool_addts_ms_bucket{network="",le="0.01"} 1
lotus_mpool_addts_ms_bucket{network="",le="0.05"} 1
lotus_mpool_addts_ms_bucket{network="",le="0.1"} 1
lotus_mpool_addts_ms_bucket{network="",le="0.3"} 1
lotus_mpool_addts_ms_bucket{network="",le="0.6"} 1
lotus_mpool_addts_ms_bucket{network="",le="0.8"} 1
lotus_mpool_addts_ms_bucket{network="",le="1"} 1
lotus_mpool_addts_ms_bucket{network="",le="2"} 2
lotus_mpool_addts_ms_bucket{network="",le="3"} 2
lotus_mpool_addts_ms_bucket{network="",le="4"} 2
lotus_mpool_addts_ms_bucket{network="",le="5"} 2
lotus_mpool_addts_ms_bucket{network="",le="6"} 2
lotus_mpool_addts_ms_bucket{network="",le="8"} 2
lotus_mpool_addts_ms_bucket{network="",le="10"} 2
lotus_mpool_addts_ms_bucket{network="",le="13"} 2
lotus_mpool_addts_ms_bucket{network="",le="16"} 2
lotus_mpool_addts_ms_bucket{network="",le="20"} 2
lotus_mpool_addts_ms_bucket{network="",le="25"} 2
lotus_mpool_addts_ms_bucket{network="",le="30"} 2
lotus_mpool_addts_ms_bucket{network="",le="40"} 2
lotus_mpool_addts_ms_bucket{network="",le="50"} 2
lotus_mpool_addts_ms_bucket{network="",le="65"} 2
lotus_mpool_addts_ms_bucket{network="",le="80"} 2
lotus_mpool_addts_ms_bucket{network="",le="100"} 2
lotus_mpool_addts_ms_bucket{network="",le="130"} 2
lotus_mpool_addts_ms_bucket{network="",le="160"} 2
lotus_mpool_addts_ms_bucket{network="",le="200"} 2
lotus_mpool_addts_ms_bucket{network="",le="250"} 2
lotus_mpool_addts_ms_bucket{network="",le="300"} 2
lotus_mpool_addts_ms_bucket{network="",le="400"} 2
lotus_mpool_addts_ms_bucket{network="",le="500"} 2
lotus_mpool_addts_ms_bucket{network="",le="650"} 2
lotus_mpool_addts_ms_bucket{network="",le="800"} 2
lotus_mpool_addts_ms_bucket{network="",le="1000"} 2
lotus_mpool_addts_ms_bucket{network="",le="2000"} 2
lotus_mpool_addts_ms_bucket{network="",le="3000"} 2
lotus_mpool_addts_ms_bucket{network="",le="4000"} 2
lotus_mpool_addts_ms_bucket{network="",le="5000"} 2
lotus_mpool_addts_ms_bucket{network="",le="7500"} 2
lotus_mpool_addts_ms_bucket{network="",le="10000"} 2
lotus_mpool_addts_ms_bucket{network="",le="20000"} 2
lotus_mpool_addts_ms_bucket{network="",le="50000"} 2
lotus_mpool_addts_ms_bucket{network="",le="100000"} 2
lotus_mpool_addts_ms_bucket{network="",le="250000"} 2
lotus_mpool_addts_ms_bucket{network="",le="500000"} 2
lotus_mpool_addts_ms_bucket{network="",le="1e+06"} 2
lotus_mpool_addts_ms_bucket{network="",le="+Inf"} 2
lotus_mpool_addts_ms_sum{network=""} 1
lotus_mpool_addts_ms_count{network=""} 2
lotus_mpool_getbalance_ms_bucket{network="",le="0.01"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="0.05"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="0.1"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="0.3"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="0.6"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="0.8"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="1"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="2"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="3"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="4"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="5"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="6"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="8"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="10"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="13"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="16"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="20"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="25"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="30"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="40"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="50"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="65"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="80"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="100"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="130"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="160"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="200"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="250"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="300"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="400"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="500"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="650"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="800"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="1000"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="2000"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="3000"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="4000"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="5000"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="7500"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="10000"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="20000"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="50000"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="100000"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="250000"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="500000"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="1e+06"} 2
lotus_mpool_getbalance_ms_bucket{network="",le="+Inf"} 2
lotus_mpool_getbalance_ms_sum{network=""} 0
lotus_mpool_getbalance_ms_count{network=""} 2
lotus_mpool_getnonce_ms_bucket{network="",le="0.01"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="0.05"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="0.1"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="0.3"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="0.6"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="0.8"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="1"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="2"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="3"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="4"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="5"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="6"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="8"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="10"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="13"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="16"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="20"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="25"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="30"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="40"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="50"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="65"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="80"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="100"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="130"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="160"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="200"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="250"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="300"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="400"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="500"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="650"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="800"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="1000"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="2000"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="3000"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="4000"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="5000"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="7500"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="10000"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="20000"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="50000"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="100000"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="250000"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="500000"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="1e+06"} 5
lotus_mpool_getnonce_ms_bucket{network="",le="+Inf"} 5
lotus_mpool_getnonce_ms_sum{network=""} 0
lotus_mpool_getnonce_ms_count{network=""} 5
lotus_mpool_message_count{network=""} 0
lotus_peer_count{network=""} 41
lotus_pubsub_delivered{network=""} 129
lotus_pubsub_duplicate{network=""} 1070
lotus_pubsub_prune{network=""} 134
lotus_pubsub_recv_rpc{network=""} 1525
lotus_pubsub_rejected{network=""} 27
lotus_pubsub_send_rpc{network=""} 1560
lotus_rcmgr_allow_conn{direction="inbound",network="calibrationnet",use_fd="false"} 17
lotus_rcmgr_allow_conn{direction="inbound",network="calibrationnet",use_fd="true"} 5
lotus_rcmgr_allow_conn{direction="outbound",network="calibrationnet",use_fd="false"} 971
lotus_rcmgr_allow_conn{direction="outbound",network="calibrationnet",use_fd="true"} 982
lotus_rcmgr_allow_mem{network=""} 1311
lotus_rcmgr_allow_peer{network="calibrationnet",peer_id=""} 1963
lotus_rcmgr_allow_proto{network="calibrationnet",proto="/f3/certexch/get/1/calibrationnet"} 10
lotus_rcmgr_allow_proto{network="calibrationnet",proto="/fil/chain/xchg/0.0.1"} 26
lotus_rcmgr_allow_proto{network="calibrationnet",proto="/fil/hello/1.0.0"} 244
lotus_rcmgr_allow_proto{network="calibrationnet",proto="/fil/kad/calibrationnet/kad/1.0.0"} 230
lotus_rcmgr_allow_proto{network="calibrationnet",proto="/ipfs/id/1.0.0"} 183
lotus_rcmgr_allow_proto{network="calibrationnet",proto="/ipfs/id/push/1.0.0"} 149
lotus_rcmgr_allow_proto{network="calibrationnet",proto="/ipfs/ping/1.0.0"} 22
lotus_rcmgr_allow_proto{network="calibrationnet",proto="/libp2p/autonat/1.0.0"} 5
lotus_rcmgr_allow_proto{network="calibrationnet",proto="/meshsub/1.1.0"} 159
lotus_rcmgr_allow_stream{direction="inbound",network="calibrationnet",peer_id=""} 454
lotus_rcmgr_allow_stream{direction="outbound",network="calibrationnet",peer_id=""} 738
lotus_rcmgr_allow_svc{network="calibrationnet",svc="libp2p.autonat"} 5
lotus_rcmgr_allow_svc{network="calibrationnet",svc="libp2p.identify"} 317
lotus_rcmgr_allow_svc{network="calibrationnet",svc="libp2p.ping"} 22
lotus_rcmgr_block_peer{network="calibrationnet",peer_id=""} 11
lotus_splitstore_miss{network="calibrationnet"} 6.73232e+06
lotus_vm_applied{network=""} 52
lotus_vm_applyblocks_cron_bucket{network="",le="0.01"} 0
lotus_vm_applyblocks_cron_bucket{network="",le="0.05"} 0
lotus_vm_applyblocks_cron_bucket{network="",le="0.1"} 0
lotus_vm_applyblocks_cron_bucket{network="",le="0.3"} 0
lotus_vm_applyblocks_cron_bucket{network="",le="0.6"} 0
lotus_vm_applyblocks_cron_bucket{network="",le="0.8"} 0
lotus_vm_applyblocks_cron_bucket{network="",le="1"} 0
lotus_vm_applyblocks_cron_bucket{network="",le="2"} 0
lotus_vm_applyblocks_cron_bucket{network="",le="3"} 3
lotus_vm_applyblocks_cron_bucket{network="",le="4"} 4
lotus_vm_applyblocks_cron_bucket{network="",le="5"} 5
lotus_vm_applyblocks_cron_bucket{network="",le="6"} 7
lotus_vm_applyblocks_cron_bucket{network="",le="8"} 9
lotus_vm_applyblocks_cron_bucket{network="",le="10"} 11
lotus_vm_applyblocks_cron_bucket{network="",le="13"} 11
lotus_vm_applyblocks_cron_bucket{network="",le="16"} 11
lotus_vm_applyblocks_cron_bucket{network="",le="20"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="25"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="30"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="40"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="50"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="65"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="80"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="100"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="130"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="160"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="200"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="250"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="300"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="400"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="500"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="650"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="800"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="1000"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="2000"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="3000"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="4000"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="5000"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="7500"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="10000"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="20000"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="50000"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="100000"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="250000"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="500000"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="1e+06"} 12
lotus_vm_applyblocks_cron_bucket{network="",le="+Inf"} 12
lotus_vm_applyblocks_cron_sum{network=""} 68.99999999999999
lotus_vm_applyblocks_cron_count{network=""} 12
lotus_vm_applyblocks_early_bucket{network="",le="0.01"} 11
lotus_vm_applyblocks_early_bucket{network="",le="0.05"} 11
lotus_vm_applyblocks_early_bucket{network="",le="0.1"} 11
lotus_vm_applyblocks_early_bucket{network="",le="0.3"} 11
lotus_vm_applyblocks_early_bucket{network="",le="0.6"} 11
lotus_vm_applyblocks_early_bucket{network="",le="0.8"} 11
lotus_vm_applyblocks_early_bucket{network="",le="1"} 11
lotus_vm_applyblocks_early_bucket{network="",le="2"} 11
lotus_vm_applyblocks_early_bucket{network="",le="3"} 11
lotus_vm_applyblocks_early_bucket{network="",le="4"} 11
lotus_vm_applyblocks_early_bucket{network="",le="5"} 11
lotus_vm_applyblocks_early_bucket{network="",le="6"} 11
lotus_vm_applyblocks_early_bucket{network="",le="8"} 12
lotus_vm_applyblocks_early_bucket{network="",le="10"} 12
lotus_vm_applyblocks_early_bucket{network="",le="13"} 12
lotus_vm_applyblocks_early_bucket{network="",le="16"} 12
lotus_vm_applyblocks_early_bucket{network="",le="20"} 12
lotus_vm_applyblocks_early_bucket{network="",le="25"} 12
lotus_vm_applyblocks_early_bucket{network="",le="30"} 12
lotus_vm_applyblocks_early_bucket{network="",le="40"} 12
lotus_vm_applyblocks_early_bucket{network="",le="50"} 12
lotus_vm_applyblocks_early_bucket{network="",le="65"} 12
lotus_vm_applyblocks_early_bucket{network="",le="80"} 12
lotus_vm_applyblocks_early_bucket{network="",le="100"} 12
lotus_vm_applyblocks_early_bucket{network="",le="130"} 12
lotus_vm_applyblocks_early_bucket{network="",le="160"} 12
lotus_vm_applyblocks_early_bucket{network="",le="200"} 12
lotus_vm_applyblocks_early_bucket{network="",le="250"} 12
lotus_vm_applyblocks_early_bucket{network="",le="300"} 12
lotus_vm_applyblocks_early_bucket{network="",le="400"} 12
lotus_vm_applyblocks_early_bucket{network="",le="500"} 12
lotus_vm_applyblocks_early_bucket{network="",le="650"} 12
lotus_vm_applyblocks_early_bucket{network="",le="800"} 12
lotus_vm_applyblocks_early_bucket{network="",le="1000"} 12
lotus_vm_applyblocks_early_bucket{network="",le="2000"} 12
lotus_vm_applyblocks_early_bucket{network="",le="3000"} 12
lotus_vm_applyblocks_early_bucket{network="",le="4000"} 12
lotus_vm_applyblocks_early_bucket{network="",le="5000"} 12
lotus_vm_applyblocks_early_bucket{network="",le="7500"} 12
lotus_vm_applyblocks_early_bucket{network="",le="10000"} 12
lotus_vm_applyblocks_early_bucket{network="",le="20000"} 12
lotus_vm_applyblocks_early_bucket{network="",le="50000"} 12
lotus_vm_applyblocks_early_bucket{network="",le="100000"} 12
lotus_vm_applyblocks_early_bucket{network="",le="250000"} 12
lotus_vm_applyblocks_early_bucket{network="",le="500000"} 12
lotus_vm_applyblocks_early_bucket{network="",le="1e+06"} 12
lotus_vm_applyblocks_early_bucket{network="",le="+Inf"} 12
lotus_vm_applyblocks_early_sum{network=""} 5.999999999999999
lotus_vm_applyblocks_early_count{network=""} 12
lotus_vm_applyblocks_flush_bucket{network="",le="0.01"} 11
lotus_vm_applyblocks_flush_bucket{network="",le="0.05"} 11
lotus_vm_applyblocks_flush_bucket{network="",le="0.1"} 11
lotus_vm_applyblocks_flush_bucket{network="",le="0.3"} 11
lotus_vm_applyblocks_flush_bucket{network="",le="0.6"} 11
lotus_vm_applyblocks_flush_bucket{network="",le="0.8"} 11
lotus_vm_applyblocks_flush_bucket{network="",le="1"} 11
lotus_vm_applyblocks_flush_bucket{network="",le="2"} 11
lotus_vm_applyblocks_flush_bucket{network="",le="3"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="4"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="5"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="6"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="8"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="10"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="13"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="16"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="20"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="25"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="30"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="40"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="50"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="65"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="80"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="100"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="130"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="160"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="200"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="250"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="300"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="400"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="500"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="650"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="800"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="1000"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="2000"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="3000"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="4000"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="5000"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="7500"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="10000"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="20000"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="50000"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="100000"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="250000"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="500000"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="1e+06"} 12
lotus_vm_applyblocks_flush_bucket{network="",le="+Inf"} 12
lotus_vm_applyblocks_flush_sum{network=""} 2
lotus_vm_applyblocks_flush_count{network=""} 12
lotus_vm_applyblocks_messages_bucket{network="",le="0.01"} 0
lotus_vm_applyblocks_messages_bucket{network="",le="0.05"} 0
lotus_vm_applyblocks_messages_bucket{network="",le="0.1"} 0
lotus_vm_applyblocks_messages_bucket{network="",le="0.3"} 0
lotus_vm_applyblocks_messages_bucket{network="",le="0.6"} 0
lotus_vm_applyblocks_messages_bucket{network="",le="0.8"} 0
lotus_vm_applyblocks_messages_bucket{network="",le="1"} 0
lotus_vm_applyblocks_messages_bucket{network="",le="2"} 0
lotus_vm_applyblocks_messages_bucket{network="",le="3"} 0
lotus_vm_applyblocks_messages_bucket{network="",le="4"} 0
lotus_vm_applyblocks_messages_bucket{network="",le="5"} 0
lotus_vm_applyblocks_messages_bucket{network="",le="6"} 0
lotus_vm_applyblocks_messages_bucket{network="",le="8"} 3
lotus_vm_applyblocks_messages_bucket{network="",le="10"} 5
lotus_vm_applyblocks_messages_bucket{network="",le="13"} 8
lotus_vm_applyblocks_messages_bucket{network="",le="16"} 10
lotus_vm_applyblocks_messages_bucket{network="",le="20"} 10
lotus_vm_applyblocks_messages_bucket{network="",le="25"} 10
lotus_vm_applyblocks_messages_bucket{network="",le="30"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="40"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="50"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="65"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="80"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="100"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="130"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="160"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="200"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="250"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="300"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="400"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="500"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="650"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="800"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="1000"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="2000"} 11
lotus_vm_applyblocks_messages_bucket{network="",le="3000"} 12
lotus_vm_applyblocks_messages_bucket{network="",le="4000"} 12
lotus_vm_applyblocks_messages_bucket{network="",le="5000"} 12
lotus_vm_applyblocks_messages_bucket{network="",le="7500"} 12
lotus_vm_applyblocks_messages_bucket{network="",le="10000"} 12
lotus_vm_applyblocks_messages_bucket{network="",le="20000"} 12
lotus_vm_applyblocks_messages_bucket{network="",le="50000"} 12
lotus_vm_applyblocks_messages_bucket{network="",le="100000"} 12
lotus_vm_applyblocks_messages_bucket{network="",le="250000"} 12
lotus_vm_applyblocks_messages_bucket{network="",le="500000"} 12
lotus_vm_applyblocks_messages_bucket{network="",le="1e+06"} 12
lotus_vm_applyblocks_messages_bucket{network="",le="+Inf"} 12
lotus_vm_applyblocks_messages_sum{network=""} 2847
lotus_vm_applyblocks_messages_count{network=""} 12
lotus_vm_applyblocks_total_ms_bucket{network="",le="0.01"} 0
lotus_vm_applyblocks_total_ms_bucket{network="",le="0.05"} 0
lotus_vm_applyblocks_total_ms_bucket{network="",le="0.1"} 0
lotus_vm_applyblocks_total_ms_bucket{network="",le="0.3"} 0
lotus_vm_applyblocks_total_ms_bucket{network="",le="0.6"} 0
lotus_vm_applyblocks_total_ms_bucket{network="",le="0.8"} 0
lotus_vm_applyblocks_total_ms_bucket{network="",le="1"} 0
lotus_vm_applyblocks_total_ms_bucket{network="",le="2"} 0
lotus_vm_applyblocks_total_ms_bucket{network="",le="3"} 0
lotus_vm_applyblocks_total_ms_bucket{network="",le="4"} 0
lotus_vm_applyblocks_total_ms_bucket{network="",le="5"} 0
lotus_vm_applyblocks_total_ms_bucket{network="",le="6"} 0
lotus_vm_applyblocks_total_ms_bucket{network="",le="8"} 0
lotus_vm_applyblocks_total_ms_bucket{network="",le="10"} 0
lotus_vm_applyblocks_total_ms_bucket{network="",le="13"} 2
lotus_vm_applyblocks_total_ms_bucket{network="",le="16"} 4
lotus_vm_applyblocks_total_ms_bucket{network="",le="20"} 7
lotus_vm_applyblocks_total_ms_bucket{network="",le="25"} 9
lotus_vm_applyblocks_total_ms_bucket{network="",le="30"} 10
lotus_vm_applyblocks_total_ms_bucket{network="",le="40"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="50"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="65"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="80"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="100"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="130"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="160"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="200"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="250"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="300"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="400"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="500"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="650"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="800"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="1000"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="2000"} 11
lotus_vm_applyblocks_total_ms_bucket{network="",le="3000"} 12
lotus_vm_applyblocks_total_ms_bucket{network="",le="4000"} 12
lotus_vm_applyblocks_total_ms_bucket{network="",le="5000"} 12
lotus_vm_applyblocks_total_ms_bucket{network="",le="7500"} 12
lotus_vm_applyblocks_total_ms_bucket{network="",le="10000"} 12
lotus_vm_applyblocks_total_ms_bucket{network="",le="20000"} 12
lotus_vm_applyblocks_total_ms_bucket{network="",le="50000"} 12
lotus_vm_applyblocks_total_ms_bucket{network="",le="100000"} 12
lotus_vm_applyblocks_total_ms_bucket{network="",le="250000"} 12
lotus_vm_applyblocks_total_ms_bucket{network="",le="500000"} 12
lotus_vm_applyblocks_total_ms_bucket{network="",le="1e+06"} 12
lotus_vm_applyblocks_total_ms_bucket{network="",le="+Inf"} 12
lotus_vm_applyblocks_total_ms_sum{network=""} 2937
lotus_vm_applyblocks_total_ms_count{network=""} 12
lotus_vm_execution_running{lane="priority",network=""} 0
lotus_vm_execution_waiting{lane="priority",network=""} 0
lotus_vm_sends{network=""} 0

rvagg
rvagg reviewed Dec 12, 2024
View reviewed changes
paychmgr/manager.go Outdated Show resolved Hide resolved
rvagg
rvagg reviewed Dec 12, 2024
View reviewed changes
storage/paths/db_index.go Outdated Show resolved Hide resolved
rvagg
rvagg reviewed Dec 16, 2024
View reviewed changes
storage/paths/db_index.go Outdated Show resolved Hide resolved
rvagg
rvagg reviewed Dec 16, 2024
View reviewed changes
storage/paths/local.go Outdated Show resolved Hide resolved
@rvagg
Copy link
Member

rvagg commented Dec 16, 2024

The biggest concern I've highlighted in here is adding tags late in the call chain instead of at the creation of the context. Ideally a global tag like this would be added wherever we make a new context, not where it's passed in - unless we have no control over context creation, which I think we should have in all cases. So I'd like to see if we can chase those down if possible. But don't spend hours doing this if you can't figure it out; because those all seem to be in lotus-miner code - everything in the storage package and its subpackages are used by lotus-miner and it's not as important as we address these (🤞 Curio deals with it and people end up using Curio!). Mostly we want to get the lotus daemon working properly.

@virajbhartiya
refactor(metrics): replace buildconstants with AddNetworkTag for cons…
01e8501 
…istent network tagging across multiple files
@virajbhartiya
Copy link
Member Author

The metrics context for cli execution comes from

lotus/cli/util/api.go

Lines 502 to 514 in 263636b

func ReqContext(cctx *cli.Context) context.Context {
tCtx := DaemonContext(cctx)
ctx, done := context.WithCancel(tCtx)
sigChan := make(chan os.Signal, 2)
go func() {
<-sigChan
done()
}()
signal.Notify(sigChan, syscall.SIGTERM, syscall.SIGINT, syscall.SIGHUP)
return ctx
}

and for daemon

lotus/cli/util/api.go

Lines 491 to 497 in 263636b

func DaemonContext(cctx *cli.Context) context.Context {
if mtCtx, ok := cctx.App.Metadata[metadataTraceContext]; ok {
return mtCtx.(context.Context)
}
return context.Background()
}

I was unable to find where the storage-miner gets its context from

@rvagg
Copy link
Member

rvagg commented Dec 16, 2024

miner probably comes from

lotus/cmd/lotus-miner/run.go

Lines 58 to 63 in 7f2068e

ctx, _ := tag.New(lcli.DaemonContext(cctx),
tag.Insert(metrics.Version, build.MinerBuildVersion),
tag.Insert(metrics.Commit, build.CurrentCommit),
tag.Insert(metrics.NodeType, "miner"),
)
// Register all metric views

Which also uses DaemonContext, and you've modified in here already. That should pass down into the lotus-miner RPC endpoint so you shouldn't need to augment it more and can remove those changes from the storage/ package and subpackages?

rvagg
rvagg reviewed Dec 17, 2024
View reviewed changes
CHANGELOG.md Outdated Show resolved Hide resolved
@virajbhartiya
chore(metrics): remove AddNetworkTag calls from multiple files and up…
fc56efa 
…date CHANGELOG to reflect network name reporting in metrics
rvagg
rvagg reviewed Dec 17, 2024
View reviewed changes
CHANGELOG.md Outdated Show resolved Hide resolved
rvagg
rvagg reviewed Dec 17, 2024
View reviewed changes
storage/paths/db_index.go Outdated Show resolved Hide resolved
rvagg
rvagg reviewed Dec 17, 2024
View reviewed changes
storage/paths/db_index.go Outdated Show resolved Hide resolved
rvagg
rvagg reviewed Dec 17, 2024
View reviewed changes
CHANGELOG.md Outdated Show resolved Hide resolved
@rvagg
Copy link
Member

rvagg commented Dec 17, 2024

Running locally on calibnet, ignoring ones that don't have network in them at all because we don't have control over them, we have these remaining to hunt down:

curl -s http://localhost:1235/debug/metrics| grep -v '^#' | grep 'network=""' | awk -F\{ '{print $1}' | uniq

lotus_block_failure
lotus_block_received
lotus_block_success
lotus_block_validation_ms_bucket
lotus_block_validation_ms_sum
lotus_block_validation_ms_count
lotus_chain_node_height
lotus_chain_node_height_expected
lotus_chain_node_worker_height
lotus_message_failure
lotus_message_received
lotus_message_success
lotus_message_validation_ms_bucket
lotus_message_validation_ms_sum
lotus_message_validation_ms_count
lotus_message_validation_ms_bucket
lotus_message_validation_ms_sum
lotus_message_validation_ms_count
lotus_mpool_add_ms_bucket
lotus_mpool_add_ms_sum
lotus_mpool_add_ms_count
lotus_mpool_addts_ms_bucket
lotus_mpool_addts_ms_sum
lotus_mpool_addts_ms_count
lotus_mpool_getbalance_ms_bucket
lotus_mpool_getbalance_ms_sum
lotus_mpool_getbalance_ms_count
lotus_mpool_getnonce_ms_bucket
lotus_mpool_getnonce_ms_sum
lotus_mpool_getnonce_ms_count
lotus_mpool_message_count
lotus_peer_count
lotus_pubsub_delivered
lotus_pubsub_duplicate
lotus_pubsub_prune
lotus_pubsub_recv_rpc
lotus_pubsub_rejected
lotus_pubsub_send_rpc
lotus_rcmgr_allow_mem
lotus_vm_applied
lotus_vm_applyblocks_cron_bucket
lotus_vm_applyblocks_cron_sum
lotus_vm_applyblocks_cron_count
lotus_vm_applyblocks_early_bucket
lotus_vm_applyblocks_early_sum
lotus_vm_applyblocks_early_count
lotus_vm_applyblocks_flush_bucket
lotus_vm_applyblocks_flush_sum
lotus_vm_applyblocks_flush_count
lotus_vm_applyblocks_messages_bucket
lotus_vm_applyblocks_messages_sum
lotus_vm_applyblocks_messages_count
lotus_vm_applyblocks_total_ms_bucket
lotus_vm_applyblocks_total_ms_sum
lotus_vm_applyblocks_total_ms_count
lotus_vm_execution_running
lotus_vm_execution_waiting
lotus_vm_sends

vm_applyblocks is particularly annoying because it's one of the more useful metrics.

@rvagg
Copy link
Member

rvagg commented Dec 17, 2024

There's so many context.Background and context.TODO in the codebase it's a little scary .. we might have to just call this good enough and open a ticket to plumb contexts properly where possible. I wouldn't mind figuring out where the applyblocks context comes from though, that one's bothering me and I can't work it out.

@rvagg rvagg mentioned this pull request Dec 17, 2024
Draft
@rvagg
Copy link
Member

rvagg commented Dec 17, 2024

#12793 I think I'll stop with that one and we should rethink how to handle this; we might merge but with an adjusted, or missing CHANGELOG entry.

rvagg
rvagg reviewed Dec 18, 2024
View reviewed changes
CHANGELOG.md Outdated Show resolved Hide resolved
@rvagg rvagg enabled auto-merge (squash) December 18, 2024 06:10
@rvagg rvagg mentioned this pull request Dec 18, 2024
Closed
@rvagg rvagg merged commit a584745 into filecoin-project:master Dec 18, 2024
81 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rvagg rvagg rvagg approved these changes

Assignees

@virajbhartiya virajbhartiya

Labels
None yet
Projects
FilOz
Status: ☑️ Done (Archive)
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Tag metrics with a network name/network tag
3 participants
@virajbhartiya @rvagg @Kubuxu