missing OSD hosts specific data

[stef97]

Hi,
Why I do not get any hosts data ( like IOPS,latency, Network ...etc )?

Any help/troubleshooting tips will be appreciated ?

node_exported seems to be up and running and I am getting cluster data

[root@osd02 ~]# curl http://127.0.0.1:9100/metrics

HELP go_gc_duration_seconds A summary of the GC invocation durations.

TYPE go_gc_duration_seconds summary

go_gc_duration_seconds{quantile="0"} 4.503e-05
go_gc_duration_seconds{quantile="0.25"} 6.8586e-05
go_gc_duration_seconds{quantile="0.5"} 0.000106391
go_gc_duration_seconds{quantile="0.75"} 0.000177064
go_gc_duration_seconds{quantile="1"} 0.000409363
go_gc_duration_seconds_sum 4.629163632
go_gc_duration_seconds_count 36718

HELP go_goroutines Number of goroutines that currently exist.

TYPE go_goroutines gauge

go_goroutines 8

HELP go_info Information about the Go environment.

TYPE go_info gauge

go_info{version="go1.9.6"} 1

HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.

TYPE go_memstats_alloc_bytes gauge

go_memstats_alloc_bytes 2.691984e+06

HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.

TYPE go_memstats_alloc_bytes_total counter

go_memstats_alloc_bytes_total 4.8003293944e+10

HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.

TYPE go_memstats_buck_hash_sys_bytes gauge

go_memstats_buck_hash_sys_bytes 2.125448e+06

HELP go_memstats_frees_total Total number of frees.

TYPE go_memstats_frees_total counter

go_memstats_frees_total 4.50034703e+08

HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.

TYPE go_memstats_gc_cpu_fraction gauge

go_memstats_gc_cpu_fraction 3.872588199040138e-05

HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.

TYPE go_memstats_gc_sys_bytes gauge

go_memstats_gc_sys_bytes 479232

HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.

TYPE go_memstats_heap_alloc_bytes gauge

go_memstats_heap_alloc_bytes 2.691984e+06

HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.

TYPE go_memstats_heap_idle_bytes gauge

go_memstats_heap_idle_bytes 2.424832e+06

HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.

TYPE go_memstats_heap_inuse_bytes gauge

go_memstats_heap_inuse_bytes 4.620288e+06

HELP go_memstats_heap_objects Number of allocated objects.

TYPE go_memstats_heap_objects gauge

go_memstats_heap_objects 10778

HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.

TYPE go_memstats_heap_released_bytes gauge

go_memstats_heap_released_bytes 0

HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.

TYPE go_memstats_heap_sys_bytes gauge

go_memstats_heap_sys_bytes 7.04512e+06

HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.

TYPE go_memstats_last_gc_time_seconds gauge

go_memstats_last_gc_time_seconds 1.5353792759204392e+09

HELP go_memstats_lookups_total Total number of pointer lookups.

TYPE go_memstats_lookups_total counter

go_memstats_lookups_total 4.133831e+06

HELP go_memstats_mallocs_total Total number of mallocs.

TYPE go_memstats_mallocs_total counter

go_memstats_mallocs_total 4.50045481e+08

HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.

TYPE go_memstats_mcache_inuse_bytes gauge

go_memstats_mcache_inuse_bytes 20832

HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.

TYPE go_memstats_mcache_sys_bytes gauge

go_memstats_mcache_sys_bytes 32768

HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.

TYPE go_memstats_mspan_inuse_bytes gauge

go_memstats_mspan_inuse_bytes 73568

HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.

TYPE go_memstats_mspan_sys_bytes gauge

go_memstats_mspan_sys_bytes 98304

HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.

TYPE go_memstats_next_gc_bytes gauge

go_memstats_next_gc_bytes 4.853152e+06

HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.

TYPE go_memstats_other_sys_bytes gauge

go_memstats_other_sys_bytes 2.644592e+06

HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.

TYPE go_memstats_stack_inuse_bytes gauge

go_memstats_stack_inuse_bytes 1.343488e+06

HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.

TYPE go_memstats_stack_sys_bytes gauge

go_memstats_stack_sys_bytes 1.343488e+06

HELP go_memstats_sys_bytes Number of bytes obtained from system.

TYPE go_memstats_sys_bytes gauge

go_memstats_sys_bytes 1.3768952e+07

HELP go_threads Number of OS threads created.

TYPE go_threads gauge

go_threads 29

HELP node_arp_entries ARP entries by device

TYPE node_arp_entries gauge

node_arp_entries{device="gth0"} 5
node_arp_entries{device="gth1"} 2

[stef97]

Just to clarify further, it seems prometheus is receiving data because I am able to query it using http://server:9090/graph

Why grafana is not showing it ?

Any help will be VERY appreciated

Thanks
Steven

[stef97]

Upon further investigation, I noticed that grafana is using/ expecting data with different name
Please clarify - I would really like to use cephmetrics with prometheus

Example ( latency by server):
node_disk_read_time_ms

Node exported does not have that
curl http://osd01:9100/metrics | grep node_disk_read_time
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0# HELP node_disk_read_time_seconds_total The total number of milliseconds spent by all reads.

TYPE node_disk_read_time_seconds_total counter

node_disk_read_time_seconds_total{device="dm-0"} 8910.801
node_disk_read_time_seconds_total{device="sda"} 0.525
node_disk_read_time_seconds_total{device="sdb"} 14221.732
node_disk_read_time_seconds_total{device="sdc"} 0.465
node_disk_read_time_seconds_total{device="sdd"} 0.46
node_disk_read_time_seconds_total{device="sde"} 0.017
node_disk_read_time_seconds_total{device="sdf"} 455.064
node_disk_read_time_seconds_total{device="sr0"} 0
100 64683 100 64683 0 0 4452k 0 --:--:-- --:--:-- --:--:-- 5263k

[pcuzner]

use node_exporter 0.15. Version 0.16 changed the names of the metrics, and arrived too late for us to work against

[stef97]

That helped - thank you The only missing info is network related - any ideas ? I am using bonding but it seems bond collector is enabled on node_exported Also, my device names are different ( eth4, gth0 ..etc) Prometheus has metrics ( see below) PS Just wanted to emphasize that the fact that you take time to respond is really appreciated node_network_receive_bytes{device="gth0",instance="osd01",job="node"} 106535337552 node_network_receive_bytes{device="gth0",instance="osd02",job="node"} 4490848688 node_network_receive_bytes{device="gth0",instance="osd04",job="node"} 4903607155 node_network_receive_bytes{device="gth1",instance="osd01",job="node"} 39710010837 node_network_receive_bytes{device="gth1",instance="osd02",job="node"} 6992071413 node_network_receive_bytes{device="gth1",instance="osd04",job="node"} 7111182860 node_network_receive_bytes{device="idrac",instance="osd04",job="node"} 0 node_network_receive_bytes{device="lo",instance="osd01",job="node"} 92896628087 node_network_receive_bytes{device="lo",instance="osd02",job="node"} 405423272 node_network_receive_bytes{device="lo",instance="osd04",job="node"} 403367859

…

On Mon, 27 Aug 2018 at 16:24, pcuzner ***@***.***> wrote: use node_exporter 0.15. Version 0.16 changed the names of the metrics, and arrived too late for us to work against — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#223 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJn1v8vQGPdfDbRxkTt0qtJyBHRG3Irgks5uVFVlgaJpZM4WN3sY> .

[pcuzner]

we whitelist the interface names in the query templates to those that we've encountered. You can update your templates/dashboards locally to get the data to appear. Apologies - I haven't seen gthX before!

[stef97]

Hi,
I've added NVME devices to my OSDs and this seems to break node_exporter
" ...msg="ERROR: diskstats collector failed "

Any suggestions / ideas will be greatly appreciated

/proc/diskstats is as follows
cat /proc/diskstats
259 0 nvme0n1 55381 0 4027200 2463 121 2631 22016 61 0 17535 19241 0 0 0 0
259 1 nvme0n1p1 608 0 46368 21 0 0 0 0 0 21 21 0 0 0 0
259 2 nvme0n1p2 661 0 48672 22 0 0 0 0 0 24 26 0 0 0 0
259 3 nvme0n1p3 560 0 42176 19 0 0 0 0 0 13 14 0 0 0 0
259 4 nvme0n1p4 631 0 48432 23 0 0 0 0 0 26 27 0 0 0 0
259 6 nvme0n1p5 631 0 48432 24 0 0 0 0 0 22 24 0 0 0 0
259 7 nvme0n1p6 560 0 42176 20 0 0 0 0 0 16 17 0 0 0 0
259 10 nvme0n1p7 631 0 48432 22 0 0 0 0 0 18 19 0 0 0 0
259 11 nvme0n1p8 631 0 48432 23 0 0 0 0 0 22 25 0 0 0 0
259 17 nvme0n1p9 560 0 42176 20 0 0 0 0 0 17 24 0 0 0 0
259 20 nvme0n1p10 631 0 48432 22 0 0 0 0 0 26 26 0 0 0 0
259 21 nvme0n1p11 631 0 48432 22 0 0 0 0 0 14 16 0 0 0 0
259 24 nvme0n1p12 631 0 48432 23 0 0 0 0 0 17 18 0 0 0 0
259 25 nvme0n1p13 583 0 44240 22 0 0 0 0 0 23 26 0 0 0 0
259 28 nvme0n1p14 631 0 48432 23 0 0 0 0 0 20 24 0 0 0 0
259 29 nvme0n1p15 535 0 40048 20 0 0 0 0 0 22 26 0 0 0 0
259 31 nvme0n1p16 606 0 46304 22 0 0 0 0 0 26 31 0 0 0 0
8 0 sda 10823138 0 94504280 1571212 26005951 19406 208223048 2236550 0 365787 3808349 0 0 0 0
8 16 sdb 17940 0 923160 1227 40 2520 20480 10 0 720 1158 0 0 0 0
8 32 sdc 17940 0 923160 1047 40 2520 20480 10 0 705 1189 0 0 0 0
8 48 sdd 17940 0 923160 1316 40 2520 20480 9 0 730 1205 0 0 0 0
8 64 sde 17940 0 923160 1292 40 2520 20480 10 0 787 1385 0 0 0 0
8 80 sdf 40684 282 1113465 140696 74404 7192 3272067 43079689 0 3633695 43197380 0 0 0 0
8 81 sdf1 322 0 18927 2832 24 0 4136 21254 0 17769 24084 0 0 0 0
8 82 sdf2 14901 267 687400 76779 27742 4760 975175 27664431 0 2085479 27734760 0 0 0 0
8 83 sdf3 259 0 11512 1156 0 0 0 0 0 773 1159 0 0 0 0
8 84 sdf4 8 0 52 26 0 0 0 0 0 26 26 0 0 0 0
8 85 sdf5 25078 14 391214 59265 46638 2432 2292756 15394003 0 1824345 15436722 0 0 0 0
11 0 sr0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
253 0 dm-0 670 0 63808 150 419 0 3352 82 0 84 232 0 0 0 0

[stef97]

It appears node_exporter had an issue with kernel 4.19
prometheus/node_exporter#1049

The problem now is that If I upgrade node_exporter, cephmetrics does not get data anymore , if I dont I'll need to downgrade the kernel

Any plans to change code so there will be no need to use node_exporter version 15 ?

Many thanks

[pcuzner]

The release of cephmetrics that's integrated into the mgr/dashboard accounts for the naming in node_exporter 0.15 and 0.16.

At this point, the standalone cephmetrics project will remain 0.15