Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Addition of pcp2opentelemetry tool #1972

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 25 additions & 3 deletions build/rpm/pcp.spec.in
Original file line number Diff line number Diff line change
Expand Up @@ -332,8 +332,8 @@ Requires: pcp-pmda-nvidia-gpu pcp-pmda-roomtemp pcp-pmda-sendmail pcp-pmda-shpin
Requires: pcp-pmda-lustrecomm pcp-pmda-docker pcp-pmda-smart pcp-pmda-farm
Requires: pcp-pmda-hacluster pcp-pmda-sockets pcp-pmda-podman
%if "@have_python@" == "true"
Requires: pcp-geolocate pcp-export-pcp2openmetrics pcp-export-pcp2json
Requires: pcp-export-pcp2spark pcp-export-pcp2xml pcp-export-pcp2zabbix
Requires: pcp-geolocate pcp-export-pcp2openmetrics pcp-export-pcp2opentelemetry
Requires: pcp-export-pcp2json pcp-export-pcp2spark pcp-export-pcp2xml pcp-export-pcp2zabbix
Requires: pcp-pmda-gluster pcp-pmda-zswap pcp-pmda-unbound pcp-pmda-mic
Requires: pcp-pmda-haproxy pcp-pmda-nfsclient pcp-pmda-lmsensors
Requires: pcp-pmda-netcheck pcp-pmda-rabbitmq pcp-pmda-uwsgi
Expand Down Expand Up @@ -704,6 +704,24 @@ Requires: %{__python2}-pcp = @package_version@
Performance Co-Pilot (PCP) front-end tools for exporting metric values
in OpenMetrics (https://openmetrics.io/) format.

#
# pcp-export-pcp2opentelemetry
#
%package export-pcp2opentelemetry
License: GPL-2.0-or-later
Summary: Performance Co-Pilot tools for exporting PCP metrics in OpenTelemetry format
URL: https://pcp.io
Requires: pcp-libs >= @package_version@
%if "@enable_python3@" == "true"
Requires: python3-pcp = @package_version@
%else
Requires: %{__python2}-pcp = @package_version@
%endif

%description export-pcp2opentelemetry
Performance Co-Pilot (PCP) front-end tools for exporting metric values
in OpenTelemetry (https://opentelemetry.io/) format.

#
# pcp-export-pcp2spark
#
Expand Down Expand Up @@ -2421,6 +2439,7 @@ basic_manifest | keep 'pcp2xlsx' >pcp-export-pcp2xlsx-files
basic_manifest | keep 'pcp2graphite' >pcp-export-pcp2graphite-files
basic_manifest | keep 'pcp2json' >pcp-export-pcp2json-files
basic_manifest | keep 'pcp2openmetrics' >pcp-export-pcp2openmetrics-files
basic_manifest | keep 'pcp2opentelemetry' >pcp-export-pcp2opentelemetry-files
basic_manifest | keep 'pcp2spark' >pcp-export-pcp2spark-files
basic_manifest | keep 'pcp2xml' >pcp-export-pcp2xml-files
basic_manifest | keep 'pcp2zabbix' >pcp-export-pcp2zabbix-files
Expand Down Expand Up @@ -2539,7 +2558,8 @@ done

for export_package in \
pcp2arrow pcp2elasticsearch pcp2graphite pcp2influxdb pcp2json \
pcp2openmetrics pcp2spark pcp2xlsx pcp2xml pcp2zabbix zabbix-agent ; \
pcp2openmetrics pcp2opentelemetry pcp2spark pcp2xlsx pcp2xml \
pcp2zabbix zabbix-agent; \
do \
export_packages="$export_packages pcp-export-$export_package"; \
done
Expand Down Expand Up @@ -3258,6 +3278,8 @@ fi

%files export-pcp2openmetrics -f pcp-export-pcp2openmetrics-files.rpm

%files export-pcp2opentelemetry -f pcp-export-pcp2opentelemetry-files.rpm

%files export-pcp2spark -f pcp-export-pcp2spark-files.rpm

%files export-pcp2xml -f pcp-export-pcp2xml-files.rpm
Expand Down
28 changes: 25 additions & 3 deletions build/rpm/redhat.spec
Original file line number Diff line number Diff line change
Expand Up @@ -568,8 +568,8 @@ Requires: pcp-pmda-bpf
Requires: pcp-pmda-bpftrace
%endif
%if !%{disable_python2} || !%{disable_python3}
Requires: pcp-geolocate pcp-export-pcp2openmetrics pcp-export-pcp2json
Requires: pcp-export-pcp2spark pcp-export-pcp2xml pcp-export-pcp2zabbix
Requires: pcp-geolocate pcp-export-pcp2openmetrics pcp-export-pcp2opentelemetry
Requires: pcp-export-pcp2json pcp-export-pcp2spark pcp-export-pcp2xml pcp-export-pcp2zabbix
Requires: pcp-pmda-gluster pcp-pmda-zswap pcp-pmda-unbound pcp-pmda-mic
Requires: pcp-pmda-libvirt pcp-pmda-lio pcp-pmda-openmetrics pcp-pmda-haproxy
Requires: pcp-pmda-lmsensors pcp-pmda-netcheck pcp-pmda-rabbitmq pcp-pmda-uwsgi
Expand Down Expand Up @@ -878,6 +878,24 @@ Requires: %{__python2}-pcp = %{version}-%{release}
Performance Co-Pilot (PCP) front-end tools for exporting metric values
in OpenMetrics (https://openmetrics.io/) format.

#
# pcp-export-pcp2opentelemetry
#
%package export-pcp2opentelemetry
License: GPL-2.0-or-later
Summary: Performance Co-Pilot tools for exporting PCP metrics in OpenTelemetry format
URL: https://pcp.io
Requires: pcp-libs >= %{version}-%{release}
%if !%{disable_python3}
Requires: python3-pcp = %{version}-%{release}
%else
Requires: %{__python2}-pcp = %{version}-%{release}
%endif

%description export-pcp2opentelemetry
Performance Co-Pilot (PCP) front-end tools for exporting metric values
in OpenTelemetry (https://opentelemetry.io/) format.

#
# pcp-export-pcp2spark
#
Expand Down Expand Up @@ -2593,6 +2611,7 @@ basic_manifest | keep 'pcp2xlsx' >pcp-export-pcp2xlsx-files
basic_manifest | keep 'pcp2graphite' >pcp-export-pcp2graphite-files
basic_manifest | keep 'pcp2json' >pcp-export-pcp2json-files
basic_manifest | keep 'pcp2openmetrics' >pcp-export-pcp2openmetrics-files
basic_manifest | keep 'pcp2opentelemetry' >pcp-export-pcp2opentelemetry-files
basic_manifest | keep 'pcp2spark' >pcp-export-pcp2spark-files
basic_manifest | keep 'pcp2xml' >pcp-export-pcp2xml-files
basic_manifest | keep 'pcp2zabbix' >pcp-export-pcp2zabbix-files
Expand Down Expand Up @@ -2712,7 +2731,8 @@ done

for export_package in \
pcp2arrow pcp2elasticsearch pcp2graphite pcp2influxdb pcp2json \
pcp2openmetrics pcp2spark pcp2xlsx pcp2xml pcp2zabbix zabbix-agent ; \
pcp2openmetrics pcp2spark pcp2xlsx pcp2xml pcp2opentelemetry \
pcp2zabbix zabbix-agent ; \
do \
export_packages="$export_packages pcp-export-$export_package"; \
done
Expand Down Expand Up @@ -3411,6 +3431,8 @@ fi

%files export-pcp2openmetrics -f pcp-export-pcp2openmetrics-files.rpm

%files export-pcp2opentelemetry -f pcp-export-pcp2opentelemetry-files.rpm

%files export-pcp2spark -f pcp-export-pcp2spark-files.rpm

%files export-pcp2xml -f pcp-export-pcp2xml-files.rpm
Expand Down
19 changes: 18 additions & 1 deletion qa/1131
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/sh
# PCP QA Test No. 1131
# Exercise pcp2json and pcp2openmetrics.
# Exercise pcp2json, pcp2openmetrics, and pcp2opentelemetry.
#
# Copyright (c) 2017 Red Hat.
#
Expand Down Expand Up @@ -52,6 +52,21 @@ _filter_pcp2openmetrics()
| LC_COLLATE=POSIX sort
}

_filter_pcp2opentelemetry()
{
tee -a $here/$seq.full \
| col -b \
| sed \
-e '/\"asDouble\":/ s/[0-9][0-9]*/NCPU/' \
-e 's/\"'$machineid'\"/MACHINEID/' \
-e 's/\"'$hostname'\"/HOSTNAME/' \
-e 's/\"'$domainid'\"/DOMAINID/' \
-e 's/\"agent\": .*/\"agent\": \"'$OSTYPE'\"/' \
-e 's/\"startTimeUnixNano\": .*/\"startTimeUnixNano\": TIME/' \
-e 's/\"userid\": .*/\"userid\": USERID/' \
-e 's/\"userid\": .*/\"userid\": GROUPID/'
}

# real QA test starts here
echo "---"
pcp2json -a $A -H -I -z "" | _archive_filter
Expand All @@ -64,6 +79,8 @@ pcp2json -a $A -H -I -z -X -b GB -P 2 -F $tmp.outfile ""
echo "---"
pcp2openmetrics -s1 -H -z hinv.ncpu | _filter_pcp2openmetrics
echo "---"
pcp2opentelemetry -s1 -H -z hinv.ncpu | _filter_pcp2opentelemetry
echo "---"


cat $tmp.outfile | _archive_filter
Expand Down
56 changes: 55 additions & 1 deletion qa/1131.out
Original file line number Diff line number Diff line change
Expand Up @@ -2269,10 +2269,64 @@ QA output created by 1131
---


# HELP hinv_ncpu number of CPUs in the system
# HELP hinv_ncpu b'number of CPUs in the system'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to upgrade your PCP python installation (./Makepkgs FTW) - this b'...' issue will be resolved then.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@natoscott I upgraded the packages, ran configure and make and still the b' issue was still there when I re ran the test?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that is a mystery - its definitely fixed for me (and in CI). I wonder if somehow an older version of the python library is being found somewhere on your system, unexpectedly.

# PCP5 hinv_ncpu 60.0.32 u32 PM_INDOM_NULL discrete
# TYPE hinv_ncpu gauge
hinv_ncpu{domainname="DOMAINID",groupid="GROUPID",hostname="HOST",machineid="MACHINEID",userid="USERID",agent="linux"} NCPU
---
{
"resourceMetrics": [
{
"resource": {
"attributes": [
{
"os.type": "linux",
"service.name": "pcp",
"telemetry.sdk.language": "python",
"telemetry.sdk.name": "opentelemetry",
"telemetry.sdk.version": "1.24.0"
}
]
},
"scopeMetrics": [
{
"metrics": [
{
"description": "number of CPUs in the system",
"gauge": {
"dataPoints": [
{
"asDouble": NCPU,
"attributes": [
{
"agent": "linux-gnu"
"domainname": DOMAINID,
"groupid": 4209557,
"hostname": HOSTNAME,
"machineid": MACHINEID,
"semantics": "discrete",
"type": "u32",
"userid": GROUPID
}
],
"startTimeUnixNano": TIME
}
]
},
"name": "hinv.ncpu",
"unit": "none"
}
],
"scope": {
"name": "pcp.hinv",
"version": 1
}
}
]
}
]
}

---
{
"@pcp": {
Expand Down
11 changes: 2 additions & 9 deletions qa/1589
Original file line number Diff line number Diff line change
Expand Up @@ -51,16 +51,8 @@ _filter_pcp2json_http()
-e "s/\(\"value\"\): \""$cpus"\"/\1:NCPU/g" \
-e 's/\(\"@timestamp\"\): \(.*\)/\1:DATE TIME/' \
-e '/HTTP/d' \
-e "s/^\(Host: localhost\):$port/\1:PORT/g" \
-e 's/^\(Content-Length:\) [1-9][0-9]*/\1 SIZE/g' \
-e 's/^\(User-Agent: python-requests\).*/\1 VERSION/g' \
-e 's/^\(Date:\).*/\1 DATE/g' \
-e 's/\(\"context\":\) [0-9][0-9]*/\1 CTXID/g' \
-e '/^Accept-Encoding: /d' \
-e '/^Connection: keep-alive/d' \
-e '/ using stream socket$/d' \
| LC_COLLATE=POSIX sort
}
} >> $tmp.python2.out

# real QA test starts here
port=`_find_free_port`
Expand All @@ -73,6 +65,7 @@ pcp2json -s1 -ZUTC -u http://localhost:$port hinv.ncpu >$tmp.json.out 2>$tmp.jso

echo "pcp2json HTTP POST (sorted):"
_filter_pcp2json_http <$tmp.python.out
_check_http_filter <$tmp.python2.out

# terminate pythonserver.py now
pmsignal $pid >/dev/null 2>&1
Expand Down
17 changes: 4 additions & 13 deletions qa/1827
Original file line number Diff line number Diff line change
Expand Up @@ -43,19 +43,9 @@ _filter_pcp2openmetrics_http()
-e '/groupid=/s/groupid="*'$MYGID'"*/groupid="GROUPID"/' \
-e '/userid=/s/userid="*'$MYUID'"*/userid="USERID"/' \
-e '/hostname=/ s/hostname="*'$hostname'"*/hostname="HOST"/' \
-e 's/} [0-9][0-9]*$/} N (cpus)/' \
-e '/HTTP/d' \
-e "s/^\(Host: localhost\):$port/\1:PORT/g" \
-e 's/^\(Content-Length:\) [1-9][0-9]*/\1 SIZE/g' \
-e 's/^\(User-Agent: python-requests\).*/\1 VERSION/g' \
-e 's/^\(Date:\).*/\1 DATE/g' \
-e 's/\(\"context\":\) [0-9][0-9]*/\1 CTXID/g' \
-e '/^Accept-Encoding: /d' \
-e 's/\(\hostname=\): \""$hostname"\"/\1:HOST/g' \
-e '/^Connection: keep-alive/d' \
-e '/ using stream socket$/d' \
| LC_COLLATE=POSIX sort
}
-e '/HTTP/d' \
-e 's/} [0-9][0-9]*$/} N (cpus)/'
} >> $tmp.python2.out

# real QA test starts here
port=`_find_free_port`
Expand All @@ -68,6 +58,7 @@ pcp2openmetrics -s1 -u http://localhost:$port hinv.ncpu >$tmp.openmetrics.out 2>

echo "pcp2openmetrics HTTP POST (sorted):"
_filter_pcp2openmetrics_http <$tmp.python.out
_check_http_filter <$tmp.python2.out

# terminate pythonserver.py now
pmsignal $pid >/dev/null 2>&1
Expand Down
26 changes: 13 additions & 13 deletions qa/1827.out
Original file line number Diff line number Diff line change
@@ -1,23 +1,23 @@
QA output created by 1827
pcp2openmetrics invocation
pcp2openmetrics HTTP POST (sorted):
INFO:root:Starting httpd...

INFO:root:POST request,
Path: /
Headers:
Host: localhost:PORT
User-Agent: python-requests VERSION
Accept: */*
Content-Type: application/openmetrics-text
Content-Length: SIZE





# HELP hinv_ncpu number of CPUs in the system
Body:
# PCP5 hinv_ncpu 60.0.32 u32 PM_INDOM_NULL discrete
# HELP hinv_ncpu b'number of CPUs in the system'
# TYPE hinv_ncpu gauge
Accept: */*
Body:
Content-Length: SIZE
Content-Type: application/openmetrics-text
Headers:
Host: localhost:PORT
INFO:root:POST request,
INFO:root:Starting httpd...
Path: /
User-Agent: python-requests VERSION
hinv_ncpu{domainname="DOMAINID",groupid="GROUPID",hostname="HOST",machineid="MACHINEID",userid="USERID",agent="linux"} N (cpus)


68 changes: 68 additions & 0 deletions qa/1977
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
#!/bin/sh
# PCP QA Test No. 1827
# Exercise pcp2opentelemetry HTTP POST functionality.
#
# Copyright (c) 2024 Red Hat. All Rights Reserved.
#


seq=`basename $0`
echo "QA output created by $seq"

. ./common.python

which pcp2opentelemetry >/dev/null 2>&1 || _notrun "pcp2opentelemetry not installed"


_cleanup()
{
cd $here
$sudo rm -rf $tmp $tmp.*
}

status=0 # success is the default!
cpus=`pmprobe -v hinv.ncpu | awk '{print $3}'`
hostname=`hostname`
machineid=`_machine_id`
domainid=`_domain_name`
$sudo rm -rf $tmp $tmp.* $seq.full
trap "_cleanup; exit \$status" 0 1 2 3 15

_filter_pcp2opentelemetry_http()
{
tee -a $here/$seq.full \
| col -b \
| sed \
-e '/\"asDouble\":/ s/[0-9][0-9]*/NCPU/' \
-e 's/\"'$machineid'\"/MACHINEID/' \
-e 's/\"'$hostname'\"/HOSTNAME/' \
-e 's/\"'$domainid'\"/DOMAINID/' \
-e 's/\"agent\": .*/\"agent\": \"'$OSTYPE'\"/' \
-e 's/\"startTimeUnixNano\": .*/\"startTimeUnixNano\": TIME/' \
-e 's/\"userid\": .*/\"userid\": USERID/' \
-e 's/\"userid\": .*/\"userid\": GROUPID/' \
-e '/HTTP/d'
} >> $tmp.python2.out

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since this pattern is repeated in three or four tests now, let's make a _filter_python_http_reponse shell function in one of the qa/common.xxx files and share it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@natoscott should this go in a specific common.xxx file or make a new one?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lmchilton lets put it in common.python


# real QA test starts here
port=`_find_free_port`
$PCP_PYTHON_PROG $here/src/pythonserver.py $port >$tmp.python.out 2>&1 &
pid=$!
sleep 2 # let server start up


echo "pcp2opentelemetry invocation" | tee -a $here/$seq.full
pcp2opentelemetry -s1 -u http://localhost:$port/receive hinv.ncpu >$tmp.json.out 2>$tmp.openjson.err


echo "pcp2opentelemetry HTTP POST (sorted):"
_filter_pcp2opentelemetry_http <$tmp.python.out
_check_http_filter <$tmp.python2.out


# terminate pythonserver.py now
pmsignal $pid >/dev/null 2>&1

# success, all done
exit
Loading
Loading