Skip to content

Commit

Permalink
derived metrics: extend some unary functions, improved error reporting
Browse files Browse the repository at this point in the history 
Most of this changes delta(x), rate(x) and instant(x).  Formerly x was
restricted to a metric name, but now x may be an arbitrary expression.

The restriction was simply guilt by association: the other unary
functions like defined(x), sum(x), min(x), count(x), ... all make
sense for a single metric (which may be set-valued), but are not
needed for the unary functions delta(x), rate(x) and instant(x).

The code change was not difficult (yacc is a wonderful invention).
The man page changes and QA were more challenging.

Specifically this allows: delta(sum(x)) which fixes Marko's RFE at
#1991.

Also there was some flakey error reporting where some errors at
bind-time were not being reported at all, unless -Dderive was in play.
  • Loading branch information
@kmcdonell
kmcdonell committed Jun 5, 2024
1 parent 131cc62 commit bab1ef8
Show file tree
Hide file tree
Showing 15 changed files with 817 additions and 127 deletions.
181 changes: 107 additions & 74 deletions man/man3/pmregisterderived.3
View file
Original file line number Diff line number Diff line change
Expand Up @@ -144,13 +144,13 @@ my.summary.disk.avgsz\c
The expression
.I expr
follows these syntactic rules:
.IP * 2n
.IP \(bu 2n
Terminal elements are either names of existing metrics or numeric constants.
Recursive definitions are not allowed, so only the names of regular
metrics (not other derived metrics) may be used. Numeric constants are
either integers constrained to the precision of 32-bit unsigned integers
or double precision floating point numbers.
.IP * 2n
.IP \(bu 2n
The usual binary arithmetic operators are supported, namely addition (``+''),
subtraction (``-''), multiplication (``*'') and division (``/'') with
the normal precedence rules where multiplication and division have
Expand All @@ -163,13 +163,13 @@ is evaluated as
a+(b*c)\c
.ft R
\&.
.IP * 2n
.IP \(bu 2n
Unary negation may be used, e.g.
.ft CR
-3*some.metric\c
.ft R
\&.
.IP * 2n
.IP \(bu 2n
C-style relational operators are supported, namely ``<'', ``<='', ``=='',
\&``>='', ``>'' and ``!=''.
Relational expressions return a value as a 32-bit unsigned number being
Expand All @@ -195,7 +195,7 @@ is evaluated as
(a>b)!=c\c
.ft R
\&.
.IP * 2n
.IP \(bu 2n
C-style boolean operators are supported, namely and (``&&'') and or (``||'').
Boolean expressions return a value as a 32-bit unsigned number being
0 for false and 1 for true.
Expand All @@ -220,7 +220,7 @@ is evaluated as
(((a>=b)||(b>c))&&(d!=e))||(f>g)\c
.ft R
\&.
.IP * 2n
.IP \(bu 2n
Additionally, the ``!'' operator may be used to negate a boolean or
relational expression, returning a value as a 32-bit unsigned number being
0 for false and 1 for true.
Expand All @@ -245,7 +245,7 @@ is evaluated as
!(a<(b+c))\c
.ft R
\&.
.IP * 2n
.IP \(bu 2n
C-style ternary conditional expressions are supported. In general terms
the expression
.ft CR
Expand Down Expand Up @@ -289,11 +289,13 @@ evaluation of the ``guard'', then the result is a set of values
(all the same) with instance enumeration being taken from the other
operand expression. For example in the expression:
.ft CR
foo ? scalar : set\c
check ? foo : bar\c
.ft R
\&, if \f(CRfoo\fP is true, then the result is a set of values
(all having the same value, \f(CRscalar\fP) over
the instance domain of \f(CRset\fP.
\&, if \f(CRfoo\fP is an expression with a singular value and
\f(CRbar\fP is a set-valued expression, then if \f(CRcheck\fP is true,
then the result is a set of values
(all having the same value, \f(CRfoo\fP) over
the instance domain of \f(CRbar\fP.
.IP (h) 4n
As a special case, either (but not both) of the
expressions
Expand Down Expand Up @@ -350,7 +352,7 @@ fumble = defined(new.metric) ? new.metric : old.metric
which is valid when \f(CRnew.metric\fP is defined
.B and
when \f(CRnew.metric\fP is
.I not
.B not
defined, although this does mean rules (b) and (c) are relaxed
in this case, which further means \f(CRnovalue()\fP may have no
peer operand to provide metadata.
Expand All @@ -369,7 +371,7 @@ bar = !defined(a) || !defined(b) ? novalue() : a + b
.RS -4n
.PD
.RS -2n
.IP * 2n
.IP \(bu 2n
Selection of a single instance can be specified by the
construct ``[\fIinstance_name\fR]'' which may be appended to
a metric name or a parenthesized expression.
Expand Down Expand Up @@ -400,7 +402,7 @@ any spurious white space.
A backslash may be used as an escape prefix
in the (unlikely) event that the external instance name contains a ``]''.
.RE
.IP * 2n
.IP \(bu 2n
Numeric constants can also be specified using the
.ft CR
mkconst()
Expand Down Expand Up @@ -466,12 +468,12 @@ type and/or semantics and/or units for operands, e.g.
idle = mem.util.free > mkconst(10485760, units=Kbyte)
.br
avg_io_size = delta(disk.dev.total) == 0 ? \e
-mkconst(1.0, semantics=instant, units="kbyte / count") : \e
mkconst(1.0, semantics=instant, units="kbyte / count") : \e
delta(disk.dev.total_bytes) / delta(disk.dev.total)
.ft R
.in
.RE
.IP * 2n
.IP \(bu 2n
Expressions may be rescaled using the
.ft CR
rescale
Expand All @@ -492,20 +494,17 @@ rescale(network.interface.total.bytes, "Mbytes/hour")
The expression and the desired units must both have the same dimension,
e.g Space=1, Time=\-1 and Count=0 in the example above.
.RE
.IP * 2n
.IP \(bu 2n
The following unary functions operate on a single performance metric
and return one or more values.
For all functions (except
.ft CR
count()\c
.ft R
,
.ft CR
defined()
.ft R
and
.ft CR
instant()\c
defined()\c
.ft R
.ft R
), the type of the operand metric must be arithmetic
(integer of various sizes and signedness, float or
Expand All @@ -523,10 +522,7 @@ _
count(x) T{
A singular instance being the count of the number of instances for the metric x.
As a special case, if fetching the metric x returns an error, then
.ft CR
count(x)
.ft R
will be 0.
\f(CRcount(x)\fP will be 0.
T}
_
defined(x) T{
Expand All @@ -545,35 +541,87 @@ context has been established will not change the value
of this function in the expression evaluation.
T}
_
delta(x) T{
Returns the difference in values for the metric x between
max(x) T{
A singular instance being the maximum value across all instances for the metric x.
T}
_
min(x) T{
A singular instance being the minimum value across all instances for the metric x.
T}
_
sum(x) T{
A singular instance being the sum of the values across all instances for the metric x.
T}
.TE
.IP \(bu 2n
The following unary function returns the instantaneous value of an
expression, not the rate-converted value that is the default
for expressions with the semantics of PM_SEM_COUNTER.
.TS
box,center;
cf(R) | cf(R)
lf(CR) | lf(R)x.
Function Value
_
instant(expr) T{
Returns the current value of the expression, even it has
the semantics of a counter, i.e. PM_SEM_COUNTER.
The semantics of the derived metric are based on the semantics of the
expression \f(CRexpr\fR; if \f(CRexpr\fR has semantics PM_SEM_COUNTER, the semantics of
\f(CRinstant(expr)\fR is PM_SEM_INSTANT, otherwise the semantics of the derived metric
is the same as the semantics of \f(CRexpr\fR.
T}
.TE
.IP \(bu 2n
The following unary functions return values computed from
the value of an expression on consecutive samples, or
.BR pmFetch (3)
calls.
The expression (\f(CRexpr\fR below) may involve one or more metrics
but must have an arithmetic value
(integer of various sizes and signedness, float or double) for all
instances.
.RS 2n
.PP
If \f(CRexpr\fP is a set-valued expression then only those instances
that appear in
.B both
samples will appear in the result.
.TS
box,center;
cf(R) | cf(R)
lf(CR) | lf(R)x.
Function Value
_
delta(expr) T{
Returns the difference in values for the expression between
one call to
.BR pmFetch (3)
and the next. There is one value in the result
for each instance that appears in both the current and the previous
sample.
If the metric x is unsigned, then the type of the result is
If the expression is unsigned, then the type of the result is
converted to ensure as much precision as possible can be retained,
so if the metric x has type PM_TYPE_U32 then the result is of type PM_TYPE_64, else
if the metric x has type PM_TYPE_U64 then the result is of type PM_TYPE_DOUBLE.
so if the expression has type PM_TYPE_U32 then the result is of type PM_TYPE_64, else
if the expression has type PM_TYPE_U64 then the result is of type PM_TYPE_DOUBLE.
Otherwise the type of the result is the same as the type of the
metric x.
expression.
T}
_
rate(x) T{
Returns the difference in values for the metric x between
rate(expr) T{
Returns the difference in values for the expression between
one call to
.BR pmFetch (3)
and the next divided by the elapsed time between the calls to
.BR pmFetch (3).
The semantics of the derived metric are based on the semantics of the
metric x with the dimension in the
expression with the dimension in the
.B time
domain decreased by one and scaling if required in the time utilization case
where the operand is in units of time, and the derived metric is unitless.
There is one value in the result
for each instance that appears in both the current and the previous
sample, except in the case where the metric x has
sample, except in the case where the expression has
the semantics of a counter, i.e. PM_SEM_COUNTER, and
current value of an instance is smaller than the previous value
of the same instance then no value is
Expand All @@ -586,29 +634,9 @@ such as
and
.BR pmchart (1).
T}
_
instant(x) T{
Returns the current value of the metric x, even it has
the semantics of a counter, i.e. PM_SEM_COUNTER.
The semantics of the derived metric are based on the semantics of the
metric x; if x has semantics PM_SEM_COUNTER, the semantics of
instant(x) is PM_SEM_INSTANT, otherwise the semantics of the derived metric
is the same as the semantics of the metric x.
T}
_
max(x) T{
A singular instance being the maximum value across all instances for the metric x.
T}
_
min(x) T{
A singular instance being the minimum value across all instances for the metric x.
T}
_
sum(x) T{
A singular instance being the sum of the values across all instances for the metric x.
T}
.TE
.IP * 2n
.RE
.IP \(bu 2n
The \f(CRmatchinst\fR function may be used to select a subset of the instances from
an instance domain for a metric or expression.
The function takes two arguments:
Expand All @@ -620,7 +648,7 @@ by a regular expression delimited by ``/'' characters.
The regular expression follows the
POSIX Extended Regular Expression syntax as described in
.BR regex (3).
Backslashes may be used as escape prefixes, but double backslash is required to
A single backslash may be used to escape the regular expression delimiter ``/'', but double backslashes are required to
escape any regular expression special characters, e.g. for the (extremely unlikely)
case of wanting to match instance names like ``some*text/other[text]'' a
regular expression of the form \f(CR/some\e\e*text\e/other\e\e[text]/\fR
Expand All @@ -646,7 +674,7 @@ matchinst(!/^(lo)|(vbir)/, network.interface.in.bytes)
.in
.br
.RE
.IP * 2n
.IP \(bu 2n
The \f(CRscalar\fR function may be used convert a metric or expression
defined over an instance domain into a scalar value that can be used
in other expressions.
Expand All @@ -672,15 +700,15 @@ or expressions that are expected to contain zero or one instances, e.g.
the construct ``[\fIinstance_name\fR]'' or the \f(CRmatchinst\fR function
with a pattern that matches at most one instance.
.RE
.IP * 2n
.IP \(bu 2n
Parenthesis may be used for explicit grouping.
.IP * 2n
.IP \(bu 2n
A line ending with ``\e'' is treated as ``to be continued'' and the
following line is appended after stripping the ``\e'' and the
embedded newline.
.IP * 2n
.IP \(bu 2n
Lines beginning with ``#'' are treated as comments and ignored.
.IP * 2n
.IP \(bu 2n
White space is ignored.
.SH "SEMANTIC CHECKS AND RULES"
There are a number of conversions required to determine the
Expand Down Expand Up @@ -715,25 +743,25 @@ a result that is not a counter, either or both operands of a relational expressi
may be counters.
.PP
The mapping of the pmUnits of the metadata uses the following rules:
.IP * 2n
.IP \(bu 2n
If both operands have a dimension of Count and the scales are not
the same, use the larger scale and convert the values of the operand
with the smaller scale.
.IP * 2n
.IP \(bu 2n
If both operands have a dimension of Time and the scales are not
the same, use the larger scale and convert the values of the operand
with the smaller scale.
.IP * 2n
.IP \(bu 2n
If both operands have a dimension of Space and the scales are not
the same, use the larger scale and convert the values of the operand
with the smaller scale.
.IP * 2n
.IP \(bu 2n
For addition and subtraction all dimensions for each of the operands
and result are identical.
.IP * 2n
.IP \(bu 2n
For multiplication, the dimensions of the result are the sum of the
dimensions of the operands.
.IP * 2n
.IP \(bu 2n
For division, the dimensions of the result are the difference of the
dimensions of the operands.
.PP
Expand Down Expand Up @@ -894,7 +922,7 @@ Metrics with counter semantics may be added or subtracted, but <op>
.TP
Semantic error: derived metric <name>: operand <bname>: Unknown metric for ternary expression
When a new context was established, the metric <bname> was not in the
PMNS of the new context and <bname> is a required operand in the defintion of
PMNS of the new context and <bname> is a required operand in the definition of
the derived metric <name>.
.SH "EXPRESSION EVALUATION"
For the binary arithmetic operators,
Expand Down Expand Up @@ -925,14 +953,14 @@ T{
otherwise (both are PM_TYPE_32)
T} any PM_TYPE_32
.TE
.SH PMIDS AND MASKING
.SH PMIDs AND MASKING
Within PCP each metric is assigned a
unique Performance Metric Identifier (PMID) and
internally a PMID is constructed from 3 fields: the domain number
(of the associated Performance Metrics Domain Agent, or PMDA),
the cluster number and the item number.
Derived metrics use the reserved domain number 511 and
special PMIDs as decribed in the following table, where
special PMIDs as described in the following table, where
the PMID is shown as domain.cluster.item:
.TS
box,center;
Expand Down Expand Up @@ -1100,7 +1128,9 @@ Error: pmRegisterDerivedMetric("my.disk.rates", ...) syntax error
The position indicator line may be followed by an additional
diagnostic line describing the nature of the error, when available.
.PP
In the case of an error, the caller is responsible for calling
In the case of an error, the
.B pmRegisterDerivedMetric
caller is responsible for calling
.BR free (3)
to release the space allocated for
.IR errmsg .
Expand All @@ -1127,3 +1157,6 @@ and
.\" +ok+ fw {from fw.bytes}
.\" +ok+ sensical {from non-sensical}
.\" +ok+ abc def {both from abc.def}
.\" +ok+ bname {from operand <bname>}
.\" +ok+ dname {from operand <dname>}
.\" +ok+ Dderive {from -Dderive}
Loading

0 comments on commit bab1ef8

Please sign in to comment.