ldmscon2023 decomposition configurtion for mapping json streams to db

[baallan]

An extension of the store decomposition syntax to support converting stream json messages to DB (maximally similar to set decomp specification) will be very helpful but does need some extensions to handle json-related issues.
We need this to eliminate the need for many custom stream stores we have today.

[baallan]

The basic set-based decomp from the 2023 ldmscon tutorial is in the first box below.

• The first extension for static decomp needed is to be able to name json parts when mapping messages to rows. A standards-based approach to this is to extend (for the json decomp) the rows.cols.src fields below to accept a IETF draft JSONpath expression as the value for the "src" field. https://datatracker.ietf.org/doc/draft-ietf-jsonpath-base/
• The second extension needed is to be able to specify coercion of json values to LDMS data types (json does NOT support the majority of LDMS data types) and the fill value to use if a coercion cannot be performed properly. To this end we need additional keys:
  • rows.cols.json_type: json value must match this and json_type may be a single value (e.g. "STRING") or a list ["STRING, NUMBER"] or a "*"; the rows.cols.type field is the ldms target type for the coercion. A notable case of this is to allow "*" as the json_type to imply conversion of the value, whatever it is, to ldms type char_array; this allows variable app-defined json blobs to be stored for the originating application.
  • rows.cols.fill_err: an ldms target value to use if type coercion fails; we cannot, in general, count on incoming messages to be well-typed because they may come from evolving user applications (users must not be able to cause us errors, and they should get the data they deserve if they change things without telling those handling their storage). A json element mis-typed is not the same as an element missing altogether; fill_err allows admins and users to easily distinguish the two.
• The third extension needed is to be able to specify flattening of an object value to a set of columns whose names have a common prefix (possibly empty prefix) and jsonpath notation used to construct the column names of children. This can be specified by allowing a json object value for the rows.cols.type key which specifies flattening is wanted and what prefix to use. See example below.

# current static set decomp syntax (excluding the defaulting of types proposal of Morrone)
{ "type": "static",
  "rows" : [
         { "schema": <row_name>,
           "cols": [
                { "src": <metric_name>,
                  "dst" : <column name. Special Characters (_), (.), (:) are valid>,
                  "type" : <value type must match the type in the set schema>,
                 "array_len" : <len if type is array>,
                 "rec_member" : <record member name if 'src' is a list of record>,
                 "fill" : <value if 'src' is missing from schema>
         },
         ...
 ],
 "indices": [
      { "name":<index name>, "cols": [ <destination column name> ] },
      { "name":<index name>, "cols": [ <destination column name>, <destination column name>,...] },
       ...
  ]
}

# proposed stream decomp syntax (with specific coercion example)
{ "type": "static",
  "rows" : [
         { "schema": <row_name>,
           "cols": [
                { "src": <metric_name>,
                  "dst" : <column name. Special Characters (_), (.), (:) are valid>,
                  "json_type": [ "integer","string" ]
                  "type" : "u64",
                 "array_len" : <len if type is array>,
                 "rec_member" : <record member name if 'src' is a list of record>,
                 "fill" : <value if 'src' is missing from schema>,
                 "fill_err": <value if 'src'.value is uncoercible from json_type to type without losing information>
         },
         { "schema": <row_name>,
           "cols": [
                { "src": <metric_name>,
                  "dst" : <column name. Special Characters (_), (.), (:) are valid>,
                  "json_type": [ "integer","string" ]
                  "type" : {
                       "flatten" : {
                          "prefix" : <string which may be empty | list where list contains a negative integer defining how many levels of nested parent element names to prepend when forming column names>
                      }
                  },
                 "array_len" : <len if type is array>,
                 "rec_member" : <record member name if 'src' is a list of record>,
                 "fill" : <value if 'src' is missing from schema>,
                 "fill_err": <value if 'src'.value is uncoercible from json_type to type without losing information>
         },
         ...
 ],
 "indices": [
      { "name":<index name>, "cols": [ <destination column name> ] },
      { "name":<index name>, "cols": [ <destination column name>, <destination column name>,...] },
       ...
  ]
}

[morrone]

I am concerned that this might be the wrong way to go about this. The "static" decomposition engine is about converting ldms metric sets to rows. It sounds like instead of converting an ldms metric set, this whole topic is about converting json into rows. Maybe a different configuration altogether should be made for that use case rather than overloading the "static" configuration for very different use cases.

[baallan]

I don't intend that 'static' be overloaded; perfectly happy to use a new keyword for the 'json rowification' case. I (for discussion purposes) started from the "static decomp" example, though, since it (a) is the prior art within the system (b) handles most of the concerns already.

One of the additional concerns/items to address is the rowification of non-json strings and binary blobs from the user.

• For binary blobs (which for the moment have been omitted from the stream specifiable message types), we really need access from the transport layer to "always available" metadata as part of the stream subscription callback:
  • time of message receipt at origin, uid/gid of sender, identity of the origin (host, port, stream name).
• For strings, we really need to be able to extract substrings to use as keys/values (beyond those from the transport layer)
  • probably regex based so that i can turn string message "app1: a=1 b=2 job=31 terminated at step 7 with error 17" into a row as if from json-equivalent:

{ "program":"app1", exit:17, job:"31", "a":1, "b":2, "raw":"app1: a=1 b=2 job=31 terminated at step 7 with error 17"}

[tom95858]

@baallan, this replicates an existing architecture that maps storage policy to schema and metric sets to rows. All of this would then be shoe-horned into a single plugin that converts JSON directly to rows and calls another plugin's commit() interface.

The benefits of converting JSON to Metric Sets and then using the existing architecture are that:

• It leverages all of the existing decomposition infrastructure
• It leverages the existing storage policy configuration infrastructure
• It helps significantly with debug because metric sets can be viewed with existing tools (i.e. ldms_ls)
• It allows us to get rid of redundant samplers and stores: specifically, slurm, slurm2, darshan, kokkos_appmon, and others. Replicating a existing infrastructure and creating a new decomposition syntax to get rid of these same samplers seems unwise.

At this point I struggle with the merits of why converting JSON "directly" to rows is preferred.

[baallan]

@tom95858 this doesn't have to replicate the "set-to-row-decomp-for-stores" portion of things if you prefer to implement it that way (sounds to me, though, that @morrone may have misgivings about trying to force them together that way, as do I).

Even if you go through sets, you still need to provide some configuration interface that lets us specify extraction of data from json to put into the set with the features identified above. Reiterating:

• naming source fields with JSONpath identification
• mapping to a ldms field type from one or more json data types, with a fill value for coercions that fail and the ability to collapse value (including object values) to a string.
• applying the name flattening pattern to the members of an object when automatically generating column names.

I know we need all these things because they are things we have done (or assumptions we have used) for json-sosdb plugins and for scripted conversion of raw stream (blob) data files to CSV for ingest with the csv-to-sosdb parser.

Separating the configuration of these json-conversion to set member step from the set-conversion to row step is a valid implementation choice (in part for the reasons you suggest). However, I expect it leads to the admin being forced to write nearly identical json/yaml conversion configuration statements for the stream:set converter and for the set:row converter.

[tom95858]

@ballan, I think at this point it would be best if instead of talking generalities we discussed an actual proposal. What does the input JSON look like, what does the digest JSON look like and what do the output rows look like?

JSON supports an arbitrarily deep level of nested dictionaries, lists of dictionaries, dictionaries of lists, etc...

Please make a proposal so we can discuss actual use cases.

[baallan]

@tom95858 I've been working through a proposal on (appdata,digest,output) using and example from empire and your assumption that the result will fit in an ldms set. I've run into a severe constraint due to the set conversion assumption: ldms sets do not support unspecified length strings.
If we had some sort of "local-only" set instance that allowed arbitrarily sized strings (char_array_unsized_local), I'd become unstuck. local-only set instances wouldn't need fixed size strings (or lists) for the rdma transport.

The driving need is to be able to roll up a named json subtree into a single string field in the database (something that both csv and sosdb would cope with, but char_array does not).

imo a better solution would be to separate set decomp processing from stream decomp processing, but you seem to be hard over on using a set intermediate. So what do you propose as the method of routing dynamically bounded json blobs through a set?

[tom95858]

I apologize but I have no idea what you are talking about. What "contraint", what use case, what "set conversion assumption". I have no idea what your are talking about. Could you please be specific about what feature, function, or attribute that you need and for what purpose?

[baallan]

@tom95858 all you puzzlement will be answered if you review the discussion of which the message is a part. there's a little github link at the bottom of emails that will take you there.

Particularly your comment of 3 weeks ago:

| At this point I struggle with the merits of why converting JSON "directly" to rows is preferred.

In the discussion chain (and at ldmscon) you strongly stated (apparently with some code in hand or at least mind at ldmscon) that the solution is to convert json to sets and then use set decomp to rows as it exists.

The case where what the app/publisher wants done is "convert a named subtree of the sent json message back to a string and store it in the database" fits much better into the "JSON "directly" to rows" scenario than into Json-to-ldms-set-to-row-decomp. Sets have fixed size strings but json users have no reason to expect an arbitrary string size limit.