CSV Store plugin - support for timestamps in output file names

[puneet336]

Hi,
I came across the LDMS tool today and I went through the tutorial present here- https://www.osti.gov/biblio/1899500
I had a query based on the slide content. Is it possible to have the store_csv plugin to put data in file labelled by timestamp ?. to be more specific -

Here are the 2 commands which specify the path and the file name. Seems that they both decide where the collected metrics would land up on file system

configname=store_csv  path=/home/user1/ldmscon2021/basic/exercises/ldms/data/  buffer=0
strgp_add name=meminfo-store_csv plugin=store_csv container=memory_metrics schema=meminfo

aforementioned configuration would put data from all configured producers in a single file at /home/user1/ldmscon2021/basic/exercises/ldms/data/memory_metrics/meminfo

here are the queries -
a) How could i get the directory name as - /home/user1/ldmscon2021/basic/exercises/ldms/data/%Y%m%d
so that i would be able to maintain the logs by date as system's time changes.
/home/user1/ldmscon2021/basic/exercises/ldms/data/20221223/meminfo
/home/user1/ldmscon2021/basic/exercises/ldms/data/20221224/meminfo

b) Can the similar approach be done for the file name ? i.e. schema=meminfo_%Y%m%d so that
/home/user1/ldmscon2021/basic/exercises/ldms/data/memory_metrics/meminfo_20221223
/home/user1/ldmscon2021/basic/exercises/ldms/data/memory_metrics/meminfo_20221224

c) Also, Is it possible to maintain separate logs for each hosts?. Example -
/home/user1/ldmscon2021/basic/exercises/ldms/data/memory_metrics/meminfo_node1
/home/user1/ldmscon2021/basic/exercises/ldms/data/memory_metrics/meminfo_node2

[baallan]

@puneet336 Thank you for your questions.

re a) You will want to examine the rollover options in the Plugin_store_csv man page installed with ldms. With rollover enabled and particularly the rename-on-close options you have a lot of flexibility in output renaming.

re b) I cannot think why you would want schema name (which in most cases is intended to map to a fixed downstream database schema) to change with the date. If you really want that, you'll need to stop ldmsd at midnight, regenerate the configuration file with the new date substituted in the configuration values that include date strings, and restart ldmsd. This could also be orchestrated via the python interface (ldmsd_controller) without restarting (just stop/reconfigure/start the plugins on all daemons).

re c) None of our storage plugins presently support writing a data file per producerName (host) in the metric sets. This would not be a difficult feature to create a custom store for, but it would scale poorly to systems with thousands of nodes. We have in the past had experimental plugins that wrote a data file per parallel job identifier (e.g. slurm_jobid), so I have evidence that the difficulty is not high.
It is also a simple matter to write a standalone tool to split our current csv output (after rollover) into files per-node; I would recommend C for performance in this case, or any existing scripting language libraries for csv support.