diff --git a/CLAUDE.md b/CLAUDE.md index 89276fcd..b4b397eb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,13 +5,13 @@ Defines a unified data model for storing and querying performance test data in O ## Languages - **JavaScript/Node.js**: Query library and server (`queries/cdmq/`) -- **Bash**: Template management scripts (`templates/`) +- **Bash**: `templates/delete.sh`, a standalone index-cleanup script ## Key Directories | Path | Purpose | |------|---------| | `queries/cdmq/` | Node.js query library and HTTP server | -| `templates/` | OpenSearch index templates and management scripts | +| `templates/` | Just `delete.sh` — a destructive cleanup script invoked by crucible's `reinit_opensearch()`. Index mappings themselves are defined in `queries/cdmq/cdm.js` (see Templates section below) | | `workflows/` | Documentation (result-calculation methodology) | ## Key Files in `queries/cdmq/` @@ -38,10 +38,9 @@ Supporting document types: `param`, `tag`, `config_*` - v10dev's key addition is `default-aggregation` — a per-metric field on `metric_desc` (`sum`/`avg`/`max`/`min`) telling query-time aggregation how to combine values across breakout dimensions, instead of always duration-weighted summing ## Templates (`templates/`) -- `.base` files define index mappings for each document type -- `build.sh` / `Makefile` generate actual template commands -- `init.sh` initializes the OpenSearch indices -- All indices are `"dynamic": "strict"` — every `metric_desc.names` breakout dimension a tool/benchmark might emit must be pre-registered as an explicit `keyword`/`double` field in `metric_desc.base`, or indexing rejects it. `dynamic_templates` do NOT provide an exception to this (verified empirically) — a field matching a `dynamic_templates` glob is rejected exactly like any other unregistered field under `"dynamic": "strict"`. A dimension family with an unbounded/platform-dependent set of names (e.g. one field per CPU cache level) still needs each concrete name registered explicitly (e.g. `shared-l1-domain` .. `shared-l4-domain`) rather than a wildcard shortcut. +- Index mappings for every document type are defined in `queries/cdmq/cdm.js`'s `indexDefs` object (`v8dev` hand-written, `v9dev`/`v10dev` built forward via `deepClone`). This is the only live schema — `add-run.js`'s `checkCreateIndex()`/`updateIndexMappings()` use it to create/update OpenSearch indices, and also use it as a client-side validation gate, rejecting any document field not present in it before the document is ever sent to OpenSearch. +- `templates/delete.sh` is the sole surviving file in this directory — a standalone, destructive script (deletes every index on `localhost:9200`) invoked by crucible's `reinit_opensearch()`. It has no dependency on `cdm.js` or any schema definition. +- All indices are `"dynamic": "strict"` — every `metric_desc.names` breakout dimension a tool/benchmark might emit must be pre-registered as an explicit `keyword`/`double` field in `cdm.js`'s `indexDefs`, or indexing rejects it. `dynamic_templates` do NOT provide an exception to this (verified empirically) — a field matching a `dynamic_templates` glob is rejected exactly like any other unregistered field under `"dynamic": "strict"`. A dimension family with an unbounded/platform-dependent set of names (e.g. one field per CPU cache level) still needs each concrete name registered explicitly (e.g. `shared-l1-domain` .. `shared-l4-domain`) rather than a wildcard shortcut. ## Code Style - JavaScript: Prettier formatting enforced (2-space indent, checked in CI via `cdm-ci.yaml`) diff --git a/README.md b/README.md index b69329e3..4dcc995a 100644 --- a/README.md +++ b/README.md @@ -7,14 +7,15 @@ Our Data is the problem. When it comes to performance benchmarking, or just mon We can define a common way to store information about our environment, our performance tests (if any), and metrics and events we collect. Having a common way to process this information allows us to query, summarize, and visualize performance data across many situations, from comparing compiler performance to identifying bottlenecks in large cloud deployments. ### What This Project Includes We provide enough information so that anyone can start storing and querying this data in a common way: -* OpenSearch index templates +* OpenSearch index mapping definitions (`queries/cdmq/cdm.js`) * A Node.js query library and HTTP query server (`queries/cdmq/`) ### What This Project Does Not Include * data conversion scripts: conversion of data is expected to be in the other projects, for example: [uperf-post-process](https://github.com/perftool-incubator/bench-uperf/blob/master/uperf-post-process) * data indexing scripts: indexing of data is also expected to be in other projects, for example: [rickshaw-index](https://github.com/perftool-incubator/rickshaw/blob/master/rickshaw-index) ## Directory/Layout [./templates](./templates) -OpenSearch index templates and management scripts (init, build, delete). +`delete.sh`, a destructive OpenSearch index cleanup script. Index mapping +definitions themselves live in `queries/cdmq/cdm.js`, not here. [./queries](./queries) The `cdmq` query implementation, built on Node.js. Includes a core query library (`cdm.js`), command-line query scripts, and an HTTP server (`server.js`) that exposes CDM queries as REST endpoints. diff --git a/queries/cdmq/README.md b/queries/cdmq/README.md index cda2f8db..d434fa11 100644 --- a/queries/cdmq/README.md +++ b/queries/cdmq/README.md @@ -5,12 +5,12 @@ [Node.js](https://nodejs.org) and npm must be installed. Dependencies are installed automatically when you run any of the shell wrapper scripts (e.g., `get-result-summary.sh`). You can also install them manually: ```bash -./setup.sh +npm install ``` ## Introduction -The contents of this directory contain a collection of scripts in Javascript intended to be executed with [node.js](https://nodejs.org). These scripts get data from an OpenSearch instance. The data must be in Common Data Format. which is documented in this project under [templates](../templates). The scripts here are meant to help inspect, compare, and export data from benchmarks and performance & resource-utilization tools, in order to report and investigate performance. +The contents of this directory contain a collection of scripts in Javascript intended to be executed with [node.js](https://nodejs.org). These scripts get data from an OpenSearch instance. The data must be in Common Data Format, whose index mapping definitions are documented in [cdm.js](./cdm.js)'s `indexDefs` object. The scripts here are meant to help inspect, compare, and export data from benchmarks and performance & resource-utilization tools, in order to report and investigate performance. In order to generate this data, you must run a benchmark via automation framework which uses the Common Data Format and index that data into OpenSearch. One of those automation frameworks is the [crucible](https://github.com/perftool-incubator/crucible) project. A subproject of crucible, [crucible-examples](https://github.com/perftool-incubator/crucible-examples), includes scenarios to run some of these benchmarks. diff --git a/queries/cdmq/create-index.js b/queries/cdmq/create-index.js deleted file mode 100755 index e1df1ca2..00000000 --- a/queries/cdmq/create-index.js +++ /dev/null @@ -1,58 +0,0 @@ -var cdm = require('./cdm'); -var path = require('path'); -var program = require('commander'); -var instances = []; // opensearch instances - -function save_host(host) { - var host_info = { host: host, header: { 'Content-Type': 'application/json' } }; - instances.push(host_info); -} - -function save_userpass(userpass) { - if (instances.length == 0) { - console.log('You must specify a --url before a --userpass'); - process.exit(1); - } - instances[instances.length - 1]['header'] = { - 'Content-Type': 'application/json', - Authorization: 'Basic ' + btoa(userpass) - }; -} - -function save_ver(ver) { - if (instances.length == 0) { - console.log('You must specify a --host before a --ver'); - process.exit(1); - } - if (cdm.isValidCdmVersion(ver)) { - instances[instances.length - 1]['ver'] = ver; - } else { - console.log('The version must be one of: ' + cdm.supportedCdmVersions.join(', ') + ', not: ' + ver); - process.exit(1); - } -} - -async function main() { - program - .version('0.1.0') - .option('--host ', 'The host and optional port of the OpenSearch instance', save_host) - .option('--userpass ', 'The user and password for the most recent --host', save_userpass) - .option(cdm.cdmVersionOptionDesc(), 'The Common Data Model version to use for the most recent --host', save_ver) - .option('--index ', 'An index name matching cdm naming format, for example, cdm-v9dev-run@202505>') - .parse(process.argv); - - // If the user does not specify any hosts, assume localhost:9200 is used - if (instances.length == 0) { - save_host('localhost:9200'); - } - getInstancesInfo(instances); - console.log('index: ' + program.index); - if (program.index) { - cdm.checkCreateIndex(instances[instances.length - 1], program.index); - } else { - console.log('--index is required'); - } - console.log('create-index is complete'); -} - -main(); diff --git a/queries/cdmq/get-instances-info.js b/queries/cdmq/get-instances-info.js deleted file mode 100755 index 0d8dc49f..00000000 --- a/queries/cdmq/get-instances-info.js +++ /dev/null @@ -1,51 +0,0 @@ -var cdm = require('./cdm'); -var path = require('path'); -var program = require('commander'); -var instances = []; // opensearch instances - -function save_host(host) { - var host_info = { host: host, header: { 'Content-Type': 'application/json' } }; - instances.push(host_info); -} - -function save_userpass(userpass) { - if (instances.length == 0) { - console.log('You must specify a --url before a --userpass'); - process.exit(1); - } - instances[instances.length - 1]['header'] = { - 'Content-Type': 'application/json', - Authorization: 'Basic ' + btoa(userpass) - }; -} - -function save_ver(ver) { - if (instances.length == 0) { - console.log('You must specify a --host before a --ver'); - process.exit(1); - } - if (cdm.isValidCdmVersion(ver)) { - instances[instances.length - 1]['ver'] = ver; - } else { - console.log('The version must be one of: ' + cdm.supportedCdmVersions.join(', ') + ', not: ' + ver); - process.exit(1); - } -} - -async function main() { - program - .version('0.1.0') - .option('--host ', 'The host and optional port of the OpenSearch instance', save_host) - .option('--userpass ', 'The user and password for the most recent --host', save_userpass) - .option(cdm.cdmVersionOptionDesc(), 'The Common Data Model version to use for the most recent --host', save_ver) - .parse(process.argv); - - // If the user does not specify any hosts, assume localhost:9200 is used - if (instances.length == 0) { - save_host('localhost:9200'); - } - getInstancesInfo(instances); - console.log(JSON.stringify(instances, null, 2)); -} - -main(); diff --git a/queries/cdmq/setup.sh b/queries/cdmq/setup.sh deleted file mode 100755 index c6550c65..00000000 --- a/queries/cdmq/setup.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/bin/bash -# Setup script for cdmq - install Node.js dependencies -set -e -script_dir=$(cd "$(dirname "$0")" && pwd) -cd "$script_dir" - -if ! command -v npm >/dev/null 2>&1; then - echo "Error: npm is not installed. Please install Node.js and npm." >&2 - exit 1 -fi - -echo "Node version: $(node --version)" -echo "npm version: $(npm --version)" -echo "Installing dependencies..." -npm install --no-fund --no-audit --loglevel verbose -echo "Setup complete." diff --git a/templates/Makefile b/templates/Makefile deleted file mode 100644 index 281038ad..00000000 --- a/templates/Makefile +++ /dev/null @@ -1,29 +0,0 @@ - -templates : run.json tag.json iteration.json param.json sample.json period.json metric_desc.json metric_data.json - -run.json : run.base - ./build.sh `cat ../VERSION` run - -tag.json : run.base tag.base - ./build.sh `cat ../VERSION` tag run - -iteration.json : run.base iteration.base - ./build.sh `cat ../VERSION` iteration run - -param.json : run.base iteration.base param.base - ./build.sh `cat ../VERSION` param iteration run - -sample.json : run.base iteration.base sample.base - ./build.sh `cat ../VERSION` sample iteration run - -period.json : run.base iteration.base sample.base period.base - ./build.sh `cat ../VERSION` period sample iteration run - -metric_desc.json : run.base iteration.base sample.base period.base metric_desc.base - ./build.sh `cat ../VERSION` metric_desc period sample iteration run - -metric_data.json : metric_data.base - ./build.sh `cat ../VERSION` metric_data run_micro - -clean : - /bin/rm -f *.json diff --git a/templates/README.md b/templates/README.md index 1b310966..b76d432e 100644 --- a/templates/README.md +++ b/templates/README.md @@ -1,121 +1,22 @@ ## Introduction -The files in this directory are for taking a newly created instance of Elastcisearch and creating the indices used for CommonDataModel - -## Directory Contents -In this directory are "base" json files, a Makefile to generate json files, and some scripts that generate commands to configure ES. - -The Makefile uses build.sh to combine one of more base files with a json header to produce an index template for ES. There is an index template for each document type used in CommonDataModel. After running "make", these files will be generated: - - iteration.json - metric_data.json - metric_desc.json - period.json - run.json - sample.json - -There are also thress scripts which generate curl commands to help you create and cleanup the elasticsearch instance: - - create-cleanup-cmds.sh - create-indices-cmds.sh - create-template-cmds.sh - -## Creating the CommonDataModel Indices -These instructions assume you already have an elastixsearch instance installed and reachable via localhost. Adjust the URL accrodingly for a instance at a different location. - -First run 'make' - - $ make - ./build.sh `cat ../VERSION` run - ./build.sh `cat ../VERSION` iteration run - ./build.sh `cat ../VERSION` sample iteration run - ./build.sh `cat ../VERSION` period sample iteration run - ./build.sh `cat ../VERSION` metric_desc period sample iteration run - ./build.sh `cat ../VERSION` metric_data - ls -l *.json - -rw-rw-r--. 1 atheurer atheurer 1570 Jun 29 13:18 iteration.json - -rw-rw-r--. 1 atheurer atheurer 808 Jun 29 13:18 metric_data.json - -rw-rw-r--. 1 atheurer atheurer 4212 Jun 29 13:18 metric_desc.json - -rw-rw-r--. 1 atheurer atheurer 2030 Jun 29 13:18 period.json - -rw-rw-r--. 1 atheurer atheurer 1256 Jun 29 13:18 run.json - -rw-rw-r--. 1 atheurer atheurer 1727 Jun 29 13:18 sample.json - -Now run create-template-cmds.sh - - $ ./create-template-cmds.sh localhost:9200 - curl -X PUT localhost:9200/_template/cdmv5dev-iteration -H 'Content-Type: application/json' -d@./iteration.json - curl -X PUT localhost:9200/_template/cdmv5dev-metric_data -H 'Content-Type: application/json' -d@./metric_data.json - curl -X PUT localhost:9200/_template/cdmv5dev-metric_desc -H 'Content-Type: application/json' -d@./metric_desc.json - curl -X PUT localhost:9200/_template/cdmv5dev-period -H 'Content-Type: application/json' -d@./period.json - curl -X PUT localhost:9200/_template/cdmv5dev-run -H 'Content-Type: application/json' -d@./run.json - curl -X PUT localhost:9200/_template/cdmv5dev-sample -H 'Content-Type: application/json' -d@./sample.json - -You will then need to copy/paste those commands to your prompt to run them: - - $ curl -X PUT localhost:9200/_template/cdmv5dev-iteration -H 'Content-Type: application/json' -d@./iteration.json - {"acknowledged":true}[atheurer@dhcp31-123 templates]$ curl -X PUT localhost:9200/_template/cdmv5dev-metric_data -H 'Content-Type: application/json' -d@./metric_data.json - {"acknowledged":true}[atheurer@dhcp31-123 templates]$ curl -X PUT localhost:9200/_template/cdmv5dev-metric_desc -H 'Content-Type: application/json' -d@./metric_desc.json - {"acknowledged":true}[atheurer@dhcp31-123 templates]$ curl -X PUT localhost:9200/_template/cdmv5dev-period -H 'Content-Type: application/json' -d@./period.json - {"acknowledged":true}[atheurer@dhcp31-123 templates]$ curl -X PUT localhost:9200/_template/cdmv5dev-run -H 'Content-Type: application/json' -d@./run.json - {"acknowledged":true}[atheurer@dhcp31-123 templates]$ curl -X PUT localhost:9200/_template/cdmv5dev-sample -H 'Content-Type: application/json' -d@./sample.json - -Then confirm the templates have been created: - - $ curl -X GET localhost:9200/_cat/templates?v | grep cdm - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed - 100 2142 100 2142 0 0 697k 0 --:--:-- --:--:-- --:--:-- 697k - cdmv5dev-run [cdmv5dev-run*] 0 - cdmv5dev-metric_data [cdmv5dev-metric_data*] 0 - cdmv5dev-sample [cdmv5dev-sample*] 0 - cdmv4dev-run [cdmv4dev-run*] 0 - cdmv5dev-metric_desc [cdmv5dev-metric_desc*] 0 - cdmv5dev-iteration [cdmv5dev-iteration*] 0 - cdmv5dev-period [cdmv5dev-period*] 0 - -Then run the command to generate the create-index commands: - - $ ./create-indices-cmds.sh localhost:9200 - curl -X PUT localhost:9200/cdmv5dev-iteration - curl -X PUT localhost:9200/cdmv5dev-metric_data - curl -X PUT localhost:9200/cdmv5dev-metric_desc - curl -X PUT localhost:9200/cdmv5dev-period - curl -X PUT localhost:9200/cdmv5dev-run - curl -X PUT localhost:9200/cdmv5dev-sample - -And like before, you will need to copy/paste these to run those commands: - - $ curl -X PUT localhost:9200/cdmv5dev-iteration - {"acknowledged":true,"shards_acknowledged":true,"index":"cdmv5dev-iteration"}[atheurer@dhcp31-123 templates]$ curl -X PUT localhost:9200/cdmv5dev-metric_data - {"acknowledged":true,"shards_acknowledged":true,"index":"cdmv5dev-metric_data"}[atheurer@dhcp31-123 templates]$ curl -X PUT localhost:9200/cdmv5dev-metric_desc - {"acknowledged":true,"shards_acknowledged":true,"index":"cdmv5dev-metric_desc"}[atheurer@dhcp31-123 templates]$ curl -X PUT localhost:9200/cdmv5dev-period - {"acknowledged":true,"shards_acknowledged":true,"index":"cdmv5dev-period"}[atheurer@dhcp31-123 templates]$ curl -X PUT localhost:9200/cdmv5dev-run - {"acknowledged":true,"shards_acknowledged":true,"index":"cdmv5dev-run"}[atheurer@dhcp31-123 templates]$ curl -X PUT localhost:9200/cdmv5dev-sample - -And then confirm those indices exist: - - $ curl -X GET localhost:9200/_cat/indices?v - health status index uuid pri rep docs.count docs.deleted store.size pri.store.size - yellow open cdmv5dev-iteration TVp86-vGSD6IFcsL1VV2Yg 1 1 0 0 230b 230b - yellow open cdmv5dev-metric_desc XmpbfBhNRAKLyHVmBkr4Pw 1 1 0 0 230b 230b - yellow open cdmv5dev-metric_data h4M_jhnMTDu5vgsxgpUQqw 1 1 0 0 230b 230b - yellow open cdmv5dev-period P5pSc2yoRPiKVEAhc5RJLQ 1 1 0 0 230b 230b - yellow open cdmv5dev-run Ng4dssP2Rr-uSuo3tZC8Pw 1 1 0 0 230b 230b - -At this point the elasticsearch instance should be ready to consume documents for CommonDataModel - -If you wish to start over with a cleared out elasticsearch instances, you can run this command which outputs commands to do so: - - $ ./create-cleanup-cmds.sh localhost:9200 - curl --stderr /dev/null -X DELETE localhost:9200/cdmv5dev-iteration - curl --stderr /dev/null -X DELETE localhost:9200/cdmv5dev-metric_desc - curl --stderr /dev/null -X DELETE localhost:9200/cdmv5dev-metric_data - curl --stderr /dev/null -X DELETE localhost:9200/cdmv5dev-period - curl --stderr /dev/null -X DELETE localhost:9200/cdmv5dev-run - curl --stderr /dev/null -X DELETE localhost:9201/_template/cdmv5dev-run - curl --stderr /dev/null -X DELETE localhost:9201/_template/cdmv5dev-metric_data - curl --stderr /dev/null -X DELETE localhost:9201/_template/cdmv5dev-sample - curl --stderr /dev/null -X DELETE localhost:9201/_template/cdmv5dev-metric_desc - curl --stderr /dev/null -X DELETE localhost:9201/_template/cdmv5dev-iteration - curl --stderr /dev/null -X DELETE localhost:9201/_template/cdmv5dev-period - -And then you can copy/paste those commands to clear out the instance. +This directory contains a single file: `delete.sh`, a destructive cleanup +script that deletes every index on a local OpenSearch instance +(`localhost:9200` only, hardcoded). + +## Usage +Crucible's `reinit_opensearch()` (in `bin/base`) invokes this script to wipe +a local OpenSearch instance's indices before reinitializing it. It requires +`curl` and takes no arguments. + + $ ./delete.sh + # Using localhost:9200 for URL + Deleting indices: + Deleting index cdmv10dev-run...success + Deleting index cdmv10dev-metric_data...success + ... + +## Index mappings +The actual OpenSearch index mappings for CommonDataModel's document types are +defined in `queries/cdmq/cdm.js`'s `indexDefs` object, not in this directory. +`add-run.js` creates and updates indices from `indexDefs` automatically as +part of normal indexing — no manual template or index setup is required. diff --git a/templates/build.sh b/templates/build.sh deleted file mode 100755 index fe6b79c4..00000000 --- a/templates/build.sh +++ /dev/null @@ -1,34 +0,0 @@ -#!/bin/bash - -ver=$1 -shift -index=$1 -file=$index.json -echo '{' >$file -echo ' "index_patterns": ["cdm'$ver'-'$index'*"],' >>$file -echo ' "settings": {' >>$file -echo ' "number_of_shards": 8,' >>$file -echo ' "number_of_replicas": 1,' >>$file -echo ' "max_result_window": 262144,' >>$file -echo ' "max_terms_count": 262144,' >>$file -echo ' "codec" : "best_compression",' >>$file -echo ' "refresh_interval" : "5s"' >>$file -echo ' },' >>$file -echo ' "mappings": {' >>$file -#if [ "$index" != "metric_desc" ]; then - echo ' "dynamic": "strict",' >>$file -#fi -echo ' "properties": {' >>$file -echo ' "cdm": {' >>$file -echo ' "properties": {' >>$file -echo ' "ver": { "type": "keyword" },' >>$file -echo ' "doctype": { "type": "keyword" }' >>$file -echo ' }' >>$file -echo ' },' >>$file -while [ ! -z "$1" ]; do - cat $1.base >> $file - shift -done -echo " }" >> $file -echo " }" >> $file -echo "}" >> $file diff --git a/templates/config_cpu_vulnerabilities.base b/templates/config_cpu_vulnerabilities.base deleted file mode 100644 index 10810753..00000000 --- a/templates/config_cpu_vulnerabilities.base +++ /dev/null @@ -1,17 +0,0 @@ - "config": { - "properties": { - "id": { "type": "keyword" }, - "host": { "type": "keyword" }, - "module": { "type": "keyword" }, - "source_type": { "type": "keyword" }, - "scribe_uuid": { "type": "keyword" }, - "cpu_vulnerabilities": { - "properties": { - "current_vulnerability_status": { "type": "boolean" }, - "mitigation_exists": { "type": "boolean" }, - "mitigation_type": { "type": "keyword" }, - "vulnerability_name": { "type": "keyword" } - } - } - } - }, diff --git a/templates/config_yum_repos.base b/templates/config_yum_repos.base deleted file mode 100644 index 4dff6700..00000000 --- a/templates/config_yum_repos.base +++ /dev/null @@ -1,16 +0,0 @@ - "config": { - "properties": { - "id": { "type": "keyword" }, - "host": { "type": "keyword" }, - "module": { "type": "keyword" }, - "source_type": { "type": "keyword" }, - "scribe_uuid": { "type": "keyword" }, - "yum_repos": { - "properties": { - "base_url" : { "type": "keyword" }, - "repo_name" : { "type": "keyword" }, - "repo_state" : { "type": "integer"} - } - } - } - }, diff --git a/templates/create-cleanup-cmds.sh b/templates/create-cleanup-cmds.sh deleted file mode 100755 index c30cc475..00000000 --- a/templates/create-cleanup-cmds.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/bin/bash - -# This will echo commands on stdout that you can copy/paste to clean up your ES instance - -es_url=$1 -if [ -z "$es_url" ]; then - echo "# Using localhost:9200 for URL" - es_url="localhost:9200" -fi - -if [ ! -x /usr/bin/curl ]; then - echo "You must have curl installed to use this script" - exit 1 -fi - -es_ver_file="../VERSION" -if [ -e "$es_ver_file" ]; then - es_ver=`cat "$es_ver_file"` -else - echo "Could not find the VERSION file which should be located in the root directory of this repository" - exit 1 -fi - -for i in `curl --stderr /dev/null -X GET "$es_url/_cat/indices?" | awk '{print $3}'`; do - echo curl --stderr /dev/null -X DELETE $es_url/$i -done - -for i in `curl --stderr /dev/null -X GET "$es_url/_cat/templates?v" | grep cdm$es_ver | awk '{print $1}'`; do - echo curl --stderr /dev/null -X DELETE $es_url/_template/$i; done - diff --git a/templates/create-indices-cmds.sh b/templates/create-indices-cmds.sh deleted file mode 100755 index beb10b31..00000000 --- a/templates/create-indices-cmds.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/bin/bash - -# This will echo to stdout commands you can copy/paste to create your ES indices -# *** Make sure you load your index templates before running any of these commands *** - -es_url=$1 -if [ -z "$es_url" ]; then - echo "# Using localhost:9200 for URL" - es_url="localhost:9200" -fi - -if [ ! -x /usr/bin/curl ]; then - echo "You must have curl installed to use this script" - exit 1 -fi - -es_ver_file="../VERSION" -if [ -e "$es_ver_file" ]; then - es_ver=`cat "$es_ver_file"` -else - echo "Could not find the VERSION file which should be located in the root directory of this repository" - exit 1 -fi - - -for i in `/bin/ls *.json | sed -e s/\.json//`; do - echo "curl -X PUT $es_url/cdm$es_ver-$i" -done -echo diff --git a/templates/create-template-cmds.sh b/templates/create-template-cmds.sh deleted file mode 100755 index 97ae08ee..00000000 --- a/templates/create-template-cmds.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/bin/bash - -# This will echo to stdout commands which you can copy/paste to load index templates into ES -# These index templates ensure new indices will adhere to the mappings - -es_url=$1 -if [ -z "$es_url" ]; then - echo "# Using localhost:9200 for URL" - es_url="localhost:9200" -fi - -if [ ! -x /usr/bin/curl ]; then - echo "You must have curl installed to use this script" - exit 1 -fi - -es_ver_file="../VERSION" -if [ -e "$es_ver_file" ]; then - es_ver=`cat "$es_ver_file"` -else - echo "Could not find the VERSION file which should be located in the root directory of this repository" - exit 1 -fi - - -for i in `/bin/ls *.json | sed -e s/\.json//`; do - echo "curl -X PUT $es_url/_template/cdm$es_ver-$i -H 'Content-Type: application/json' -d@./$i.json" -done -echo diff --git a/templates/init.sh b/templates/init.sh deleted file mode 100755 index 8460683c..00000000 --- a/templates/init.sh +++ /dev/null @@ -1,116 +0,0 @@ -#!/bin/bash -# -*- mode: sh; indent-tabs-mode: nil; sh-basic-offset: 4 -*- -# vim: autoindent tabstop=4 shiftwidth=4 expandtab softtabstop=4 filetype=bash - -pushd "`dirname $0`" >/dev/null - -es_url=$1 -if [ -z "$es_url" ]; then - es_url="localhost:9200" -fi - -if [ ! -x /usr/bin/curl ]; then - echo "You must have curl installed to use this script" - exit 1 -else - curl_cmd='curl --silent --show-error --stderr -' -fi - -es_ver_file="../VERSION" -if [ -e "$es_ver_file" ]; then - es_ver=`cat "$es_ver_file"` - echo "Current CDM version is $es_ver" -else - echo "Could not find the VERSION file which should be located in the root directory of this repository" - exit 1 -fi - -# Build the template and index jsons -make clean >/dev/null -make >/dev/null - -echo "Initializing ES..." - -echo -n "Adding cluster settings..." -# Don't allow index auto-creation -qresult=$(${curl_cmd} -X PUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d' -{ - "persistent": { - "action.auto_create_index": "false", - "search.max_buckets": 1000000 - } -} -') -if echo "${qresult}" | grep -q '"acknowledged":true'; then - echo "success" -else - echo "failed" - echo "${qresult}" -fi - #"indices.query.bool.max_clause_count": 1000000, - -echo "Checking templates and indices:" -# Create the templates and indices -for i in `/bin/ls *.json | sed -e s/\.json//`; do - echo "Checking for cdm${es_ver}-${i}:" - qresult=$(${curl_cmd} -X GET localhost:9200/_cat/templates) - if echo "${qresult}" | grep -q "$es_ver-$i"; then - echo " found template" - else - echo -n " template missing, creating..." - qresult=$(${curl_cmd} -X PUT $es_url/_template/cdm$es_ver-$i -H 'Content-Type: application/json' -d@./$i.json) - if echo "${qresult}" | grep -q '"acknowledged":true'; then - echo "success" - else - echo "failed" - echo "${qresult}" - fi - fi - older_templates=$(${curl_cmd} -X GET localhost:9200/_cat/templates | grep ^cdm | grep -- -$i | grep -v $es_ver | awk '{print $1}') - if [ ! -z "$older_templates" ]; then - for this_older_template in $older_templates; do - older_ver=`echo $this_older_template | sed -e s/^cdm// | awk -F- '{print $1}'` - echo -n " found older version ($older_ver) of this template, deleteing..." - qresult=$(${curl_cmd} -X DELETE $es_url/_template/$this_older_template) - if echo "${qresult}" | grep -q '"acknowledged":true'; then - echo "success" - else - echo "failed" - echo "${qresult}" - fi - done - fi - - qresult=$(${curl_cmd} -X GET localhost:9200/_cat/indices) - if echo "${qresult}" | grep -q "$es_ver-$i"; then - echo " found index" - else - echo -n " index missing, creating..." - create_cmd="${curl_cmd} -X PUT $es_url/cdm$es_ver-$i" - qresult=$($create_cmd) - if echo "${qresult}" | grep -q '"acknowledged":true'; then - echo "success" - else - echo "failed" - echo "${qresult}" - fi - fi - older_indices=$(${curl_cmd} -X GET localhost:9200/_cat/indices | awk '{print $3}' | grep ^cdm | grep -- -$i | grep -v $es_ver | awk '{print $1}') - if [ ! -z "$older_indices" ]; then - for this_older_index in $older_indices; do - older_ver=`echo $this_older_index | sed -e s/^cdm// | awk -F- '{print $1}'` - echo -n " found older version ($older_ver) of this index, deleting..." - qresult=$(${curl_cmd} -X DELETE $es_url/$this_older_index) - if echo "${qresult}" | grep -q '"acknowledged":true'; then - echo "success" - else - echo "failed" - echo "${qresult}" - fi - done - fi -done - -echo "...ES initilization complete!" - -popd >/dev/null diff --git a/templates/iteration.base b/templates/iteration.base deleted file mode 100644 index 24c4ee54..00000000 --- a/templates/iteration.base +++ /dev/null @@ -1,10 +0,0 @@ - "iteration": { - "properties": { - "iteration-uuid": { "type": "keyword" }, - "num": { "type": "unsigned_long" }, - "status": { "type": "keyword" }, - "path": { "type": "keyword" }, - "primary-metric": { "type": "keyword" }, - "primary-period": { "type": "keyword" } - } - }, diff --git a/templates/metric_data.base b/templates/metric_data.base deleted file mode 100644 index e7e1aab8..00000000 --- a/templates/metric_data.base +++ /dev/null @@ -1,13 +0,0 @@ - "metric_desc": { - "properties": { - "metric_desc-uuid": { "type": "keyword" } - } - }, - "metric_data": { - "properties": { - "value": { "type": "double" }, - "begin": { "type" : "date" }, - "end": { "type" : "date" }, - "duration": { "type": "long" } - } - }, diff --git a/templates/metric_desc.base b/templates/metric_desc.base deleted file mode 100644 index 1ae26202..00000000 --- a/templates/metric_desc.base +++ /dev/null @@ -1,139 +0,0 @@ - "metric_desc": { - "properties": { - "metric_desc-uuid": { "type": "keyword" }, - "default-aggregation": { "type": "keyword" }, - "class": { "type": "keyword" }, - "type": { "type": "keyword" }, - "source": { "type": "keyword" }, - "names-list": { "type": "keyword" }, - "names": { - "properties": { - "tool-name": { "type": "keyword" }, - "benchmark-name": { "type": "keyword" }, - "benchmark-group": { "type": "keyword" }, - "benchmark-id": { "type": "keyword" }, - "benchmark-role": { "type": "keyword" }, - "hostname": { "type": "keyword" }, - "engine-type": { "type": "keyword" }, - "engine-role": { "type": "keyword" }, - "engine-id": { "type": "keyword" }, - "userenv": { "type": "keyword" }, - "osruntime": { "type": "keyword" }, - "endpoint-label": { "type": "keyword" }, - "hypervisor-host": { "type": "keyword" }, - "hosted-by": { "type": "keyword" }, - "interface-type": { "type": "keyword" }, - "bridge": { "type": "keyword" }, - "interface": { "type": "keyword" }, - "id": { "type": "keyword" }, - "num": { "type": "double" }, - "class": { "type": "keyword" }, - "type": { "type": "keyword" }, - "host": { "type": "keyword" }, - "role": { "type": "keyword" }, - "dev": { "type": "keyword" }, - "cmd": { "type": "keyword" }, - "tid": { "type": "keyword" }, - "pid": { "type": "keyword" }, - "job": { "type": "keyword" }, - "group": { "type": "keyword" }, - "tier": { "type": "keyword" }, - "level": { "type": "keyword" }, - "package": { "type": "keyword" }, - "die": { "type": "keyword" }, - "core": { "type": "keyword" }, - "thread": { "type": "keyword" }, - "kthread": { "type": "keyword" }, - "node": { "type": "keyword" }, - "mode": { "type": "keyword" }, - "socket": { "type": "keyword" }, - "domain": { "type": "keyword" }, - "shared-l1-domain": { "type": "keyword" }, - "shared-l2-domain": { "type": "keyword" }, - "shared-l3-domain": { "type": "keyword" }, - "shared-l4-domain": { "type": "keyword" }, - "cluster": { "type": "keyword" }, - "container": { "type": "keyword" }, - "cgroup": { "type": "keyword" }, - "parent": { "type": "keyword" }, - "source": { "type": "keyword" }, - "controller": { "type": "keyword" }, - "pod": { "type": "keyword" }, - "port": { "type": "keyword" }, - "tx_port": { "type": "keyword" }, - "rx_port": { "type": "keyword" }, - "port_pair": { "type": "keyword" }, - "status": { "type": "keyword" }, - "error": { "type": "keyword" }, - "stream": { "type": "keyword" }, - "direction": { "type": "keyword" }, - "clientserver": { "type": "keyword" }, - "protocol": { "type": "keyword" }, - "action": { "type": "keyword" }, - "cpu": { "type": "keyword" }, - "irq": { "type": "keyword" }, - "desc": { "type": "keyword" }, - "counter": { "type": "keyword" }, - "cstype": { "type": "keyword" }, - "csid": { "type": "keyword" }, - "cookie": { "type": "keyword" }, - "table": { "type": "keyword" }, - "priority": { "type": "keyword" }, - "reg14": { "type": "keyword" }, - "metadata": { "type": "keyword" }, - "dp": { "type": "keyword" }, - "flags": { "type": "keyword" }, - "recirc_id": { "type": "keyword" }, - "dp_hash": { "type": "keyword" }, - "skb_priority": { "type": "keyword" }, - "skb_mark": { "type": "keyword" }, - "ct_state": { "type": "keyword" }, - "ct_zone": { "type": "keyword" }, - "ct_mark": { "type": "keyword" }, - "ct_label": { "type": "keyword" }, - "icmp_type": { "type": "keyword" }, - "icmp_code": { "type": "keyword" }, - "in_port": { "type": "keyword" }, - "ipv4_src": { "type": "keyword" }, - "ipv4_dst": { "type": "keyword" }, - "ipv4_proto": { "type": "keyword" }, - "ipv4_frag": { "type": "keyword" }, - "udp_src": { "type": "keyword" }, - "udp_dst": { "type": "keyword" }, - "tcp_src": { "type": "keyword" }, - "tcp_dst": { "type": "keyword" }, - "eth_src": { "type": "keyword" }, - "eth_dst": { "type": "keyword" }, - "eth_type": { "type": "keyword" }, - "vlan": { "type": "keyword" }, - "dl_dst": { "type": "keyword" }, - "dl_src": { "type": "keyword" }, - "dl_vlan": { "type": "keyword" }, - "ipv6_src": { "type": "keyword" }, - "ipv6_dst": { "type": "keyword" }, - "actions": { "type": "keyword" }, - "ufid": { "type": "keyword" }, - "src": { "type": "keyword" }, - "dst": { "type": "keyword" }, - "sport": { "type": "keyword" }, - "dport": { "type": "keyword" }, - "mark": { "type": "keyword" }, - "output": { "type": "keyword" }, - "use": { "type": "keyword" }, - "step": { "type": "keyword" }, - "epoch": { "type": "keyword" }, - "batch": { "type": "keyword" }, - "slot": { "type": "keyword" }, - "blade": { "type": "keyword" }, - "rank": { "type": "keyword" } - } - }, - "value-format": { "type": "keyword" }, - "values": { - "properties": { - "pass": { "type": "keyword" }, - "fail": { "type": "keyword" } - } - } - } - }, diff --git a/templates/param.base b/templates/param.base deleted file mode 100644 index 6993b9ed..00000000 --- a/templates/param.base +++ /dev/null @@ -1,9 +0,0 @@ - "param": { - "properties": { - "param-uuid": { "type": "keyword" }, - "id": { "type": "keyword" }, - "arg": { "type": "keyword" }, - "role": { "type": "keyword" }, - "val": { "type": "keyword" } - } - }, diff --git a/templates/period.base b/templates/period.base deleted file mode 100644 index 9ad54cf0..00000000 --- a/templates/period.base +++ /dev/null @@ -1,9 +0,0 @@ - "period": { - "properties": { - "period-uuid" : { "type": "keyword" }, - "begin" : { "type" : "date" }, - "end" : { "type" : "date" }, - "name" : { "type": "keyword" }, - "prev_id" : { "type": "keyword" } - } - }, diff --git a/templates/run.base b/templates/run.base deleted file mode 100644 index 2e84ff9e..00000000 --- a/templates/run.base +++ /dev/null @@ -1,15 +0,0 @@ - "run": { - "properties": { - "run-uuid": { "type": "keyword" }, - "begin": { "type": "date" }, - "end": { "type": "date" }, - "harness": { "type": "keyword" }, - "benchmark": { "type": "keyword" }, - "host": { "type": "keyword" }, - "email": { "type": "keyword" }, - "name": { "type": "keyword" }, - "desc": { "type": "text", "analyzer": "standard" }, - "tags": { "type": "text", "analyzer": "whitespace", "fields": { "raw": { "type": "keyword" } } }, - "source": { "type": "keyword" } - } - } diff --git a/templates/run_micro.base b/templates/run_micro.base deleted file mode 100644 index 75649e6c..00000000 --- a/templates/run_micro.base +++ /dev/null @@ -1,5 +0,0 @@ - "run": { - "properties": { - "run-uuid": { "type": "keyword" } - } - } diff --git a/templates/sample.base b/templates/sample.base deleted file mode 100644 index cceb9993..00000000 --- a/templates/sample.base +++ /dev/null @@ -1,8 +0,0 @@ - "sample": { - "properties": { - "sample-uuid": { "type": "keyword" }, - "num": { "type": "unsigned_long" }, - "status": { "type": "keyword" }, - "path": { "type": "keyword" } - } - }, diff --git a/templates/tag.base b/templates/tag.base deleted file mode 100644 index b823b33b..00000000 --- a/templates/tag.base +++ /dev/null @@ -1,7 +0,0 @@ - "tag": { - "properties": { - "tag-uuid": { "type": "keyword" }, - "name": { "type": "keyword" }, - "val": { "type": "keyword" } - } - },