CSV Improvements: RVI/MVI and Units

design/FY2022/EnergyPlusOutputCSVFormatting_NFP.md

@@ -0,0 +1,111 @@
+# EnergyPlus Output CSV Formatting (RVI/MVI) #
+
+**Brett Bass, Mark Adams**
+
+**Oak Ridge National Laboratory**
+
+- Date: April 7, 2022
+
+## Justification for new feature ##
+
+---
+EnergyPlus output CSV format does not currently allow for ordering of reporting or meter variables or specification of variable units.
+
+## Overview ##
+
+---
+
+The EnergyPlus native CSV output is a useful new feature but it eliminates the need for “ReadVarsESO”. “ReadVarsESO” previously had the capability to use “.rvi” or “.mvi” file types. The “.rvi” file was used to order and filter the reporting variables while the “.mvi” was used to order and filter meter variables. A need for unit conversions in the native CSV output has also been raised. This was previously possible using the convertESOMTR executable.
+These new features will address these deficiencies in the native CSV output by adding “rvi”, “mvi”, “CSV:Style” objects which can be used to:
+
+- Filter and order the output reporting (rvi) and meter variables (mvi)
+- Convert output units (CSV:Style)
+
+## Approach ##
+
+---
+
+This new feature will be added through and execution of these subtasks:
+
+1. Create three new EnergyPlus input objects (rvi, mvi, ouput units)
+2. Properly read and integrate new EnergyPlus input objects
+3. Add filtering and sorting mechanism based on rvi and mvi input object
+4. Add ability to convert units between SI and imperial units to “CSVWriter::parseTSOutputs” based on output unit input object
+5. Create unit tests to ensure that column order and unit conversions are correct
+6. Document changes
+
+## Testing/Validation/Data Sources ##
+
+---
+Additions will be made to the Input Output Reference for the mvi, rvi, and output unit objects.
+
+## Input Description ##
+
+---
+
+```
+OutputControl:RVI,
+    \extensible:1
+       \memo Determine order of CSV output reporting variables
+    \unique-object
+  A1,  \field Key Name 1
+       \begin-extensible
+  A2,  \field Key Name 2
+  A3,  \field Key Name 3
+  A4,  \field Key Name 4
+  A5,  \field Key Name 5
+  A6,  \field Key Name 6
+  A7,  \field Key Name 7
+  A8,  \field Key Name 8
+  A9,  \field Key Name 9  
+  A10;  \field Key Name 10  
+
+OutputControl:MVI,
+    \extensible:1
+       \memo Determine order of CSV output meter variables
+    \unique-object
+  A1,  \field Key Name 1
+       \begin-extensible
+  A2,  \field Key Name 2
+  A3,  \field Key Name 3
+  A4,  \field Key Name 4
+  A5,  \field Key Name 5
+  A6,  \field Key Name 6
+  A7,  \field Key Name 7
+  A8,  \field Key Name 8
+  A9,  \field Key Name 9  
+  A10;  \field Key Name 10  
+
+OutputControl:CSV:Style,
+       \memo Unit conversions on reporting and meter variables
+       \unique-object
+   A1; \field Unit Conversion
+       \type choice
+       \key None
+       \key JtoKWH
+       \key JtoMJ
+       \key JtoGJ
+       \key InchPound
+       \default None 
+```
+
+## Outputs Description ##
+
+---
+The order of output reporting and meter variables in the CSV output and the units of the CSV output may change based on the rvi, mvi, and output unit input objects.
+
+## Engineering Reference ##
+
+---
+There will be no change to the Engineering Reference as only column order and units will be changed with this new feature.
+
+## Example File and Transition Changes ##
+
+---
+Suitable example file will be generated to demonstrate the ability to change output CSV column order and units
+
+## References ##
+
+---
+
+NA

idd/Energy+.idd.in

@@ -105095,6 +105095,55 @@ Meter:CustomDecrement,
    A47; \field Output Variable or Meter Name 22
 
 
+OutputControl:RVI,
+       \extensible:1
+       \memo Determine order of CSV output reporting variables
+       \unique-object
+  A1,  \field Key Name 1
+       \begin-extensible
+  A2,  \field Key Name 2
+  A3,  \field Key Name 3
+  A4,  \field Key Name 4
+  A5,  \field Key Name 5
+  A6,  \field Key Name 6
+  A7,  \field Key Name 7
+  A8,  \field Key Name 8
+  A9,  \field Key Name 9
+  A10;  \field Key Name 10
+
+
+OutputControl:MVI,
+       \extensible:1
+       \memo Determine order of CSV output meter variables
+       \unique-object
+  A1,  \field Key Name 1
+       \begin-extensible
+  A2,  \field Key Name 2
+  A3,  \field Key Name 3
+  A4,  \field Key Name 4
+  A5,  \field Key Name 5
+  A6,  \field Key Name 6
+  A7,  \field Key Name 7
+  A8,  \field Key Name 8
+  A9,  \field Key Name 9
+  A10;  \field Key Name 10
+
+OutputControl:Timestamp,
+       \memo Control timestamp format, currently applies only to JSON and CSV
+       \unique-object
+  A1,  \field ISO 8601 Format
+       \note Use the ISO 8601 format YYYY-MM-DDThh:mm:ss
+       \type choice
+       \key Yes
+       \key No
+       \default No
+  A2;  \field Timestamp at end of interval
+       \note Determines where the timestamp is produced, either at the beginning (No) or end (Yes) of the interval
+       \type choice
+       \key Yes
+       \key No
+       \default Yes
+
 OutputControl:Files,
        \memo Conditionally turn on/off output from EnergyPlus.
        \unique-object

scripts/dev/generate_epJSON_schema/modify_schema.py

@@ -183,6 +183,8 @@ def isInt(s):
     'PythonPlugin:Variables': 'global_py_vars',
     'PythonPlugin:SearchPaths': 'py_search_paths',
     'Output:Diagnostics': 'diagnostics',
+    'OutputControl:RVI': 'output_variables',
+    'OutputControl:MVI': 'output_meters'
 }
 remaining_objects = [
     'Site:SpectrumData',

src/EnergyPlus/IOFiles.cc

@@ -52,6 +52,7 @@
 #include "FileSystem.hh"
 #include "InputProcessing/EmbeddedEpJSONSchema.hh"
 #include "InputProcessing/InputProcessor.hh"
+#include "ResultsFramework.hh"
 #include "UtilityRoutines.hh"
 
 #include <algorithm>
@@ -442,6 +443,67 @@ void IOFiles::OutputControl::getInput(EnergyPlusData &state)
             }
         }
     }
+
+    auto &ip = state.dataInputProcessing->inputProcessor;
+    auto const rvi_instances = ip->epJSON.find("OutputControl:RVI");
+    if (rvi_instances != ip->epJSON.end()) {
+        auto const &objectSchemaProps = ip->getObjectSchemaProps(state, "OutputControl:RVI");
+        auto const &instancesValue = rvi_instances.value();
+        for (auto instance = instancesValue.begin(); instance != instancesValue.end(); ++instance) {
+            auto const &fields = instance.value();
+            ip->markObjectAsUsed("OutputControl:RVI", instance.key());
+
+            auto extensibles = fields.find("output_variables");
+            if (extensibles != fields.end()) {
+                auto const &extensionSchemaProps = objectSchemaProps["output_variables"]["items"]["properties"];
+                auto const &extensiblesArray = extensibles.value();
+                rviKeyNames.reserve(extensiblesArray.size());
+                for (auto extensibleInstance = extensiblesArray.begin(); extensibleInstance != extensiblesArray.end(); ++extensibleInstance) {
+                    rviKeyNames.emplace_back(ip->getAlphaFieldValue(extensibleInstance.value(), extensionSchemaProps, "key_name"));
+                }
+            }
+        }
+    }
+
+    auto const mvi_instances = ip->epJSON.find("OutputControl:MVI");
+    if (mvi_instances != ip->epJSON.end()) {
+        auto const &objectSchemaProps = ip->getObjectSchemaProps(state, "OutputControl:MVI");
+        auto const &instancesValue = mvi_instances.value();
+        for (auto instance = instancesValue.begin(); instance != instancesValue.end(); ++instance) {
+            auto const &fields = instance.value();
+            ip->markObjectAsUsed("OutputControl:MVI", instance.key());
+
+            auto extensibles = fields.find("output_meters");
+            if (extensibles != fields.end()) {
+                auto const &extensionSchemaProps = objectSchemaProps["output_meters"]["items"]["properties"];
+                auto const &extensiblesArray = extensibles.value();
+                mviKeyNames.reserve(extensiblesArray.size());
+                for (auto extensibleInstance = extensiblesArray.begin(); extensibleInstance != extensiblesArray.end(); ++extensibleInstance) {
+                    mviKeyNames.emplace_back(ip->getAlphaFieldValue(extensibleInstance.value(), extensionSchemaProps, "key_name"));
+                }
+            }
+        }
+    }
+
+    auto const timestamp_instances = ip->epJSON.find("OutputControl:Timestamp");
+    if (timestamp_instances != ip->epJSON.end()) {
+        auto const &objectSchemaProps = ip->getObjectSchemaProps(state, "OutputControl:Timestamp");
+        auto const &instancesValue = timestamp_instances.value();
+        for (auto instance = instancesValue.begin(); instance != instancesValue.end(); ++instance) {
+            auto const &fields = instance.value();
+            ip->markObjectAsUsed("OutputControl:Timestamp", instance.key());
+
+            auto item = fields.find("iso_8601_format");
+            if (item != fields.end()) {
+                state.dataResultsFramework->resultsFramework->setISO8601(item->get<std::string>() == "Yes");
+            }
+
+            item = fields.find("timestamp_at_end_of_interval");
+            if (item != fields.end()) {
+                state.dataResultsFramework->resultsFramework->setStartOfInterval(item->get<std::string>() == "Yes");
+            }
+        }
+    }
 }
 
 void IOFiles::flushAll()

src/EnergyPlus/IOFiles.hh

@@ -701,6 +701,9 @@ public:
         bool json = true;
         bool tabular = true;
         bool sqlite = true;
+
+        std::vector<std::string> rviKeyNames;
+        std::vector<std::string> mviKeyNames;
     };
 
     OutputControl outputControl;

src/EnergyPlus/OutputProcessor.cc

@@ -2465,7 +2465,7 @@ namespace OutputProcessor {
                                          ScheduleManager::dayTypeNames[CurDayType]);
                 if (state.dataResultsFramework->resultsFramework->TSMeters.rDataFrameEnabled()) {
                     state.dataResultsFramework->resultsFramework->TSMeters.newRow(
-                        state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, EndMinute);
+                        state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, EndMinute, state.dataGlobal->CalendarYear);
                 }
                 PrintTimeStamp = false;
                 PrintTimeStampToSQL = false;
@@ -2573,7 +2573,7 @@ namespace OutputProcessor {
                                          ScheduleManager::dayTypeNames[CurDayType]);
                 if (state.dataResultsFramework->resultsFramework->HRMeters.rDataFrameEnabled()) {
                     state.dataResultsFramework->resultsFramework->HRMeters.newRow(
-                        state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0);
+                        state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0, state.dataGlobal->CalendarYear);
                 }
                 PrintTimeStamp = false;
                 PrintTimeStampToSQL = false;
@@ -2656,7 +2656,7 @@ namespace OutputProcessor {
                                          ScheduleManager::dayTypeNames[CurDayType]);
                 if (state.dataResultsFramework->resultsFramework->DYMeters.rDataFrameEnabled()) {
                     state.dataResultsFramework->resultsFramework->DYMeters.newRow(
-                        state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0);
+                        state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0, state.dataGlobal->CalendarYear);
                 }
                 PrintTimeStamp = false;
                 PrintTimeStampToSQL = false;
@@ -2729,7 +2729,7 @@ namespace OutputProcessor {
                                          state.dataEnvrn->Month);
                 if (state.dataResultsFramework->resultsFramework->MNMeters.rDataFrameEnabled()) {
                     state.dataResultsFramework->resultsFramework->MNMeters.newRow(
-                        state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0);
+                        state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0, state.dataGlobal->CalendarYear);
                 }
                 PrintTimeStamp = false;
                 PrintTimeStampToSQL = false;
@@ -2795,7 +2795,7 @@ namespace OutputProcessor {
                     state, state.files.mtr, op->YearlyStampReportChr, state.dataGlobal->CalendarYearChr, PrintTimeStamp && PrintTimeStampToSQL);
                 if (state.dataResultsFramework->resultsFramework->YRMeters.rDataFrameEnabled()) {
                     state.dataResultsFramework->resultsFramework->YRMeters.newRow(
-                        state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0);
+                        state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0, state.dataGlobal->CalendarYear);
                 }
                 PrintTimeStamp = false;
                 PrintTimeStampToSQL = false;
@@ -2872,7 +2872,7 @@ namespace OutputProcessor {
                                          PrintTimeStamp && PrintTimeStampToSQL);
                 if (state.dataResultsFramework->resultsFramework->SMMeters.rDataFrameEnabled()) {
                     state.dataResultsFramework->resultsFramework->SMMeters.newRow(
-                        state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0);
+                        state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0, state.dataGlobal->CalendarYear);
                 }
                 PrintTimeStamp = false;
                 PrintTimeStampToSQL = false;
@@ -4991,13 +4991,19 @@ void UpdateDataandReport(EnergyPlusData &state, OutputProcessor::TimeStepType co
 
     if (state.dataResultsFramework->resultsFramework->timeSeriesEnabled()) {
         if (t_TimeStepTypeKey == TimeStepType::Zone) {
-            state.dataResultsFramework->resultsFramework->RIDetailedZoneTSData.newRow(
-                state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, op->TimeValue.at(TimeStepType::Zone).CurMinute);
+            state.dataResultsFramework->resultsFramework->RIDetailedZoneTSData.newRow(state.dataEnvrn->Month,
+                                                                                      state.dataEnvrn->DayOfMonth,
+                                                                                      state.dataGlobal->HourOfDay,
+                                                                                      op->TimeValue.at(TimeStepType::Zone).CurMinute,
+                                                                                      state.dataGlobal->CalendarYear);
         }
         if (t_TimeStepTypeKey == TimeStepType::System) {
             // TODO this was an error probably, was using TimeValue(1)
-            state.dataResultsFramework->resultsFramework->RIDetailedHVACTSData.newRow(
-                state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, op->TimeValue.at(TimeStepType::System).CurMinute);
+            state.dataResultsFramework->resultsFramework->RIDetailedHVACTSData.newRow(state.dataEnvrn->Month,
+                                                                                      state.dataEnvrn->DayOfMonth,
+                                                                                      state.dataGlobal->HourOfDay,
+                                                                                      op->TimeValue.at(TimeStepType::System).CurMinute,
+                                                                                      state.dataGlobal->CalendarYear);
         }
     }
 
@@ -5187,8 +5193,11 @@ void UpdateDataandReport(EnergyPlusData &state, OutputProcessor::TimeStepType co
                 state.dataResultsFramework->resultsFramework->initializeITSDataFrame(
                     ReportingFrequency::TimeStep, op->IVariableTypes, op->NumOfIVariable);
             }
-            state.dataResultsFramework->resultsFramework->RITimestepTSData.newRow(
-                state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, op->TimeValue.at(TimeStepType::Zone).CurMinute);
+            state.dataResultsFramework->resultsFramework->RITimestepTSData.newRow(state.dataEnvrn->Month,
+                                                                                  state.dataEnvrn->DayOfMonth,
+                                                                                  state.dataGlobal->HourOfDay,
+                                                                                  op->TimeValue.at(TimeStepType::Zone).CurMinute,
+                                                                                  state.dataGlobal->CalendarYear);
         }
 
         for (auto &thisTimeStepType : {TimeStepType::Zone, TimeStepType::System}) { // Zone, HVAC
@@ -5358,7 +5367,7 @@ void UpdateDataandReport(EnergyPlusData &state, OutputProcessor::TimeStepType co
                     ReportingFrequency::Hourly, op->IVariableTypes, op->NumOfIVariable);
             }
             state.dataResultsFramework->resultsFramework->RIHourlyTSData.newRow(
-                state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0);
+                state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0, state.dataGlobal->CalendarYear);
         }
 
         for (auto &thisTimeStepType : {TimeStepType::Zone, TimeStepType::System}) { // Zone, HVAC
@@ -5461,7 +5470,7 @@ void UpdateDataandReport(EnergyPlusData &state, OutputProcessor::TimeStepType co
                     ReportingFrequency::Daily, op->IVariableTypes, op->NumOfIVariable);
             }
             state.dataResultsFramework->resultsFramework->RIDailyTSData.newRow(
-                state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0);
+                state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0, state.dataGlobal->CalendarYear);
         }
 
         op->NumHoursInMonth += 24;
@@ -5510,7 +5519,7 @@ void UpdateDataandReport(EnergyPlusData &state, OutputProcessor::TimeStepType co
                     ReportingFrequency::Monthly, op->IVariableTypes, op->NumOfIVariable);
             }
             state.dataResultsFramework->resultsFramework->RIMonthlyTSData.newRow(
-                state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0);
+                state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0, state.dataGlobal->CalendarYear);
         }
 
         op->NumHoursInSim += op->NumHoursInMonth;
@@ -5557,7 +5566,7 @@ void UpdateDataandReport(EnergyPlusData &state, OutputProcessor::TimeStepType co
                     ReportingFrequency::Simulation, op->IVariableTypes, op->NumOfIVariable);
             }
             state.dataResultsFramework->resultsFramework->RIRunPeriodTSData.newRow(
-                state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0);
+                state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0, state.dataGlobal->CalendarYear);
         }
         for (auto &thisTimeStepType : {TimeStepType::Zone, TimeStepType::System}) { // Zone, HVAC
             for (int Loop = 1; Loop <= op->NumOfRVariable; ++Loop) {
@@ -5594,7 +5603,7 @@ void UpdateDataandReport(EnergyPlusData &state, OutputProcessor::TimeStepType co
                     ReportingFrequency::Yearly, op->IVariableTypes, op->NumOfIVariable);
             }
             state.dataResultsFramework->resultsFramework->RIYearlyTSData.newRow(
-                state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0);
+                state.dataEnvrn->Month, state.dataEnvrn->DayOfMonth, state.dataGlobal->HourOfDay, 0, state.dataGlobal->CalendarYear);
         }
         for (auto &thisTimeStepType : {TimeStepType::Zone, TimeStepType::System}) { // Zone, HVAC
             for (int Loop = 1; Loop <= op->NumOfRVariable; ++Loop) {