2.7.1 merge (#1941)

* Core naming fix (#1919) * Fix the issue with the automated naming of cores and add a max_federate and max_broker condition to cores and brokers as well * add additional test cases and files * Automated formatting of repo files (#1920) Co-authored-by: Philip Top <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * FIx cpplint issue * fix failing test cases due to broken function call * Automated formatting of repo files (#1925) Co-authored-by: Philip Top <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Environmental variables (#1921) * Add support for environmental variables * add tests for environmental variables * Automated formatting of repo files [skip ci] (#1922) * add a global status query (#1870) * add a global status query * Automated formatting of repo files (#1871) Co-authored-by: Philip Top <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * update date and duplicate naming race condition for cores * Automated formatting of repo files (#1872) Co-authored-by: Philip Top <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Automated formatting of repo files Co-authored-by: Philip Top <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Ryan Mast <[email protected]> * fix a few missing clears and rename a variable and method in the comms to make it clearer. * Automated formatting of repo files (#1924) Co-authored-by: Philip Top <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * update docs to environment_variables Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Ryan Mast <[email protected]> * Fix an issue with the recorder capture text file (#1910) * Fix an issue with the recorder capture text file storing the type in the wrong location which could cause issues on reading the file in a player * Automated formatting of repo files (#1911) Co-authored-by: Philip Top <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * clang-tidy fixes * use Json string converters instead of a custom escaping function * Automated formatting of repo files (#1918) Co-authored-by: Philip Top <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Update docs/apps/Player.md Co-authored-by: Ryan Mast <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Ryan Mast <[email protected]> * signal handler (#1915) * add some test code for signal handling in the library * Make helicsLoadSignalHandler compile * add a signal handler callback operation and an abort operation * Add endline for signalHandler * add globalError call to broker and make abort all trigger on all federates, cores and brokers, but don't destroy the objects. * add functions and methods to the C++98 API and some docs * fix clang-tidy issues * Automated formatting of repo files (#1916) Co-authored-by: Philip Top <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix punctuation in docs * change error_code argument to match consistent Styles * Automated formatting of repo files (#1917) Co-authored-by: Philip Top <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * change the signal hander to have a boolean return value which controls whether the default handler is executed * change abort all to capture all autogenerated cores and brokers. * make changing the broker State restricted so it can't move from the error State easily. * fix callback to be C compatible * make the abort work for tcp cores as well * remove the death test from the circle ci execution * tweak the udp core timeouts and the abort calls * Fix test cases from rebase * fix some formatting issues * Automated formatting of repo files (#1927) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * warning cleanup * Apply suggestions from code review Co-authored-by: Ryan Mast <[email protected]> * Update src/helics/core/CommonCore.cpp Co-authored-by: Ryan Mast <[email protected]> Co-authored-by: Dheepak Krishnamurthy <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Ryan Mast <[email protected]> * Automated update to SWIG generated interface files (#1930) Co-authored-by: HELICS-bot <[email protected]> * Query timeouts (#1931) * Add query timeouts and some additional checks to prevent deadlocks from queries * add a tickForwarding method to the brokerBase to clarify intentions and simplify constructions. * Automated formatting of repo files (#1932) Co-authored-by: Philip Top <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix cpplint warning Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * add a test case and try to fix the wait for current time bug (#1933) * add a test case and try to fix the wait for current time bug * Automated formatting of repo files (#1935) Co-authored-by: Philip Top <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Webserver tweaks (#1936) * Fix a few minor issues with the webserver and add a default page. * add svg index page * update some webserver tests * Automated formatting of repo files (#1938) Co-authored-by: Philip Top <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Add css for broker server page * Apply suggestions from code review Co-authored-by: Ryan Mast <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Dheepak Krishnamurthy <[email protected]> Co-authored-by: Ryan Mast <[email protected]> * Update some docs and other things related to version numbers in prepa… (#1939) * Update some docs and other things related to version numbers in preparation for the 2.7.1 release. * Automated formatting of repo files (#1940) Co-authored-by: Philip Top <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Ryan Mast <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Ryan Mast <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Ryan Mast <[email protected]> Co-authored-by: Dheepak Krishnamurthy <[email protected]> Co-authored-by: HELICS-bot <[email protected]>
GMLC-TDC · Jun 4, 2021 · 5518b5b · 5518b5b
1 parent 688be69
commit 5518b5b
Show file tree

Hide file tree

Showing 476 changed files with 41,618 additions and 1,495 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,7 +6,32 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 A note on future revisions.
-Everything within a major version number should be code compatible (with the exception of experimental interfaces). The most notable example of an experimental interface is the support for multiple source inputs. The APIs to deal with this will change in future minor releases. Everything within a single minor release should be network compatible with other federates on the same minor release number. Compatibility across minor release numbers may be possible in some situations but we are not going to guarantee this as those components are subject to performance improvements and may need to be modified at some point. Patch releases will be limited to bug fixes and other improvements not impacting the public API or network compatibility. Check the [Public API](./docs/Public_API.md) for details on what is included and excluded from the public API and version stability.
+Everything within a major version number should be code compatible (with the exception of experimental interfaces). Everything within a single minor release should be network compatible with other federates on the same minor release number. Compatibility across minor release numbers may be possible in some situations but we are not going to guarantee this as those components are subject to performance improvements and may need to be modified at some point. Patch releases will be limited to bug fixes and other improvements not impacting the public API or network compatibility. Check the [Public API](./docs/Public_API.md) for details on what is included and excluded from the public API and version stability.
+
+## [2.7.1][] - 2021-06-03
+
+There were several bug fixes in this patch release. Some of them related to changes in [2.7.0][] and some new ones that came up from bug reports. Some new enhancements are experimental signal handlers in the C-api, which will be used in the python interface to provide a little better user experience when trying to kill a co-simulation.
+
+### Changed
+
+- String output on recorders is now always JSON compatible and allows escaped characters. This allows some additional values to be displayed in ascii format vs base 64 encoding. #1910
+- Players read the string fields through a JSON parser unless marked with b64\[\] to match the string output on recorders #1910
+- The default webserver port is now 8080 to allow user space execution on non-Windows platforms #1936
+
+### Fixed
+
+- An issue with recorders writing text fields in the incorrect order which could result in incorrect playback #1910
+- Fix an issue with core naming that occasionally resulted in same broker name errors when using default names on federates #1919
+- Fix an issue where queries were not being resolved when a core disconnects which could result in deadlock. #1931
+- The `wait_for_current_time` flag was not working properly in some cases where time interruption was also taking place #1933
+- Fixed issue with the webserver not responding with the index page when requested or detecting the correct broker for certain trivial requests #1936
+
+### Added
+
+- Signal handlers for catching SIGINT and optional user callback are available in the C shared API #1915
+- Added support for environment variables for setting some network connection settings and other information #1921
+- Queries now have timeouts #1931
+- Command line and environment variable options for setting the webserver port numbers #1936
 
 ## [2.7.0][] - 2021-04-28
 
@@ -837,3 +862,5 @@ This is a major revision so this changelog will not capture all the changes that
 [2.5.2]: https://github.com/GMLC-TDC/HELICS/releases/tag/v2.5.2
 [2.6.0]: https://github.com/GMLC-TDC/HELICS/releases/tag/v2.6.0
 [2.6.1]: https://github.com/GMLC-TDC/HELICS/releases/tag/v2.6.1
+[2.7.0]: https://github.com/GMLC-TDC/HELICS/releases/tag/v2.7.0
+[2.7.1]: https://github.com/GMLC-TDC/HELICS/releases/tag/v2.7.1
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -23,14 +23,14 @@ if(DEFINED ENV{VCPKG_ROOT} AND NOT DEFINED CMAKE_TOOLCHAIN_FILE)
     set(CMAKE_TOOLCHAIN_FILE "$ENV{VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake" CACHE STRING "")
 endif()
 
-project(HELICS VERSION 2.7.0)
+project(HELICS VERSION 2.7.1)
 
 # -----------------------------------------------------------------------------
 # HELICS Version number
 # -----------------------------------------------------------------------------
 set(HELICS_VERSION_BUILD)
 # use ISO date YYYY-MM-DD
-set(HELICS_DATE "2021-04-29")
+set(HELICS_DATE "2021-06-05")
 
 set(HELICS_VERSION_UNDERSCORE
     "${HELICS_VERSION_MAJOR}_${HELICS_VERSION_MINOR}_${HELICS_VERSION_PATCH}"

diff --git a/README.md b/README.md
@@ -69,6 +69,8 @@ HELICS is also highly scalable, enabling everything from simple connections betw
 
 ## Getting Started
 
+The users guide has been completely redone for the upcoming HELICS 3. This version is not released yet but it is close, and the [docs](https://docs.helics.org/en/helics3/) are mostly done. We suggest you check it out if you are looking for more documentation.
+
 We've created a series of roughly 10-minute mini-tutorial videos that discuss various design topics, concepts, and interfaces, including how to use the tool. They can be found on our [YouTube channel](https://www.youtube.com/channel/UCPa81c4BVXEYXt2EShTzbcg).
 
 The [Introduction to the HELICS documentation](https://helics.readthedocs.io/en/latest/introduction/index.html) goes through a series of examples that step through the basic usage and concepts of HELICS.

diff --git a/ThirdParty/concurrency b/ThirdParty/concurrency
diff --git a/appveyor.yml b/appveyor.yml
@@ -7,7 +7,7 @@ branches:
     - develop
     - helics3
 
-version: 2.7.0.{build}
+version: 2.7.1.{build}
 
 image: Visual Studio 2015
 

diff --git a/docs/apps/Player.md b/docs/apps/Player.md
@@ -133,7 +133,7 @@ m <sendtime> <deliverytime> <source> <dest> <time> <data>
 ```
 
 the second option allows sending events at a different time than they are triggered
-the data portion of messages can be encoded in base64 by marking as b64[<data>] or base64[X] all data between the brackets will be converted to raw binary. A ']' must be last
+the data portion of messages can be encoded in base64 by marking as b64[<data>] or base64[X] all data between the brackets will be converted to raw binary. A ']' must be last. The string interpreter can also handle messages with any escapable characters including tab ("\t"), newline ("\n"), and quote ("\""), this can be marked by using quotes as in `"<message>"` to make it interpret the message as a JSON quoted string.
 
 ### JSON configuration
 

diff --git a/docs/user-guide/debugging.md b/docs/user-guide/debugging.md
@@ -11,10 +11,10 @@ HELICS provides built-in support for various types of logging from federates, br
 
 Often an early step in debugging is to isolate the problem. HELICS provides the [**Recorder**](../apps/Recorder.md) and [**Player**](../apps/Player.md) apps to help with doing so for federates and to enable modular development and testing of federations even before all of the federates are working:
 
-- A recorder is just that: a tool that readily records some or all of the traffic that is sent via HELICS (both value-based publication/subscrition and message end points). Looking through this output can confirm that the data and timing in HELICS behaves as expected.
-- The basic idea for a player is that rather than running all of the other federates, you can test out a single or subset by reading the data that other federates would send from a file instead.
+- A recorder is just that: a tool that readily records some or all of the traffic that is sent via HELICS (both value-based publication/subscription and message end points). Looking through this output can confirm that the data and timing in HELICS behaves as expected.
+- The basic idea for a player is that rather than running all of the other federates, you can test out a single federate or subset by reading the data that other federates would send from a file instead.
 
-Players and recorders can also compliment each other, since the files created by a recorder can be played back directly by the players, making it possible to take the partial results from a part of a federation as a recording and then play back--without possible delays or other challenges from the other federates--to troubleshoot the rest of the federation.
+Players and recorders can also complement each other, since the files created by a recorder can be played back directly by the players, making it possible to take the partial results from a part of a federation as a recording and then play back--without possible delays or other challenges from the other federates--to troubleshoot the rest of the federation.
 
 ## Queries
 

diff --git a/docs/user-guide/environment_variables.md b/docs/user-guide/environment_variables.md
@@ -0,0 +1,24 @@
+# Environment variables
+
+The HELICS command line processor has some ability to read and interpret command line environment variables. These can assist in setting up co-simulations.
+In general the configuration of HELICS comes from 3 sources during setup. After setup API's exist for changing the configuration later. The highest priority is given to command line arguments. The second priority is given to configuration files, which can be given through a command string such as `--config=configFile.ini`. The file can be a `.ini, .toml, or .json`. By default HELICS looks for a `helicsConfig.ini` file. The lowest priority option is though environment variables. Only a subset of controls work with environment variables. All environment variables used by HELICS begin with `HELICS_`.
+
+## Federate environment variables
+
+For setting up a federate a few environment variables are used
+
+- `HELICS_LOG_LEVEL`: the log level for the federate to use. Equivalent to `--loglevel=X`
+- `HELICS_CORE_INIT_STRING`: the init string to pass to the core when creating it. Equivalent to `--coreinit=X`
+- `HELICS_CORE_TYPE`: the type of core to use e.g. "ZMQ", "TCP", "IPC", "MPI", etc. Equivalent to specifying `--coretype=X`
+
+## Core and Broker environment variables
+
+- `HELICS_BROKER_LOG_LEVEL`: the log level for the broker to use. Equivalent to `--loglevel=X`
+- `HELICS_BROKER_KEY`: the key to use for connecting a core to a broker. See [broker key]()
+- `HELICS_BROKER_ADDRESS`: the interface address of the broker. Equivalent to `--brokeraddress=X`
+- `HELICS_BROKER_PORT`: the port number of the broker. Equivalent to `--brokerport=X`
+- `HELICS_CONNECTION_PORT`: the port number to use for connecting. This has different behavior for cores and brokers. For cores this is the broker port and for brokers this is the local port
+- `HELICS_CONNECTION_ADDRESS`: the interface address to use for connecting. This has different behavior for cores and brokers. For cores this is the broker address and for brokers this is the interface.
+- `HELICS_LOCAL_PORT`: the port number to use on the local interface for external connections. Equivalent to `--localport=X`
+- `HELICS_BROKER_INIT`: the command line arguments to give to an autogenerated broker. Equivalent to `--brokerinit=X`
+- `HELICS_CORE_TYPE` : the type of core to use e.g. "ZMQ", "TCP", "IPC", "MPI", etc. Equivalent to specifying `--coretype=X`
diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md
@@ -16,6 +16,7 @@ There are a number of classes of HELICS users:
 - [**Co-Simulation Overview**](./co-simulation_overview.md) - A more detailed discussion of what co-simulation is and how it is used
 - [**HELICS Key Concepts**](./helics_key_concepts) - Key terms and concepts to understand before running co-simulations with HELICS
 - [**HELICS Co-Simulation Walk-through**](./helics_co-sim_sequence.md) - A notional walk-through of a simple transmission and distribution HELICS co-simulation to show the basic steps the software runs through
+- [**environment variables**](./environment_variables.md) - A discussion of HELICS supported environment variables for use in setting up a co-simulation
 - [**Federates**](./federates.md) - Discussion of the different types of federates in HELICS ([value federates](./value_federates.md) and [message federates](./message_federates.md)) and how configure them
 - [**Message Filters**](./filters) - How HELICS message filters can be implemented natively in HELICS or as stand-alone federates
 - [**Co-Simulation Timing**](./timing.md) - How HELICS coordinates the simulation time of all the federates in the federation
@@ -33,6 +34,7 @@ There are a number of classes of HELICS users:
 - [**N to 1 input connections**](./multiSourceInputs.md) - Handling multiple publications to a single input
 - **Large Co-Simulations in HELICS (forthcoming)** - How to run HELICS co-simulations with a large (100+) number of federates
 - [**Debugging**](./debugging.md) - Capabilities to help with debugging
+- [**Terminating a co-simulation**](./program_termination.md) - Some helpful tools for terminating a co-simulation
 
 ## Additional Resources
 

diff --git a/docs/user-guide/program_termination.md b/docs/user-guide/program_termination.md
@@ -0,0 +1,76 @@
+# Terminating HELICS
+
+If executing from a C or C++ based program. Ctrl-C should do the right thing, and terminate the local program. If the co-simulation is running across multiple machines then the remaining programs won't terminate properly and will either timeout or if that was disabled potentially deadlock.
+
+## Signal handler facilities
+
+The C shared library has some facilities to enable a signal handler.
+
+```c
+/** load a signal handler that handles Ctrl-C (SIGINT) and shuts down the library*/
+void helicsLoadSignalHandler();
+
+/** clear HELICS based signal Handlers*/
+void helicsClearSignalHandler();
+
+```
+
+This function will insert a signal handler that generates a global error on known objects and waits a certain amount time, clears the print buffer, and terminates.
+
+**NOTE** : the signal handlers use unsafe operations, so there is no guarantee they will work, or that they will work as expected. Testing indicates they work in most situations and improve operations where needed but it is not 100% reliable or safe code. They make use of atomic variables, mutexes, and other constructs that are not technically safe in signal handlers. The primary use case is program termination so the effects are minimized and they usually work, but the unsafe nature of them should be kept in mind.
+
+```c
+/** load a custom signal handler to execute prior to the abort signal handler
+@details  This function is not 100% reliable it will most likely work but uses some functions and
+techniques that are not 100% guaranteed to work in a signal handler
+and in worst case it could deadlock.  That is somewhat unlikely given usage patterns
+but it is possible*/
+void helicsLoadSignalHandlerCallback(helics_bool (*handler)(int));
+
+```
+
+It is also possible to insert a custom callback into the signal handler chain. Again this is not 100% reliable. But is useful for some language API's that do other things to signals. This allows for inserting a custom operation that does some other cleanup. The callback has a helics_boolean return value. If the value is set to `helics_true`(or any positive value) then the normal Signal handler is called which aborts ongoing federations and exits. If it is set to `helics_false` then the default callback is not executed.
+
+### Signal handlers in C++
+
+Facilities for signal handling in C++ were not put in place since it is easy enough for a user to place their own handlers which would likely do a better job than any built in ones, so a default one was not put in place at present though may be at a later time.
+
+## Generating an error
+
+A global error generated anywhere in a federation will terminate the co-simulation.
+
+```c
+/**
+ * generate a global error through a broker  this will terminate the federation
+ *
+ * @param broker The broker to set the time barrier for.
+ * @param errorCode the error code to associate with the global error
+ * @param errorString an error message to associate with the error
+ * @param[in,out] err An error object that will contain an error code and string if any error occurred during the execution of the function.
+ */
+void helicsBrokerGlobalError(helics_broker broker, int errorCode, const char *errorString, helics_error* err);
+
+void helicsCoreGlobalError(helics_core core, int errorCode, const char* errorString, helics_error* err);
+
+/**
+ * Generate a global error from a federate.
+ *
+ * @details A global error halts the co-simulation completely.
+ *
+ * @param fed The federate to create an error in.
+ * @param error_code The integer code for the error.
+ * @param error_string A string describing the error.
+ */
+HELICS_EXPORT void helicsFederateGlobalError(helics_federate fed, int error_code, const char* error_string);
+
+```
+
+Corresponding functions are available in the C++ API as well. Any global error will cause a termination of the co-simulation.
+
+### Some modifying flags
+
+Setting the `helics_terminate_on_error` flag to true will escalate any local error into a global one and terminate the co-simulation. This includes any mismatched configuration or other local issues.
+
+## Comments
+
+Generally it isn't a wise idea to just terminate the co-simulation without letting everyone else know. If you control everything it probably works fine but as co-simulations get larger more care needs to be taken to prevent zombie processes and hung federates and brokers. Which can cause issues on the next one. This is an evolving area of how best to handle terminating large co-simulations in abnormal conditions and hopefully the best practices will make it easier for users.
diff --git a/docs/user-guide/queries.md b/docs/user-guide/queries.md
@@ -243,7 +243,7 @@ The Following queries will be answered by a broker.
 +--------------------------+-------------------------------------------------------------------------------------+
 | ``global_flush``         | a query that just flushes the current system and returns the id's [JSON]            |
 +--------------------------+-------------------------------------------------------------------------------------+
-| ``global_status``        | an aggregate query that returns a combo of global_time and current state [JSON]     |
+| ``global_status``        | an aggregate query that returns a combo of global_time and current_state [JSON]     |
 +--------------------------+-------------------------------------------------------------------------------------+
 ```
 
@@ -284,3 +284,7 @@ This function returns a query object that can be used in one of the execute func
 It can be called asynchronously on a federate. The target field may be empty if the query is intended to be used on a local federate, in which case the target is assumed to be the federate itself.
 A query must be freed after use.
 The interface api's (python, matlab, octave, Java, etc) will work similarly.
+
+## Timeouts
+
+As long as timeouts are enabled in the library itself, queries have a timeout system so they don't block forever if a federate fails or some other condition occurs. The current default is 15 seconds. It can be changed by using the command line option `--querytimeout` on cores or brokers (or in `--coreinitstring` on cores). In a later version an ability to set this and some other timeout values through properties will likely be added (HELICS 3.1). If the query times out a value of #timeout will be returned in the string.
+2 −2		.circleci/config.yml
+2 −2		CMakeLists.txt
+1 −1		azure-pipelines.yml
+56 −15		gmlc/concurrency/DelayedObjects.hpp
+20 −0		tests/DelayedObjectsTests.cpp
-Original file line number
+Diff line change
@@ Expand Up / @@ -7,7 +7,7 @@ branches: @@
         - develop
         - helics3
-    version: 2.7.0.{build}
+    version: 2.7.1.{build}
     image: Visual Studio 2015
@@ Expand Down @@