WIP: Shell commands and function callbacks #1175

siramok · 2023-07-17T19:20:03Z

Adds the following new capabilities to ascent:

Execute arbitrary shell commands, defined as ascent actions.
Register simulation functions with ascent so that they can be executed later as callbacks.
Execute arbitrary callbacks (have to be registered first), defined as ascent actions.

Example shell command usage (subject to change):

-
  action: "add_shell_commands"
  shell_commands:
    c1:
      params:
        command: 'echo This shell command was executed by ascent > test1'
    c2:
      params:
        command: 'echo This shell command was also executed ascent > test2'

Example callback usage (subject to change):

void test_func() {
  ...
}

int main() {
  ...
  Ascent a;
  a.open();
  a.register_callback("test_func", test_func);
  ...
}

-
  action: "add_callbacks"
  callbacks:
    c1:
      params:
        name: "test_func"

Still to do, we discussed:

Allowing users to define multi-line shell commands under a single command:.
Finding a different place for m_callback_map to live, other than flow_workspace.
Allowing functions that return bool to be registered as callbacks (only void functions with no arguments will work right now).
Dependent on the previous, letting callbacks that return bool be used directly as trigger conditions (maybe simpler if we only allow callbacks that return bool and not void in the first place?).

cyrush · 2023-07-18T17:25:25Z

Awesome, thank you for creating this is a nice foundation!

Here are some ideas to discuss how we can evolve:

How about "commands" as an abstraction for either a shell command or a callback?

For example:

-
  action: "add_commands"
  commands:
    c1:
      params:
        shell_command: 'echo This shell command was executed by ascent > test1'
    c2:
      params:
        callback: 'calbackname'

This approach would also work if we want to use either shell cmds/callbacks as trigger inputs in the future, for example:

-
  action: "add_triggers"
  triggers:
    t1:
      params:
        shell_command: 'echo This shell command was executed by ascent > test1'
        actions_file : "my_trigger_actions.yaml"

-
  action: "add_triggers"
  triggers:
    t1:
      params:
        callback: 'calbackname'
        actions_file : "my_trigger_actions.yaml"

I agree callback should be of the form bool func() to simplify both the command-like and trigger-like cases.

cyrush · 2023-07-18T17:27:38Z

src/libs/ascent/runtimes/flow_filters/ascent_runtime_callback_filters.cpp

+//-----------------------------------------------------------------------------
+
+#include <mpi.h>
+


to heal CI issues, need to guard including mpi with:

#ifdef ASCENT_MPI_ENABLED ... #endif

cyrush · 2023-07-18T17:33:18Z

For the callback table:

We can move callback table into the Callback filter (or a combined Command class that implements (Shell Cmds + Callbacks), so it does not need to be supported by the Data Flow Workspace API.

You should be able to move the static table instance and methods right over from Workspace to the Callback (or Command) class, and wire up their use in the main ascent runtime.

Let me know how this sounds and if you have questions.

siramok · 2023-07-18T17:42:21Z

Thanks for the comments, I like the idea of combining them into a single filter. I'll make some progress this week and ping you here when it's ready to be looked at again.

cyrush · 2023-07-18T17:46:43Z

Thanks for the comments, I like the idea of combining them into a single filter. I'll make some progress this week and ping you here when it's ready to be looked at again.

Sounds good!

siramok · 2023-07-19T20:48:52Z

Ok, the PR is ready for re-review! I'm unsure why curl is failing to download conduit in the CI, but that's unrelated to my changes.

Summary:

The Callback and ShellCommand filters were merged into a single Command filter.
Callback registration and execution are now handled in Command - the Data Flow Workspace API changes were reverted.
Registered callbacks must now return bool.
Multi-line shell commands and callbacks are now supported, via the different yaml multi-line syntaxes.

I haven't yet done the work to use these directly as trigger conditions, but think that could be done as a follow-up PR.

Multi-line shell command example:

-
  action: "add_commands"
  commands:
    c1:
      params:
        shell_command: "echo A single-line shell command"
    c2:
      params:
        shell_command: >
          echo A single shell command
          defined on multiple lines
    c3:
      params:
        shell_command: |
          echo The first of a multi-line shell command
          echo The second of a multi-line shell command
    c4:
      params:
        shell_command: "echo The first of a \
          multi-line shell command\necho The \
          second of a multi-line shell command"

Multi-line callback example:

-
  action: "add_commands"
  commands:
    c1:
      params:
        callback: "test_func1"
    c2:
      params:
        callback: |
          test_func1
          test_func2

Mixed example, with mpi behavior controls:

-
  action: "add_commands"
  commands:
    c1:
      params:
        shell_command: |
          echo This is the first shell command
          echo This is the second shell command
        mpi_behavior: "all"
    c2:
      params:
        callback: |
          test_func1
          test_func2
        mpi_behavior: "root"

mvictoras · 2023-07-19T22:24:39Z

Does it make sense to also add parameters to the callbacks?

siramok · 2023-07-20T23:22:19Z

After trying to implement some use cases, I realize that callbacks should probably be allowed to return either void or bool (as opposed to only allowing one or the other). Void for cases where you want to perform an operation (like checkpointing) and bool for cases where you want to compute something and use the result in a trigger condition. It feels a bit odd to add return true; to a function that doesn't compute anything only so that ascent will let it be registered as a callback.

What do you think?

cyrush · 2023-07-21T00:41:12Z

After trying to implement some use cases, I realize that callbacks should probably be allowed to return either void or bool (as opposed to only allowing one or the other). Void for cases where you want to perform an operation (like checkpointing) and bool for cases where you want to compute something and use the result in a trigger condition. It feels a bit odd to add return true; to a function that doesn't compute anything only so that ascent will let it be registered as a callback.

What do you think?

Sounds good, we would just need to support the two signatures and have two maps, and logic to make sure registered name is a unique name across both.

cyrush · 2023-07-21T00:48:48Z

@mvictoras for passing params to callbacks:

We could have a callback that passes a conduit node as a generic mechanism -- but the question of how to select what we put in that Node from Ascent isn't clear.

Another option - if they want info from ascent, we could pass the ascent instance in the callback.

That use case could still be tricky b/c the info details won't be fully complete until all of the current actions are executed. People could also do bad things if they try to execute new actions in the middle of current execution.

cyrush · 2023-08-15T16:37:11Z

Everything looks good.

We should add unit tests that exercise the system command and call back cases.

I did have an idea to simplify overall: Registered callbacks are effectively static and shared across ascent instances.

Should we just make this part of the main interface as static methods?

That would cut out the plumbing and the extra logic for other runtime.

It also helps with the jupyter use cases b/c you don't need an ascent instance, it could be a static method as well (like ascent.about()

If we do this, it's more rework (I am happy take it on), but i wonder if what folks think.

@nicolemarsaglia ideas?

siramok · 2023-08-21T07:52:36Z

Moving callback registration/execution into the main interface as static methods does significantly simplify the PR, but I had trouble exposing execute_callback in Jupyter afterwards. I didn't bother committing that attempt, but feel free to give it a go yourself. Assuming that the Jupyter piece gets resolved, I think the rework should be a positive change.

For testing, I committed a bunch of unit tests for register_callback to get the ball rolling. They don't currently assert anything since register_callback either silently succeeds or throws an ASCENT_ERROR. Do you have advice for what to assert in that scenario?

Finally, I'll work on creating tests that actually execute some simple shell commands and callbacks. I'll also be sure to create (at least) one that demonstrates using a bool callback as a trigger condition.

cyrush · 2023-08-21T18:46:40Z

@siramok thanks for the updates!

The basic strategy for testing callbacks is close to what you have. Use a global var that gets changed when the call back is exec-ed, and then assert we have the changed value.

Also, I am happy to take over the refactoring and get the final steps done, just let me know when you are done with your updates.

… change

siramok · 2023-08-22T21:15:27Z

I couldn't help myself and attempted the refactor again. I had better luck this time around and found the issue that I was having with the Python/Jupyter interface, so I committed the work!

The register_callback API has been moved to the top level (on par with ascent::about) and does not depend on an Ascent instance. An execute_callback API has been created at the top level, also independent of an Ascent instance, moving that logic out of the command filter.

Callback invocation from Jupyter will now require importing ascent explicitly, which is perfectly fine:

import ascent
import conduit
ascent.execute_callback("myCallback", conduit.Node(), conduit.Node())

In addition to getting that refactor done, I added tests for executing callbacks, one for executing a shell command, and one that uses a bool callback as a trigger condition. The tests now reflect the API changes as well.

siramok · 2023-08-22T21:29:37Z

I gave the changes another look and found a few minor things that needed to be cleaned up (basically just imports made unnecessary by the refactor).

Besides that, I feel that everything is good to go. The only notably missing piece I can think of is a Python test for execute_callback. The challenge there being that execute_callback requires a callback to first be registered using register_callback, which itself is not exposed to Python. If you have an idea for how to workaround that, please let me know.

Finally, I see that CI is failing again due to a third party dependency issue. We'll obviously need to get CI passing again, but in the meantime you're good to give it another review @cyrush. :)

Edit: worth considering, I overloaded execute_callback to handle both void and bool callbacks. It works, but I would understand if you'd rather have them be separate and explicit. Let me know what you think.

cyrush

Awesome, the outer level interface was exactly what I was hoping for, nice refactoring!

I changed the target of the branch from develop to a branch in the ascent repo that we can both work on.
I have some minor suggestions but I can tackle those.

I will also explore how we can register python callbacks ...

cyrush · 2023-08-22T22:46:12Z

src/libs/ascent/ascent.hpp

@@ -66,13 +66,31 @@ class ASCENT_API Ascent
    conduit::Node  m_info;
 };

+// Callback maps
+static std::map<std::string, void (*)(conduit::Node &, conduit::Node &)> m_void_callback_map;
+static std::map<std::string, bool (*)(void)> m_bool_callback_map;


we should define these in the cpp file, otherwise they could be multiple defined by users calling ascent.

cyrush · 2023-08-22T22:47:31Z

src/libs/ascent/python/ascent_module/ascent_mpi_python.cpp

+    PyObject *py_output = NULL;
+
+    if (!PyArg_ParseTuple(args,
+                          "sOO",


we can use s|00 to make all the args optional (maybe to support the bool callback case)

cyrush · 2023-08-22T22:47:58Z

src/libs/ascent/python/ascent_module/ascent_python.cpp

+    PyObject *py_output = NULL;
+
+    if (!PyArg_ParseTuple(args,
+                          "sOO",


also s|OO possible here

cyrush · 2023-08-22T22:52:28Z

src/tests/ascent/t_ascent_commands.cpp

+    ascent.open(ascent_opts);
+
+    // An error should be thrown due to not including a name
+    ascent::register_callback("", void_callback_1);


we can wrap this with a google test, EXPECT_THROW to make sure an error is thrown

Suggested change

ascent::register_callback("", void_callback_1);

EXPECT_THROW(ascent::register_callback("", void_callback_1),conduit::Error);

cyrush · 2023-08-22T22:53:19Z

src/tests/ascent/t_ascent_commands.cpp

+    Node actions;
+    std::string msg = "An example of registering a void callback";
+                      " without a callback name.";
+    ASCENT_ACTIONS_DUMP(actions, std::string("register_no_name_void"), msg);


FYI: we don't need to use this marco here b/c we aren't dumping any actions.
We use this to help seed our docs.

cyrush · 2023-08-22T23:03:35Z

@siramok FYI: I invited you to join the ascent repo as a contributor, keep an eye out for github request.

@siramok

* WIP: Shell commands and function callbacks (#1175) Work from @siramok. * Initial callback and shell command implementations * Initial work to combine commands and callbacks * Move callback registration and execution into Command * Catch the case where neither action is defined * Allow defining multiline shell commands and callbacks * Minor formatting tweaks * Initial implementation of triggers with callbacks * Improved organization, more error catching * Make sure that triggers either have a condition or callback * Make void callbacks take conduit node parameters * Initial attempt at exposing callbacks through ascent-jupyter-bridge * Let void callbacks return arbitrary data via conduit nodes * Disallow anonymous callbacks, add some tests * Add tests for shell commands and callbacks * Refactor callback API into the main API, adjusts tests to reflect the change * Minor cleanups that I missed * Fix minor bug with the trigger + callback test * alt download link for zlib * typo * Fix several bugs in the tests * Implement all suggestions --------- Co-authored-by: Andres Sewell <[email protected]> Co-authored-by: Andres Sewell <[email protected]>

Initial callback and shell command implementations

486d7ce

cyrush reviewed Jul 18, 2023

View reviewed changes

siramok and others added 6 commits July 18, 2023 12:21

Merge branch 'Alpine-DAV:develop' into develop

a62462f

Initial work to combine commands and callbacks

43489d5

Move callback registration and execution into Command

75288a2

Catch the case where neither action is defined

1ef9834

Allow defining multiline shell commands and callbacks

c35d61c

Minor formatting tweaks

c816e88

siramok and others added 9 commits July 24, 2023 12:10

Initial implementation of triggers with callbacks

8629395

Merge branch 'Alpine-DAV:develop' into develop

01f06b7

Improved organization, more error catching

9ccfa0f

Make sure that triggers either have a condition or callback

3d1dcf7

Make void callbacks take conduit node parameters

1dc2aeb

Initial attempt at exposing callbacks through ascent-jupyter-bridge

1b005dd

Merge branch 'Alpine-DAV:develop' into develop

30333d6

Let void callbacks return arbitrary data via conduit nodes

0b3c769

Merge branch 'Alpine-DAV:develop' into develop

95815b8

siramok and others added 2 commits August 20, 2023 22:49

Merge branch 'Alpine-DAV:develop' into develop

422b89e

Disallow anonymous callbacks, add some tests

713857b

siramok added 2 commits August 22, 2023 14:16

Add tests for shell commands and callbacks

a286cab

Refactor callback API into the main API, adjusts tests to reflect the…

156ff58

… change

Minor cleanups that I missed

e4f82b2

Fix minor bug with the trigger + callback test

7043d5d

cyrush changed the base branch from develop to task/2023_08_siramok_command_integration August 22, 2023 22:55

cyrush approved these changes Aug 22, 2023

View reviewed changes

cyrush merged commit e741d17 into Alpine-DAV:task/2023_08_siramok_command_integration Aug 22, 2023
4 of 20 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Shell commands and function callbacks #1175

WIP: Shell commands and function callbacks #1175

siramok commented Jul 17, 2023 •

edited

Loading

cyrush commented Jul 18, 2023

cyrush Jul 18, 2023

cyrush commented Jul 18, 2023

siramok commented Jul 18, 2023

cyrush commented Jul 18, 2023

siramok commented Jul 19, 2023

mvictoras commented Jul 19, 2023

siramok commented Jul 20, 2023 •

edited

Loading

cyrush commented Jul 21, 2023

cyrush commented Jul 21, 2023

cyrush commented Aug 15, 2023

siramok commented Aug 21, 2023 •

edited

Loading

cyrush commented Aug 21, 2023

siramok commented Aug 22, 2023 •

edited

Loading

siramok commented Aug 22, 2023 •

edited

Loading

cyrush left a comment

cyrush Aug 22, 2023

cyrush Aug 22, 2023

cyrush Aug 22, 2023

cyrush Aug 22, 2023

cyrush Aug 22, 2023

cyrush commented Aug 22, 2023

		//-----------------------------------------------------------------------------

		#include <mpi.h>

	ascent::register_callback("", void_callback_1);
	EXPECT_THROW(ascent::register_callback("", void_callback_1),conduit::Error);

WIP: Shell commands and function callbacks #1175

WIP: Shell commands and function callbacks #1175

Conversation

siramok commented Jul 17, 2023 • edited Loading

cyrush commented Jul 18, 2023

cyrush Jul 18, 2023

Choose a reason for hiding this comment

cyrush commented Jul 18, 2023

siramok commented Jul 18, 2023

cyrush commented Jul 18, 2023

siramok commented Jul 19, 2023

mvictoras commented Jul 19, 2023

siramok commented Jul 20, 2023 • edited Loading

cyrush commented Jul 21, 2023

cyrush commented Jul 21, 2023

cyrush commented Aug 15, 2023

siramok commented Aug 21, 2023 • edited Loading

cyrush commented Aug 21, 2023

siramok commented Aug 22, 2023 • edited Loading

siramok commented Aug 22, 2023 • edited Loading

cyrush left a comment

Choose a reason for hiding this comment

cyrush Aug 22, 2023

Choose a reason for hiding this comment

cyrush Aug 22, 2023

Choose a reason for hiding this comment

cyrush Aug 22, 2023

Choose a reason for hiding this comment

cyrush Aug 22, 2023

Choose a reason for hiding this comment

cyrush Aug 22, 2023

Choose a reason for hiding this comment

cyrush commented Aug 22, 2023

siramok commented Jul 17, 2023 •

edited

Loading

siramok commented Jul 20, 2023 •

edited

Loading

siramok commented Aug 21, 2023 •

edited

Loading

siramok commented Aug 22, 2023 •

edited

Loading

siramok commented Aug 22, 2023 •

edited

Loading