fix: honor Event(internal=True) and give internal events from outside a step the external contract (#668)

---------
Signed-off-by: Fernando Macedo <fgmacedo@gmail.com>
Co-authored-by: Vladimir Babin <vovababin@gmail.com>

Author: fgmacedo

AGENTS.md

@@ -39,7 +39,14 @@ The engine follows the SCXML run-to-completion (RTC) model with two processing l
 ### Event queues
 
 - `send()` → **external queue** (processed after current macrostep ends).
-- `raise_()` → **internal queue** (processed within the current macrostep, before external events).
+- `raise_()`, `send(internal=True)`, `Event(internal=True)` → **internal queue** (processed
+  within the current macrostep, before external events), but only when raised while the
+  machine runs a step in the same thread or task (`_running_engine` in `engines/base.py`, a
+  `ContextVar` so it is right for threads and coroutines alike). From outside a step there
+  is no macrostep to join: the trigger goes to the external queue, so its caller gets the
+  result, the exception or `TransitionNotAllowed`.
+- `delay` and `internal` together raise `InvalidDefinition`: a step takes no time.
+- Each external event is followed by its macrostep before the next external event is taken.
 
 ### Error handling (`catch_errors_as_events`)
 

docs/async.md

@@ -155,10 +155,11 @@ resolves each event's future as it processes the queue. Callers that didn't
 acquire the lock simply `await` their future.
 
 ```{note}
-Futures are only created for **external** events sent from outside the
-processing loop. Events triggered from within callbacks (via `send()` or
-`raise_()`) follow the {ref}`run-to-completion <rtc-model>` model — they
-are enqueued and processed within the current macrostep.
+Futures are created for every event sent from outside the processing loop,
+including one declared `internal=True`, which goes to the external queue
+when there is no macrostep to join. Events triggered from within callbacks
+(via `send()` or `raise_()`) follow the {ref}`run-to-completion <rtc-model>`
+model: they are enqueued and processed by the running loop.
 ```
 
 If an exception occurs during processing (with `catch_errors_as_events=False`),

docs/processing_model.md

@@ -76,17 +76,51 @@ After the macrostep completes, the engine picks the next event from the
 
 ### Event queues
 
-The engine maintains two separate FIFO queues:
+The engine maintains two separate queues, each ordered by due time. Events due at the same
+moment run in the order they were sent.
 
 | Queue | How to enqueue | When processed |
 |---|---|---|
-| **Internal** | {func}`raise_() <StateMachine.raise_>` or `send(..., internal=True)` | Within the current macrostep |
-| **External** | {func}`send() <StateMachine.send>` | After the current macrostep ends |
+| **Internal** | {func}`raise_() <StateMachine.raise_>`, `send(..., internal=True)`, or an event declared with `Event(..., internal=True)`, from a callback | Within the current macrostep |
+| **External** | {func}`send() <StateMachine.send>`, and any event triggered from outside a callback | After the current macrostep ends |
 
 This distinction matters when you trigger events from inside callbacks.
 Using `raise_()` ensures the event is handled as part of the current
 processing cycle, while `send()` defers it to after the machine reaches
-a stable configuration.
+a stable configuration. An event declared `internal=True` behaves like
+`raise_()` without spelling it at each call:
+
+```py
+>>> from statemachine import Event, State, StateChart
+
+>>> class Muster(StateChart):
+...     camp = State(initial=True)
+...     riding = State(final=True)
+...     besieged = State(final=True)
+...
+...     summon = camp.to.itself(internal=True, on="call")
+...     ride = Event(camp.to(riding), internal=True)
+...     siege = camp.to(besieged)
+...
+...     def call(self):
+...         self.send("siege")  # after this macrostep
+...         self.ride()         # within this macrostep, so first
+
+>>> sm = Muster()
+>>> sm.send("summon")
+>>> sm.riding.is_active
+True
+
+```
+
+The internal queue belongs to the macrostep in progress. Triggered from
+outside a callback, there is no macrostep to join: an internal event goes
+to the external queue, and the call returns its result like `send()`.
+
+A delay cannot be internal, since a macrostep takes no time. Declaring or
+sending an event with both `delay` and `internal=True`, or calling
+`raise_()` with a `delay`, raises {class}`~statemachine.exceptions.InvalidDefinition`.
+Use `send(..., delay=...)` to deliver an event later.
 
 ```{seealso}
 See {ref}`sending-events` for examples of `send()` vs `raise_()`.

docs/releases/3.2.2.md

@@ -117,8 +117,7 @@ Delays are now measured on `time.monotonic()`, which no setting of the wall cloc
 
 `Event(dark.to(lit), delay=50)` rebuilt the event without its `delay`, so
 `BeaconsOfGondor.light.delay` was `0` and the event fired immediately instead of being queued.
-The same happened to `internal`. Both are now preserved, although `internal` still has no
-effect at trigger time:
+The same happened to `internal`. Each is now preserved:
 
 ```py
 >>> class BeaconsOfGondor(StateChart):
@@ -132,6 +131,62 @@ effect at trigger time:
 
 ```
 
+### `internal` declared on an `Event` ignored at trigger time
+
+`go = Event(a.to(b), internal=True)` kept `internal` on the declaration, but neither `sm.go()`
+nor `sm.send("go")` carried it to the queue, so the event always went to the external queue. Both
+now honor the declaration: triggered from a callback, the event runs within the current
+macrostep, before any pending external event.
+
+Honoring it exposed how the internal queue treated a trigger that no running step had raised.
+On the async engine, the first `await sm.go()` on a fresh machine was lost before the initial
+state was entered. The caller got `None` instead of the callback result, and no
+`TransitionNotAllowed` on a `StateMachine`. The internal queue now receives only what the
+machine raises while it runs a step. Triggered from outside, an internal event goes to the
+external queue and returns its result, as `send()` does:
+
+```py
+>>> class Beacons(StateChart):
+...     dark = State(initial=True)
+...     lit = State(final=True)
+...
+...     light = Event(dark.to(lit, on="kindle"), internal=True)
+...
+...     def kindle(self):
+...         return "kindled"
+
+>>> Beacons().light()
+'kindled'
+
+```
+
+A delay never reached the internal queue either: `raise_("go", delay=50)` fired at once. A
+macrostep takes no time, and SCXML forbids `delay` on a `<send>` to `#_internal`, so a delayed
+internal event is now an invalid definition, whether declared, sent, raised, or loaded
+from an SCXML, JSON or YAML `<send>` to `#_internal` (also rejected by the JSON Schema):
+
+```py
+>>> class DelayedBeacons(StateChart):
+...     dark = State(initial=True)
+...     lit = State(final=True)
+...
+...     light = Event(dark.to(lit), delay=50, internal=True)
+Traceback (most recent call last):
+...
+InvalidDefinition: Event 'light' cannot be both delayed and internal: ...
+
+```
+
+Drop `internal` to deliver the event later on the external queue, or drop `delay` to run it
+within the current macrostep. An explicit `send(..., internal=False)` overrides the declaration, so
+`send("light", delay=50, internal=False)` delivers a declared-internal event later.
+
+Contributed by [@chiliec](https://github.com/chiliec).
+
+[#655](https://github.com/fgmacedo/python-statemachine/issues/655),
+[#663](https://github.com/fgmacedo/python-statemachine/pull/663),
+[#668](https://github.com/fgmacedo/python-statemachine/pull/668).
+
 ### `is_active` on a state nested in a parallel region
 
 Reaching a child state through its parent (`sm.door.shut`) returned the class-level `State`,
@@ -167,3 +222,38 @@ Reported by [@MartinBernstorff](https://github.com/MartinBernstorff).
 
 [#651](https://github.com/fgmacedo/python-statemachine/issues/651),
 [#652](https://github.com/fgmacedo/python-statemachine/pull/652).
+
+### Each external event gets its own macrostep
+
+With several external events already queued, the engine took them one after the other and
+only then ran the internal events and eventless transitions they had enabled. An internal
+event raised while handling the first external event ran after the second, breaking the
+run-to-completion order the {ref}`processing model <macrostep-microstep>` describes:
+
+```py
+>>> class Beacons(StateChart):
+...     dark = State(initial=True)
+...     lit = State(final=True)
+...     besieged = State(final=True)
+...
+...     signal = dark.to.itself(internal=True, on="relay")
+...     light = dark.to(lit)
+...     siege = dark.to(besieged)
+...
+...     def relay(self):
+...         self.raise_("light")
+...         self.send("siege")
+
+>>> sm = Beacons()
+>>> sm.send("signal")
+>>> sm.lit.is_active
+True
+
+```
+
+Each external event is now followed by its macrostep before the next one is taken, on both
+engines. Eventless transitions are now checked after every external event instead of once per
+batch, so a burst of queued external events costs one guard evaluation each, and an event
+queued behind one that reaches a final state is no longer processed.
+
+[#668](https://github.com/fgmacedo/python-statemachine/pull/668).

docs/statechart.md

@@ -88,6 +88,24 @@ True
 
 ```
 
+Called from outside a callback, `raise_()` has no macrostep to join, so it
+behaves like `send()` and returns the result:
+
+```py
+>>> class Greeter(StateChart):
+...     idle = State(initial=True)
+...     done = State(final=True)
+...
+...     greet = idle.to(done, on="say_hello")
+...
+...     def say_hello(self):
+...         return "hello"
+
+>>> Greeter().raise_("greet")
+'hello'
+
+```
+
 Both methods accept arbitrary keyword arguments that are forwarded to all
 callbacks via {ref}`dependency injection <dependency-injection>`.
 
@@ -115,6 +133,10 @@ sm.cancel_event("my_timeout")
 
 On the async engine, a coroutine awaiting the `send()` of a cancelled event gets `None`.
 
+Only `send()` takes a delay. An internal event runs within the current macrostep,
+which takes no time, so `raise_(..., delay=...)` and `send(..., delay=..., internal=True)`
+raise {class}`~statemachine.exceptions.InvalidDefinition`.
+
 ```{note}
 The delay is **blocking** in the sync engine — the processing loop sleeps until the
 delay elapses, holding the calling thread. In the async engine, delays are scheduled