diff --git a/aws-lambda-durable-functions-power/POWER.md b/aws-lambda-durable-functions-power/POWER.md index 578e286..20403ee 100644 --- a/aws-lambda-durable-functions-power/POWER.md +++ b/aws-lambda-durable-functions-power/POWER.md @@ -79,7 +79,7 @@ Load the appropriate reference file based on what the user is working on: - **Getting started**, **basic setup**, **example**, **ESLint**, or **Jest setup** -> see [getting-started.md](steering/getting-started.md) - **Understanding replay model**, **determinism**, or **non-deterministic errors** -> see [replay-model-rules.md](steering/replay-model-rules.md) -- **Creating steps**, **atomic operations**, or **retry logic** -> see [step-operations.md](steering/step-operations.md) +- **Creating steps**, **step operations**, or **retry logic** -> see [step-operations.md](steering/step-operations.md) - **Waiting**, **delays**, **callbacks**, **external systems**, or **polling** -> see [wait-operations.md](steering/wait-operations.md) - **Parallel execution**, **map operations**, **batch processing**, or **concurrency** -> see [concurrent-operations.md](steering/concurrent-operations.md) - **Error handling**, **retry strategies**, **saga pattern**, or **compensating transactions** -> see [error-handling.md](steering/error-handling.md) @@ -117,10 +117,11 @@ def handler(event: dict, context: DurableContext) -> dict: ### Critical Rules -1. **All non-deterministic code MUST be in steps** (Date.now, Math.random, API calls) -2. **Cannot nest durable operations** - use `runInChildContext` to group operations -3. **Closure mutations are lost on replay** - return values from steps -4. **Side effects outside steps repeat** - use `context.logger` (replay-aware) +1. **All non-deterministic code outside durable operations MUST be moved into durable operations** (`context.step`, `waitForCallback`, `waitForCondition`, `parallel`/`map` branches) +2. **Durable operation bodies are not guaranteed to be atomic** - prefer stable identity and idempotent behavior for external side effects; for non-idempotent steps, consider at-most-once-per-retry semantics with zero retries +3. **Cannot nest durable operations** - use `runInChildContext` to group operations +4. **Closure mutations are lost on replay** - return values from steps +5. **Side effects outside durable operations repeat** - prefer `context.logger`; custom loggers may duplicate on replay ### Python API Differences @@ -163,10 +164,11 @@ See here: https://docs.aws.amazon.com/lambda/latest/dg/durable-security.html When writing or reviewing durable function code, ALWAYS check for these replay model violations: -1. **Non-deterministic code outside steps**: `Date.now()`, `Math.random()`, UUID generation, API calls, database queries must all be inside steps -2. **Nested durable operations in step functions**: Cannot call `context.step()`, `context.wait()`, or `context.invoke()` inside a step function — use `context.runInChildContext()` instead -3. **Closure mutations that won't persist**: Variables mutated inside steps are NOT preserved across replays — return values from steps instead -4. **Side effects outside steps that repeat on replay**: Use `context.logger` for logging (it is replay-aware and deduplicates automatically) +1. **Non-deterministic code outside durable operations**: `Date.now()`, `Math.random()`, UUID generation, API calls, database queries must all be inside durable operations +2. **Non-atomic durable operation bodies**: Functions passed to `context.step()`, `waitForCallback()`, `waitForCondition()`, and `parallel()`/`map()` branches may be re-attempted before persistence is fully committed — prefer stable identity and idempotent external effects; for non-idempotent steps, use at-most-once-per-retry semantics with zero retries when duplicate execution is unacceptable +3. **Nested durable operations in step functions**: Cannot call `context.step()`, `context.wait()`, or `context.invoke()` inside a step function — use `context.runInChildContext()` instead +4. **Closure mutations that won't persist**: Variables mutated inside steps are NOT preserved across replays — return values from steps instead +5. **Side effects outside durable operations that repeat on replay**: Prefer `context.logger` because it is replay-aware and deduplicates automatically; custom loggers are allowed but may emit duplicates unless `context.logger` is configured to wrap them When implementing or modifying tests for durable functions, ALWAYS verify: diff --git a/aws-lambda-durable-functions-power/README.md b/aws-lambda-durable-functions-power/README.md index d634613..2603167 100644 --- a/aws-lambda-durable-functions-power/README.md +++ b/aws-lambda-durable-functions-power/README.md @@ -5,7 +5,7 @@ A Kiro power for building resilient, long-running multi-step applications and AI ## Overview - **Replay Model Guidance** - Critical rules to avoid non-deterministic bugs -- **Step Operations** - Atomic operations with retry strategies +- **Step Operations** - Step patterns with retry strategies - **Wait Operations** - Delays, callbacks, and polling patterns - **Concurrent Operations** - Map and parallel execution with concurrency control - **Error Handling** - Retry strategies, saga pattern, and compensating transactions @@ -36,7 +36,7 @@ Kiro will load the appropriate steering files and guide you through: ### Critical Concepts - **Replay Model** - How Lambda durable functions execute and replay -- **Determinism Rules** - What must be inside steps vs outside +- **Determinism Rules** - What must be inside durable operations vs outside - **Qualified ARNs** - Why versions/aliases are required - **Checkpoint Strategy** - When and how state is persisted @@ -83,4 +83,4 @@ When you mention these keywords, Kiro will automatically load this power: ## License -This project is licensed under the Apache-2.0 License. \ No newline at end of file +This project is licensed under the Apache-2.0 License. diff --git a/aws-lambda-durable-functions-power/steering/advanced-patterns.md b/aws-lambda-durable-functions-power/steering/advanced-patterns.md index c0983ee..f34d164 100644 --- a/aws-lambda-durable-functions-power/steering/advanced-patterns.md +++ b/aws-lambda-durable-functions-power/steering/advanced-patterns.md @@ -105,28 +105,30 @@ def handler(event: dict, context: DurableContext) -> str: ```typescript import { StepSemantics } from '@aws/durable-execution-sdk-js'; -// AtMostOncePerRetry (DEFAULT) - For idempotent operations -// Step executes at most once per retry attempt -// If step fails partway through, it won't re-execute the same attempt +// AtLeastOncePerRetry (DEFAULT) - For idempotent operations +// Step executes at least once per retry attempt +// If checkpointing fails after success, the step may re-execute on replay await context.step( 'update-database', async () => { // This is idempotent - safe to retry return await updateUserRecord(userId, data); }, - { semantics: StepSemantics.AtMostOncePerRetry } + { semantics: StepSemantics.AtLeastOncePerRetry } ); -// AtLeastOncePerRetry - For operations that can execute multiple times -// Step may execute multiple times per retry attempt -// Use when idempotency is handled externally +// AtMostOncePerRetry - For non-idempotent operations +// Step executes at most once per retry attempt +// Disable retries as well when duplicate execution is unacceptable await context.step( 'send-notification', async () => { - // External system handles deduplication return await sendEmail(email, message); }, - { semantics: StepSemantics.AtLeastOncePerRetry } + { + semantics: StepSemantics.AtMostOncePerRetry, + retryStrategy: () => ({ shouldRetry: false }) + } ); ``` @@ -134,8 +136,8 @@ await context.step( | Semantic | Use When | Example Operations | | ----------------------- | ----------------------------- | ------------------------------------------------- | -| **AtMostOncePerRetry** | Operation is idempotent | Database updates, API calls with idempotency keys | -| **AtLeastOncePerRetry** | External deduplication exists | Queuing systems, event streams | +| **AtLeastOncePerRetry** | Operation is idempotent | Database updates, API calls with idempotency keys | +| **AtMostOncePerRetry** | Duplicate execution is unacceptable | Payments, one-time notifications, non-idempotent downstream calls | ## Completion Policies - Interaction and Combination diff --git a/aws-lambda-durable-functions-power/steering/concurrent-operations.md b/aws-lambda-durable-functions-power/steering/concurrent-operations.md index 4df17b9..fcd8f81 100644 --- a/aws-lambda-durable-functions-power/steering/concurrent-operations.md +++ b/aws-lambda-durable-functions-power/steering/concurrent-operations.md @@ -293,6 +293,10 @@ const results = await context.map( ## Advanced Patterns +### Replay Safety in Branches + +Functions passed to `context.parallel(...)` and `context.map(...)` are durable operation bodies. Treat branch and item functions the same way you treat step and wait bodies: they may be re-attempted before progress is fully persisted, so any external side effect they trigger must use stable identity and idempotent behavior. Derive identifiers from durable inputs such as item IDs, indexes, or prior durable state instead of `Date.now()`, randomness, or fresh UUIDs created inside the branch. See [replay-model-rules.md](replay-model-rules.md). + ### Map with Callbacks **TypeScript:** @@ -416,3 +420,4 @@ const results = await context.map( 6. **Monitor concurrency limits** to avoid overwhelming systems 7. **Use child contexts** for complex per-item workflows 8. **Implement circuit breakers** for external service calls +9. **Use stable identity inside branches** when starting or addressing external work diff --git a/aws-lambda-durable-functions-power/steering/getting-started.md b/aws-lambda-durable-functions-power/steering/getting-started.md index fa14c90..e77109d 100644 --- a/aws-lambda-durable-functions-power/steering/getting-started.md +++ b/aws-lambda-durable-functions-power/steering/getting-started.md @@ -252,7 +252,7 @@ my-durable-function/ ## ESLint Plugin Setup -Install the ESLint plugin to catch common durable function mistakes at development time: +For TypeScript durable-function projects, use the ESLint plugin to catch common mistakes at development time: ```bash npm install --save-dev @aws/durable-execution-sdk-js-eslint-plugin @@ -304,6 +304,8 @@ export default [ - Incorrect usage of durable context outside handler - Common replay model violations +Use the plugin by default in new TypeScript projects. It is a strong static guardrail, not a runtime guarantee, so equivalent enforcement is acceptable if your team already has it. + ## Jest Configuration **jest.config.js:** @@ -339,7 +341,7 @@ Add `aws-durable-execution-sdk-python-testing` to your dev/test dependencies in 1. **Write handler** with durable operations 2. **Test locally** with `LocalDurableTestRunner` -3. **Validate replay rules** (no non-deterministic code outside steps) +3. **Validate replay rules** (determinism outside durable operations; stable identity and idempotent side effects inside durable operation bodies) 4. **Deploy** with qualified ARN (version or alias) 5. **Monitor** execution state and logs @@ -347,13 +349,13 @@ Add `aws-durable-execution-sdk-python-testing` to your dev/test dependencies in 1. **Write handler** with `@durable_execution` decorator 2. **Test locally** with `DurableFunctionTestRunner` and pytest -3. **Validate replay rules** (no non-deterministic code outside steps) +3. **Validate replay rules** (determinism outside durable operations; stable identity and idempotent side effects inside durable operation bodies) 4. **Deploy** with qualified ARN (version or alias) 5. **Monitor** execution state and logs ## Key Concepts -- **Steps**: Atomic operations with automatic retry and checkpointing +- **Steps**: Persisted operations with automatic retry and checkpointing - **Waits**: Suspend execution without compute charges (up to 1 year) - **Child Contexts**: Group multiple durable operations - **Callbacks**: Wait for external systems to respond @@ -368,13 +370,13 @@ When starting a new durable function project: - [ ] Install dependencies (`@aws/durable-execution-sdk-js`, testing & eslint packages) - [ ] Create `jest.config.js` with ts-jest preset - [ ] Configure `tsconfig.json` with proper module resolution -- [ ] Set up ESLint with durable execution plugin +- [ ] Set up ESLint with durable execution plugin (strongly recommended default for TypeScript) - [ ] Create handler with `withDurableExecution` wrapper - [ ] Write tests using `LocalDurableTestRunner` - [ ] Use `skipTime: true` for fast test execution - [ ] Verify TypeScript compilation: `npx tsc --noEmit` - [ ] Run tests to confirm setup: `npm test` -- [ ] Review replay model rules (no non-deterministic code outside steps) +- [ ] Review replay model rules (determinism outside durable operations; stable identity and idempotent side effects inside durable operation bodies) ### Python @@ -384,7 +386,7 @@ When starting a new durable function project: - [ ] Define step functions with `@durable_step` decorator - [ ] Write tests using `DurableFunctionTestRunner` class - [ ] Run tests: `pytest` -- [ ] Review replay model rules (no non-deterministic code outside steps) +- [ ] Review replay model rules (determinism outside durable operations; stable identity and idempotent side effects inside durable operation bodies) ## Error Scenarios diff --git a/aws-lambda-durable-functions-power/steering/replay-model-rules.md b/aws-lambda-durable-functions-power/steering/replay-model-rules.md index dc3a6ee..cf0838c 100644 --- a/aws-lambda-durable-functions-power/steering/replay-model-rules.md +++ b/aws-lambda-durable-functions-power/steering/replay-model-rules.md @@ -21,11 +21,11 @@ await context.wait({ seconds: 60 }); // Line 3: W const result = await context.step('process', async () => process(data)); // Line 5: Executes after wait ``` -## Rule 1: Deterministic Code Outside Steps +## Rule 1: Deterministic Code Outside Durable Operations -**ALL code outside steps MUST produce the same result on every replay.** +**ALL code outside durable operations MUST produce the same result on every replay.** -### ❌ WRONG - Non-Deterministic Outside Steps +### ❌ WRONG - Non-Deterministic Outside Durable Operations **TypeScript:** @@ -51,7 +51,7 @@ now = datetime.now() # Different datetime each time context.step(lambda _: save_data({"id": id}), name='save') ``` -### ✅ CORRECT - Non-Deterministic Inside Steps +### ✅ CORRECT - Non-Deterministic Inside Durable Operations **TypeScript:** @@ -75,7 +75,7 @@ now = context.step(lambda _: datetime.now(), name='get-date') context.step(lambda _: save_data({"id": id}), name='save') ``` -### Must Be In Steps +### Must Be In Durable Operations - `Date.now()`, `new Date()`, `time.time()`, `datetime.now()` - `Math.random()`, `random.random()` @@ -86,7 +86,76 @@ context.step(lambda _: save_data({"id": id}), name='save') - Environment variable reads (if they can change) - Any external system interaction -## Rule 2: No Nested Durable Operations +Durable operations include `context.step(...)`, `waitForCallback(...)`, `waitForCondition(...)`, and branch/item functions passed to `context.parallel(...)` and `context.map(...)`. + +## Rule 2: Durable Operation Bodies Are Not Guaranteed To Be Atomic + +**Functions passed to durable context APIs must assume the operation is not guaranteed to be atomic with respect to external side effects, and may be re-attempted before the durable runtime has fully recorded the result.** + +This rule applies to: + +- `context.step(...)` +- `waitForCallback(...)` submitters +- `waitForCondition(...)` check functions +- Branch/item functions used by `context.parallel(...)` and `context.map(...)` + +### What This Means + +- Non-deterministic computation inside a durable operation body is acceptable because the result can be checkpointed +- External side effects started from that body should still be safe under re-attempt whenever possible +- If the side effect needs an identifier for idempotency, derive it from durable inputs/state or generate it once from durable state and reuse it +- If a **step** cannot be made idempotent and duplicate execution is unacceptable, use `StepSemantics.AtMostOncePerRetry` (TypeScript) or `StepSemantics.AT_MOST_ONCE_PER_RETRY` (Python) with retries disabled so the behavior is effectively zero-or-once rather than more than once + +### ❌ WRONG - Unstable External Identity Inside Durable Operation Body + +**TypeScript:** + +```typescript +await context.step('start-export', async () => { + const jobId = `export-${Date.now()}`; + await exportClient.start({ jobId, orderId }); +}); +``` + +**Python:** + +```python +context.step( + lambda _: export_client.start({ + 'job_id': f'export-{time.time()}', + 'order_id': order_id + }), + name='start-export' +) +``` + +### ✅ CORRECT - Stable Identity Derived From Durable State + +**TypeScript:** + +```typescript +const jobId = `export-${orderId}`; + +await context.step('start-export', async () => { + await exportClient.start({ jobId, orderId }); +}); +``` + +**Python:** + +```python +job_id = f'export-{order_id}' + +context.step( + lambda _: export_client.start({ + 'job_id': job_id, + 'order_id': order_id + }), + name='start-export' +) +``` + +## Rule 3: No Nested Durable Operations **You CANNOT call durable operations inside a step function.** @@ -141,7 +210,7 @@ def process_child(child_ctx: DurableContext): context.run_in_child_context(func=process_child, name='process') ``` -## Rule 3: Closure Mutations Are Lost +## Rule 4: Closure Mutations Are Lost **Variables mutated inside steps are NOT preserved across replays.** @@ -188,9 +257,9 @@ counter = context.step(lambda _: counter + 1, name='increment') print(counter) # Correct value ``` -## Rule 4: Side Effects Outside Steps Repeat +## Rule 5: Side Effects Outside Durable Operations Repeat -**Side effects outside steps happen on EVERY replay.** +**Side effects outside durable operations happen on EVERY replay.** ### ❌ WRONG - Repeated Side Effects @@ -214,7 +283,7 @@ update_database(data) # Updates multiple times! context.step(lambda _: process(), name='process') ``` -### ✅ CORRECT - Side Effects In Steps +### ✅ CORRECT - Replay-Aware Logging And Checkpointed Side Effects **TypeScript:** @@ -239,6 +308,8 @@ context.step(process()) `context.logger` is replay-aware and safe to use anywhere. It automatically deduplicates logs across replays. +Custom loggers are still allowed. If you use a non-replay-aware logger outside durable operations, expect duplicate log entries on replay. If you want to keep an existing logging interface, configure `context.logger` to wrap that existing logger inside the durable handler. + ## Common Pitfalls ### Pitfall 1: Reading Environment Variables @@ -286,15 +357,38 @@ if (shouldTakePathA) { } ``` +### Pitfall 4: Assuming Durable Operation Bodies Are Atomic + +```typescript +// ❌ WRONG +await context.waitForCallback( + 'wait-payment', + async (callbackId) => { + const requestId = `payment-${Date.now()}`; + await paymentProvider.createPayment({ requestId, callbackId }); + } +); + +// ✅ CORRECT +const requestId = `payment-${orderId}`; +await context.waitForCallback( + 'wait-payment', + async (callbackId) => { + await paymentProvider.createPayment({ requestId, callbackId }); + } +); +``` + ## Debugging Replay Issues If you see inconsistent behavior: -1. **Check for non-deterministic code outside steps** -2. **Verify no nested durable operations** -3. **Look for closure mutations** -4. **Search for side effects outside steps** -5. **Use `context.logger` to trace execution flow** +1. **Check for non-deterministic code outside durable operations** +2. **Check durable operation bodies for non-atomic external side effects** +3. **Verify no nested durable operations** +4. **Look for closure mutations** +5. **Search for side effects outside durable operations** +6. **Use `context.logger` to trace execution flow** ## Testing Replay Behavior diff --git a/aws-lambda-durable-functions-power/steering/step-operations.md b/aws-lambda-durable-functions-power/steering/step-operations.md index 0b3f62d..e7e2d0d 100644 --- a/aws-lambda-durable-functions-power/steering/step-operations.md +++ b/aws-lambda-durable-functions-power/steering/step-operations.md @@ -1,6 +1,6 @@ # Step Operations -Steps are atomic operations with automatic retry and state persistence. +Steps are persisted operations with automatic retry and state persistence. Keep each step focused on one logical unit of work, but assume external side effects are not guaranteed to be atomic. ## Basic Step Patterns @@ -53,6 +53,42 @@ const result = await context.step('fetch-user', async () => { **Best Practice:** Always name steps for easier debugging and testing. +## Replay Safety for External Side Effects + +Functions passed to `context.step(...)` may be re-attempted before the durable runtime has fully recorded the result. Non-deterministic computation inside the step body is fine. For external side effects, prefer stable identity and idempotent behavior. If that is not possible and duplicate execution is unacceptable, use `StepSemantics.AtMostOncePerRetry` (TypeScript) or `StepSemantics.AT_MOST_ONCE_PER_RETRY` (Python) with retries disabled. See [replay-model-rules.md](replay-model-rules.md). + +**TypeScript:** + +```typescript +const exportJobId = `export-${orderId}`; + +await context.step('start-export', async () => { + await exportClient.start({ + jobId: exportJobId, + orderId, + }); +}); +``` + +**Python:** + +```python +export_job_id = f'export-{order_id}' + +def start_export(_): + export_client.start({ + 'job_id': export_job_id, + 'order_id': order_id + }) + +context.step( + start_export, + name='start-export' +) +``` + +If you need a fresh identifier, generate it once from durable state and reuse it rather than minting a new one inside the step body with wall-clock time, randomness, or a fresh UUID. + ## Retry Configuration ### Exponential Backoff @@ -178,42 +214,64 @@ retry_config = RetryStrategyConfig( ## Step Semantics -### AT_LEAST_ONCE (Default) +### AtLeastOncePerRetry (Default) -Step executes at least once, may execute multiple times on failure/retry. +Step executes at least once per retry attempt and is the default. If a step succeeds but checkpointing fails, it may re-execute on replay. This does not make external side effects atomic. **TypeScript:** ```typescript +import { StepSemantics } from '@aws/durable-execution-sdk-js'; + const result = await context.step( 'idempotent-operation', async () => idempotentAPI(), - { semantics: 'AT_LEAST_ONCE' } + { semantics: StepSemantics.AtLeastOncePerRetry } ); ``` -### AT_MOST_ONCE +**Python:** + +```python +from aws_durable_execution_sdk_python.config import StepConfig, StepSemantics + +result = context.step( + idempotent_operation(), + config=StepConfig(step_semantics=StepSemantics.AT_LEAST_ONCE_PER_RETRY) +) +``` + +### AtMostOncePerRetry -Step executes at most once, never retries. Use for non-idempotent operations. +Step executes at most once per retry attempt. Pair it with a retry strategy that disables retries to get effectively zero-or-once behavior: the step may not complete successfully, but it will not be re-executed by retries. **TypeScript:** ```typescript +import { StepSemantics } from '@aws/durable-execution-sdk-js'; + const result = await context.step( 'charge-payment', async () => chargeCard(amount), - { semantics: 'AT_MOST_ONCE' } + { + semantics: StepSemantics.AtMostOncePerRetry, + retryStrategy: () => ({ shouldRetry: false }) + } ); ``` **Python:** ```python -from aws_durable_execution_sdk_python.config import StepSemantics +from aws_durable_execution_sdk_python.config import StepConfig, StepSemantics +from aws_durable_execution_sdk_python.retries import RetryDecision result = context.step( charge_card(amount), - config=StepConfig(step_semantics=StepSemantics.AT_MOST_ONCE_PER_RETRY) + config=StepConfig( + step_semantics=StepSemantics.AT_MOST_ONCE_PER_RETRY, + retry_strategy=lambda error, attempt: RetryDecision(should_retry=False) + ) ) ``` @@ -266,7 +324,7 @@ user = context.step( ### Use Steps For: -- Single atomic operations +- Single logical operations - API calls - Database queries - Data transformations @@ -339,9 +397,11 @@ except Exception as error: ## Best Practices 1. **Always name steps** for debugging and testing -2. **Keep steps atomic** - one logical operation per step -3. **Make steps idempotent** when possible +2. **Keep steps focused** - one logical operation per step +3. **Prefer idempotent step design** when possible 4. **Use appropriate retry strategies** based on operation type 5. **Handle errors explicitly** - don't let them propagate unexpectedly 6. **Use custom serialization** for complex types -7. **Choose correct semantics** (AT_LEAST_ONCE vs AT_MOST_ONCE) +7. **Choose correct semantics** (`AtLeastOncePerRetry` vs `AtMostOncePerRetry`) +8. **Use stable identity for external work** - derive identifiers from durable inputs/state, not `Date.now()`, randomness, or fresh UUIDs created inside the step body +9. **Use `AtMostOncePerRetry` with zero retries for non-idempotent steps** when duplicate execution is unacceptable and you can accept zero-or-once behavior diff --git a/aws-lambda-durable-functions-power/steering/wait-operations.md b/aws-lambda-durable-functions-power/steering/wait-operations.md index 664c192..f499c0c 100644 --- a/aws-lambda-durable-functions-power/steering/wait-operations.md +++ b/aws-lambda-durable-functions-power/steering/wait-operations.md @@ -38,6 +38,8 @@ context.wait(duration=Duration.from_seconds(60), name='rate-limit-delay') Wait for external systems to respond (human approval, webhook, async job): +The submitter function passed to `waitForCallback(...)` and the check function passed to `waitForCondition(...)` are durable operation bodies. They are not guaranteed to be atomic with respect to external side effects, so if they start or address external work, use stable identity and idempotent behavior. See [replay-model-rules.md](replay-model-rules.md). + **TypeScript:** ```typescript @@ -284,11 +286,13 @@ export const handler = withDurableExecution(async (event, context: DurableContex const order = await context.step('create-order', async () => createOrder(event) ); + const paymentRequestId = `payment-${order.id}`; const payment = await context.waitForCallback( 'wait-payment', async (callbackId) => { await paymentProvider.createPayment({ + requestId: paymentRequestId, orderId: order.id, amount: order.total, webhookUrl: `${webhookUrl}?callback=${callbackId}` @@ -348,6 +352,7 @@ export const handler = withDurableExecution(async (event, context: DurableContex 6. **Keep check functions lightweight** in waitForCondition 7. **Store callback IDs securely** when sending to external systems 8. **Validate callback payloads** before processing +9. **Use stable logical identifiers** for external work started by submitter/check functions ## Error Handling