Skip to content

Commit a53996c

Browse files
committed
docs(uts): document direct-sandbox tier and per-module test layout
Reflect the new test-source structure in the UTS guide, website, and the uts-to-kotlin skill: - Add the direct-sandbox (`integration/standard/<module>/`) tier alongside the existing unit and proxy tiers, and document that every tier is now organised by module (`realtime`, `liveobjects`, …). - Update §2 tier table, §4.2 directory tree + mental model, §7.3 SandboxApp (shared by both integration kinds), and §12 run commands in README.md, and mirror all of it in index.html (tags verified balanced, sections intact). - Generalise the skill's spec→test path mapping to `<module>`, add a direct-sandbox row, and split integration specs into fault-injecting (proxy) vs happy-path (direct sandbox) flows. - Correct stale "both tiers" wording now that there are three tiers.
1 parent aa0504e commit a53996c

3 files changed

Lines changed: 102 additions & 52 deletions

File tree

‎.claude/skills/uts-to-kotlin/SKILL.md‎

Lines changed: 46 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -59,17 +59,26 @@ Read the file at `$ARGUMENTS`. Identify:
5959

6060
## Step 2 — Determine output path and package
6161

62-
Map the spec path to a test path:
62+
Map the spec path to a test path. **Tests are organised tier-first** (`unit/` vs `integration/standard/`
63+
vs `integration/proxy/`), then **by module** (`realtime`, `rest`, `liveobjects`, …), all under the
64+
`io.ably.lib.uts` package. (`…/uts/` below is shorthand for `uts/src/test/kotlin/io/ably/lib/uts/`.)
6365

64-
| Spec location | Test location |
65-
|---|---|
66-
| `.../uts/test/rest/unit/<name>.md` | `uts/src/test/kotlin/io/ably/lib/rest/unit/<Name>Test.kt` |
67-
| `.../uts/test/realtime/unit/<sub>/<name>.md` | `uts/src/test/kotlin/io/ably/lib/realtime/unit/<sub>/<Name>Test.kt` |
68-
| `.../uts/test/realtime/integration/<sub>/<name>.md` | `uts/src/test/kotlin/io/ably/lib/realtime/integration/<sub>/<Name>Test.kt` |
66+
| Spec location | Test location | Package |
67+
|---|---|---|
68+
| `.../<module>/unit/<…>/<name>.md` | `…/uts/unit/<module>/<Name>Test.kt` | `io.ably.lib.uts.unit.<module>` |
69+
| `.../<module>/integration/<…>/<name>.md` (direct sandbox) | `…/uts/integration/standard/<module>/<Name>Test.kt` | `io.ably.lib.uts.integration.standard.<module>` |
70+
| `.../<module>/integration/<…>/<name>.md` (proxy) | `…/uts/integration/proxy/<module>/<Name>Test.kt` | `io.ably.lib.uts.integration.proxy.<module>` |
71+
72+
`<module>` is the SDK area — `realtime`, `rest`, `liveobjects`, … (existing folders: `unit/realtime/`,
73+
`unit/liveobjects/`). The spec's own `<sub>` grouping (e.g. `connection/`) is **not** carried into a
74+
sub-package — tests sit directly under the tier/module folder (e.g. `connection_recovery_test.md` →
75+
`…/uts/unit/realtime/ConnectionRecoveryTest.kt`).
6976

7077
Class name: take the file name, strip `_test` suffix, convert `snake_case` → `PascalCase`, append `Test`.
7178

72-
**Integration specs that drive traffic through the programmable proxy** (they reference `create_proxy_session()`, proxy rules, or `uts/test/realtime/integration/helpers/proxy.md`) follow a different translation flow — see the **Proxy integration tests** section at the end of this skill instead of the unit-test rules below.
79+
**Integration specs come in two kinds:**
80+
- Those that **inject faults** (reference `create_proxy_session()`, proxy `rules`, `trigger_action`, `get_log`, or `uts/test/realtime/integration/helpers/proxy.md`) are **proxy** tests under `integration/proxy/<module>/` — follow the **Proxy integration tests** section at the end of this skill instead of the unit-test rules below.
81+
- Those that exercise only **happy-path interop** against the real sandbox (no fault injection) are **direct sandbox** tests under `integration/standard/<module>/`. They use `SandboxApp` alone (no `ProxySession`), connecting straight to `ProxyManager.sandboxRealtimeHost` / `sandboxRestHost`. *(No example exists yet — model the suite setup/teardown on the proxy section but drop the `ProxySession`/`connectThroughProxy` wiring.)*
7382

7483
Example: `connection_state_machine_test.md` → `ConnectionStateMachineTest`
7584

@@ -79,7 +88,13 @@ Package: derived from the output path under `kotlin/`.
7988

8089
## Step 3 — Read infrastructure files
8190

82-
Read ALL of files in `uts/src/test/kotlin/io/ably/lib/uts/infra` before generating any code (you need exact method signatures).
91+
Infrastructure is split by tier under `uts/src/test/kotlin/io/ably/lib/uts/infra/`:
92+
93+
- `infra/Utils.kt` — shared async helpers (`awaitState`, `awaitChannelState`, `pollUntil`), package `io.ably.lib.uts.infra`.
94+
- `infra/unit/` — unit-test mocks/factories (`ClientFactories.kt`, `MockWebSocket.kt`, `MockHttpClient.kt`, `FakeClock.kt`, `MockEvent.kt`, the `PendingConnection`/`PendingRequest` pairs, and `Utils.kt` with the `ConnectionDetails { }` builder), package `io.ably.lib.uts.infra.unit`.
95+
- `infra/integration/` + `infra/integration/proxy/` — proxy/sandbox helpers (`SandboxApp.kt`, `ProxyManager.kt`, `ProxySession.kt`) — see the **Proxy integration tests** section.
96+
97+
For a **unit** test, read all files under `infra/unit/` plus `infra/Utils.kt` before generating any code (you need exact method signatures).
8398

8499
## Step 4 — Generate the Kotlin test file
85100

@@ -266,16 +281,14 @@ fun `RTN4a - description of what is being tested`() = runTest {
266281
### File template
267282

268283
```kotlin
269-
package io.ably.lib.<category>.unit[.<subcategory>]
284+
package io.ably.lib.uts.unit.realtime // io.ably.lib.uts.unit.<module> — realtime, rest, liveobjects, …
270285

271-
import io.ably.lib.TestRealtimeClient // or TestRestClient
272-
import io.ably.lib.awaitChannelState // if testing channels
273-
import io.ably.lib.awaitState
274-
import io.ably.lib.realtime.ChannelState // if testing channels
286+
import io.ably.lib.uts.infra.unit.* // TestRealtimeClient/TestRestClient, MockWebSocket, MockHttpClient, FakeClock, CONNECTED_MESSAGE, ConnectionDetails { } builder
287+
import io.ably.lib.uts.infra.awaitState
288+
import io.ably.lib.uts.infra.awaitChannelState // if testing channels
289+
import io.ably.lib.uts.infra.pollUntil // if polling on a predicate
290+
import io.ably.lib.realtime.ChannelState // if testing channels
275291
import io.ably.lib.realtime.ConnectionState
276-
import io.ably.lib.test.mock.FakeClock // if using fake timers
277-
import io.ably.lib.test.mock.MockWebSocket // or MockHttpClient
278-
import io.ably.lib.types.ConnectionDetails
279292
import io.ably.lib.types.ErrorInfo
280293
import io.ably.lib.types.ProtocolMessage
281294
import kotlinx.coroutines.launch
@@ -334,10 +347,18 @@ Fix any compilation errors and recompile until clean. Common issues:
334347

335348
## Step 6 — Run tests
336349

350+
Use the per-tier task that matches what you generated (both are registered in `uts/build.gradle.kts`):
351+
337352
```bash
338-
./gradlew :uts:test --tests "<package>.<ClassName>"
353+
# Unit test (io.ably.lib.uts.unit.*)
354+
./gradlew :uts:runUtsUnitTests --tests "io.ably.lib.uts.unit.realtime.<ClassName>"
355+
356+
# Proxy integration test (io.ably.lib.uts.integration.*)
357+
./gradlew :uts:runUtsIntegrationTests --tests "io.ably.lib.uts.integration.proxy.realtime.<ClassName>"
339358
```
340359

360+
(`./gradlew :uts:test` still runs all tiers — unit, standard, and proxy.)
361+
341362
Handle test failures using this decision tree (see [reference doc](https://github.com/ably/specification/blob/main/uts/docs/writing-derived-tests.md) for full detail):
342363

343364
```
@@ -379,7 +400,7 @@ assertEquals(40160, error.errorInfo.code)
379400

380401
### Deviations file
381402

382-
Append to `uts/src/test/kotlin/io/ably/lib/deviations.md`. Each entry needs:
403+
Append to `uts/src/test/kotlin/io/ably/lib/uts/deviations.md`. Each entry needs:
383404
1. The spec point (e.g. `RSA4c2`)
384405
2. What the spec says
385406
3. What the SDK does
@@ -417,7 +438,7 @@ For each test case, verify:
417438

418439
For any place where the generated test diverges from the spec pseudocode (adapted assertion, env-gated skip, or omitted step):
419440
- [ ] A `// DEVIATION:` comment explains why
420-
- [ ] The deviation is recorded in `uts/src/test/kotlin/io/ably/lib/deviations.md`
441+
- [ ] The deviation is recorded in `uts/src/test/kotlin/io/ably/lib/uts/deviations.md`
421442

422443
If you find gaps during this review, fix them and re-run Steps 5–6 before finishing.
423444

@@ -439,11 +460,13 @@ Recognise them by: a reference to `create_proxy_session()`, proxy `rules`, `trig
439460

440461
### Infrastructure
441462

442-
Three helpers live in `uts/src/test/kotlin/io/ably/lib/test/helper/`. **Read them before translating a proxy spec** — they hold the exact method signatures.
463+
Three helpers live under `uts/src/test/kotlin/io/ably/lib/uts/infra/integration/`. **Read them before translating a proxy spec** — they hold the exact method signatures.
464+
465+
- **`ProxyManager`** (`infra/integration/proxy/ProxyManager.kt`, package `io.ably.lib.uts.infra.integration.proxy`) — downloads/starts the shared `uts-proxy` process and exposes the sandbox host. Call `ProxyManager.ensureProxy()` once per suite in setup. `ProxyManager.sandboxRealtimeHost` / `sandboxRestHost` are the upstream sandbox hosts (the default target of every session).
466+
- **`ProxySession`** (`infra/integration/proxy/ProxySession.kt`, same package) — one programmable session wrapping the proxy control API; also defines the `connectThroughProxy` extension and the rule-builder helpers.
467+
- **`SandboxApp`** (`infra/integration/SandboxApp.kt`, package `io.ably.lib.uts.infra.integration`) — provisions/deletes a sandbox test app from the shared `test-app-setup.json` in ably-common. `SandboxApp.create()` returns a `SandboxApp` with `appId`, `defaultKey`, and `keys` (`defaultKey` is a full-capability `appId.keyId:keySecret`); `app.delete()` tears it down. Provision in suite setup, delete in teardown.
443468

444-
- **`ProxyManager`** — downloads/starts the shared `uts-proxy` process and exposes the sandbox host. Call `ProxyManager.ensureProxy()` once per suite in setup. `ProxyManager.sandboxRealtimeHost` / `sandboxRestHost` are the upstream sandbox hosts (the default target of every session).
445-
- **`ProxySession`** — one programmable session wrapping the proxy control API.
446-
- **`SandboxApp`** — provisions/deletes a sandbox test app from the shared `test-app-setup.json` in ably-common. `SandboxApp.create()` returns a `SandboxApp` with `appId`, `defaultKey`, and `keys` (`defaultKey` is a full-capability `appId.keyId:keySecret`); `app.delete()` tears it down. Provision in suite setup, delete in teardown.
469+
Import these into a proxy test from their packages, e.g. `io.ably.lib.uts.infra.integration.SandboxApp`, `io.ably.lib.uts.infra.integration.proxy.{ProxyManager, ProxySession, connectThroughProxy}`, plus `io.ably.lib.uts.infra.unit.TestRealtimeClient` and `io.ably.lib.uts.infra.{awaitState, pollUntil}`.
447470

448471
`ensureProxy()`, the `ProxySession` methods, and the `SandboxApp` methods are all **`suspend`** functions. Per-test bodies use `runTest { }`; JUnit5 `@BeforeAll`/`@AfterAll` (with `@TestInstance(Lifecycle.PER_CLASS)`) wrap their suspend calls in `runBlocking { }`.
449472

‎uts/README.md‎

Lines changed: 36 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -81,9 +81,13 @@ tests you asked about sit in two different tiers.
8181
| Tier | Transport | Backend | Purpose | Example in this repo |
8282
|------|-----------|---------|---------|----------------------|
8383
| **Unit** | **Mocked** (`MockWebSocket`, `MockHttpClient`) | none | Client-side logic: state machines, request formation, response parsing, timer behaviour. Fast & deterministic. | `unit/realtime/ConnectionRecoveryTest.kt` |
84-
| **Direct sandbox integration** | Real network | Real Ably sandbox | Happy-path interop: connect, publish, subscribe. No fault injection. | *(not in the two you asked about)* |
84+
| **Direct sandbox integration** | Real network | Real Ably sandbox | Happy-path interop: connect, publish, subscribe. No fault injection. | `integration/standard/<module>/` *(tier exists; no tests yet)* |
8585
| **Proxy integration** | Real network **through a programmable proxy** | Real Ably sandbox | Fault behaviour: dropped connections, injected errors, timeouts, re-auth. | `integration/proxy/realtime/AuthReauthTest.kt` |
8686

87+
Each tier folder is further organised **by module** (`realtime`, `liveobjects`, …): `unit/<module>/`,
88+
`integration/standard/<module>/`, and `integration/proxy/<module>/`. So a feature's tests sit together
89+
by SDK area — the two example tests live at `unit/realtime/` and `integration/proxy/realtime/`.
90+
8791
Key principles (from [`integration-testing.md`](https://github.com/ably/specification/blob/main/uts/docs/integration-testing.md)):
8892

8993
- **Integration tests do not replace unit tests.** A spec point covered by a proxy test should
@@ -203,8 +207,10 @@ Takeaways:
203207
### 4.2 Directory layout
204208

205209
Everything lives under the `io.ably.lib.uts` package, split cleanly into **infrastructure** (`infra/`,
206-
no `@Test`s) and the **tests** themselves (`unit/`, `integration/`), each mirroring the unit /
207-
integration tiers:
210+
no `@Test`s) and the **tests** themselves. Tests are organised **by tier, then by module**: `unit/` for
211+
mocked-transport tests, and `integration/` for real-backend tests — the latter splitting again into
212+
`standard/` (direct sandbox, happy-path) and `proxy/` (sandbox through the fault-injecting proxy). Under
213+
each, a per-module folder (`realtime`, `liveobjects`, …) holds the actual test classes:
208214

209215
```
210216
uts/src/test/kotlin/io/ably/lib/uts/
@@ -233,20 +239,27 @@ uts/src/test/kotlin/io/ably/lib/uts/
233239
│ ├── ProxyManager.kt # downloads/launches the uts-proxy binary
234240
│ └── ProxySession.kt # proxy session: rules, actions, log + connectThroughProxy
235241
│
236-
├── unit/ # ── UNIT TESTS (mock transport) ──
237-
│ └── realtime/
238-
│ └── ConnectionRecoveryTest.kt # ← the UNIT test (RTN16*)
242+
├── unit/ # ── UNIT TESTS (mock transport) ── · per module
243+
│ ├── realtime/
244+
│ │ └── ConnectionRecoveryTest.kt # ← the UNIT test (RTN16*)
245+
│ └── liveobjects/ # (further modules as coverage grows)
239246
│
240-
└── integration/ # ── INTEGRATION TESTS (real backend) ──
241-
└── proxy/
242-
└── realtime/
243-
└── AuthReauthTest.kt # ← the PROXY test (RTN22, RTC8a)
247+
└── integration/ # ── INTEGRATION TESTS (real backend) ── · per module
248+
├── standard/ # direct sandbox: happy-path, no fault injection
249+
│ ├── realtime/
250+
│ └── liveobjects/
251+
└── proxy/ # sandbox through the fault-injecting uts-proxy
252+
├── realtime/
253+
│ └── AuthReauthTest.kt # ← the PROXY test (RTN22, RTC8a)
254+
└── liveobjects/
244255
```
245256

246-
The mental model: **`infra/unit/` powers the unit tests, `infra/integration/` powers the integration
247-
tests, and `infra/Utils.kt` serves both.** The `unit/` ↔ `infra/unit/` and `integration/` ↔
248-
`infra/integration/` pairing is what the `runUtsUnitTests` / `runUtsIntegrationTests` Gradle tasks
249-
key off (§12).
257+
The mental model: **`infra/unit/` powers the unit tests, `infra/integration/` powers both integration
258+
kinds (`standard` + `proxy`), and `infra/Utils.kt` serves all of them.** Every tier is sub-divided **by
259+
module** (`realtime`, `liveobjects`, …) so a feature's tests sit together regardless of SDK area. The
260+
top-level `unit/` ↔ `infra/unit/` and `integration/` ↔ `infra/integration/` pairing is what the
261+
`runUtsUnitTests` / `runUtsIntegrationTests` Gradle tasks key off (§12) — `runUtsIntegrationTests`
262+
covers **both** `integration/standard/` and `integration/proxy/`.
250263

251264
---
252265

@@ -469,12 +482,17 @@ independent of the fault rules):
469482
- The Ktor client retries only **idempotent GETs** (never re-POSTs `/apps`, to avoid duplicate
470483
apps).
471484

485+
`SandboxApp` is the shared backbone of *both* integration kinds: **proxy** tests pair it with a
486+
`ProxySession`, while **direct sandbox** tests (`integration/standard/<module>/`) use it alone —
487+
connecting straight to `ProxyManager.sandboxRealtimeHost` / `sandboxRestHost` with no proxy and no
488+
fault rules, for happy-path interop.
489+
472490
---
473491

474492
## 8. Shared Async Helpers
475493

476-
`Utils.kt` provides the coroutine glue both tiers rely on. All three run on a **single-thread real
477-
dispatcher** so their timeouts measure **wall-clock** time (not the virtual time of
494+
`Utils.kt` provides the coroutine glue every tier relies on (unit, direct sandbox, and proxy). All
495+
three run on a **single-thread real dispatcher** so their timeouts measure **wall-clock** time (not the virtual time of
478496
`kotlinx.coroutines.test`). The two state-waiters (`awaitState`/`awaitChannelState`) register their
479497
listener *before* checking current state, to avoid a check-then-register race; `pollUntil` has no
480498
listener — it re-evaluates the predicate every `interval` until it holds or the timeout fires.
@@ -663,7 +681,8 @@ mirror `runLiveObjectsUnitTests` / `runLiveObjectsIntegrationTests` in the `live
663681
# Unit tests only — io.ably.lib.uts.unit.* (fast, no network). This is the PR gate.
664682
./gradlew :uts:runUtsUnitTests
665683

666-
# Integration tests only — io.ably.lib.uts.integration.* (real sandbox; downloads/launches the proxy).
684+
# Integration tests only — io.ably.lib.uts.integration.* (real sandbox; covers both
685+
# integration/standard/ and integration/proxy/ — proxy tests also download/launch the uts-proxy).
667686
./gradlew :uts:runUtsIntegrationTests
668687

669688
# Everything (the default Test task still runs both):

0 commit comments

Comments
 (0)