Schema migration system for ZIO Schema 2

.scalafmt.conf

@@ -24,6 +24,9 @@ rewriteTokens = {
 }
 
 fileOverride {
+  "glob:**/scala-2/**" {
+    runner.dialect = scala213
+  }
   "glob:**/scala-3/**" {
     runner.dialect = scala3
   }

docs/reference/migration.md

@@ -0,0 +1,319 @@
+# Schema Migration
+
+Schema migration provides a pure, algebraic system for transforming data between schema versions.
+
+## Overview
+
+The migration system enables:
+- **Type-safe migrations**: Define transformations between typed schemas
+- **Dynamic migrations**: Operate on untyped `DynamicValue` for flexibility
+- **Reversibility**: All migrations can be structurally reversed
+- **Serialization**: Migrations are pure data that can be serialized and stored
+- **Build-time validation**: Structural correctness is verified when you call `build`
+- **Compile-time field tracking** (Scala 3): Verify all fields are handled at compile time
+- **Path-aware errors**: Detailed error messages with exact location information
+
+## Core Types
+
+### Migration[A, B]
+
+A typed migration from schema `A` to schema `B`. Use `MigrationBuilder` to construct one:
+
+```scala
+import zio.blocks.schema._
+import zio.blocks.schema.migration._
+
+// Import serialization schemas when (de)serializing migrations
+import zio.blocks.schema.migration.MigrationSchemas._
+
+@schema case class PersonV1(name: String, age: Int)
+@schema case class PersonV2(fullName: String, age: Int, country: String)
+
+val migration: Migration[PersonV1, PersonV2] =
+  Migration
+    .newBuilder[PersonV1, PersonV2]
+    .renameField(MigrationBuilder.paths.field("name"), MigrationBuilder.paths.field("fullName"))
+    .addField(MigrationBuilder.paths.field("country"), "US")
+    .buildPartial  // skips structural validation
+```
+
+`build` validates that the migration actions produce a structurally correct target schema.
+`buildPartial` skips validation and is useful during development or when validation is too strict.
+
+### DynamicMigration
+
+An untyped, serializable migration that operates directly on `DynamicValue`. Every `Migration[A, B]`
+contains a `DynamicMigration` accessible via `.dynamicMigration`:
+
+```scala
+val dynamicMigration: DynamicMigration = migration.dynamicMigration
+
+import zio.blocks.chunk.Chunk
+
+val oldValue: DynamicValue = DynamicValue.Record(Chunk(
+  "name" -> DynamicValue.Primitive(PrimitiveValue.String("John")),
+  "age"  -> DynamicValue.Primitive(PrimitiveValue.Int(30))
+))
+
+val newValue: Either[MigrationError, DynamicValue] = dynamicMigration(oldValue)
+// Right(Record(Chunk("fullName" -> Primitive(String("John")), "age" -> Primitive(Int(30)), "country" -> Primitive(String("US")))))
+```
+
+### MigrationAction
+
+Individual migration steps are represented as an algebraic data type. Each action is reversible:
+
+| Action | Description |
+|--------|-------------|
+| `AddField` | Add a new field with a default value expression |
+| `DropField` | Remove a field (stores a reverse default) |
+| `RenameField` | Rename a field |
+| `TransformValue` | Transform a field's value using a `DynamicSchemaExpr` |
+| `Mandate` | Make an optional field mandatory (unwrap `Option`) |
+| `Optionalize` | Make a mandatory field optional (wrap in `Option`) |
+| `ChangeType` | Convert between primitive types (e.g., `Int` → `Long`) |
+| `Join` | Combine multiple source fields into one target field |
+| `Split` | Split one source field into multiple target fields |
+| `RenameCase` | Rename a case in a variant/enum |
+| `TransformCase` | Apply nested actions within a specific variant case |
+| `TransformElements` | Transform every element in a sequence |
+| `TransformKeys` | Transform every key in a map |
+| `TransformValues` | Transform every value in a map |
+| `Identity` | No-op action (useful as a placeholder) |
+
+### DynamicSchemaExpr
+
+A serializable expression language for computing values during migration. Expressions are evaluated
+against `DynamicValue` at runtime:
+
+```scala
+// Literal value
+val lit = DynamicSchemaExpr.Literal(
+  DynamicValue.Primitive(PrimitiveValue.Int(42))
+)
+
+// Extract a value by path
+val nameExpr = DynamicSchemaExpr.Path(DynamicOptic.root.field("name"))
+
+// Arithmetic on numeric fields
+val doubled = DynamicSchemaExpr.Arithmetic(
+  nameExpr,
+  DynamicSchemaExpr.Literal(DynamicValue.Primitive(PrimitiveValue.Int(2))),
+  DynamicSchemaExpr.ArithmeticOperator.Multiply
+)
+
+// String operations
+val concat = DynamicSchemaExpr.StringConcat(expr1, expr2)
+val length = DynamicSchemaExpr.StringLength(stringExpr)
+
+// Primitive type coercion (e.g., Int → String)
+val coerced = DynamicSchemaExpr.CoercePrimitive(intExpr, "String")
+```
+
+## MigrationBuilder API
+
+The builder provides a fluent API for constructing migrations. All path arguments are `DynamicOptic`
+values — use `MigrationBuilder.paths` helpers or the type-safe selector syntax below:
+
+```scala
+Migration
+  .newBuilder[OldType, NewType]
+  // Record operations
+  .addField(path, defaultExpr)                            // add a field with a default
+  .dropField(path, defaultForReverse)                     // remove a field
+  .renameField(fromPath, toPath)                          // rename a field
+  .transformField(path, transform, reverseTransform)      // transform a field value
+  .mandateField(path, default)                            // Option[T] → T
+  .optionalizeField(path)                                 // T → Option[T]
+  .changeFieldType(path, converter, reverseConverter)     // change primitive type
+  .joinFields(targetPath, sourcePaths, combiner, splitter)
+  .splitField(sourcePath, targetPaths, splitter, combiner)
+  // Enum/variant operations
+  .renameCaseAt(path, from, to)
+  .transformCaseAt(path, caseName, nestedActions)
+  // Collection operations
+  .transformElements(path, transform, reverseTransform)
+  .transformKeys(path, transform, reverseTransform)
+  .transformValues(path, transform, reverseTransform)
+  // Build
+  .build        // validates structural correctness, then builds
+  .buildPartial // builds without validation
+```
+
+## Type-Safe Selector Syntax (Scala 2 & 3)
+
+For ergonomic, type-checked paths, import the selector syntax. The macro inspects selector
+lambdas like `_.fieldName.nested` and converts them to `DynamicOptic` paths at compile time:
+
+```scala
+import zio.blocks.schema.migration.MigrationBuilderSyntax._
+
+val migration: Migration[PersonV1, PersonV2] =
+  Migration
+    .newBuilder[PersonV1, PersonV2]
+    .renameField(_.name, _.fullName)
+    .addField(_.country, "US")
+    .buildPartial
+```
+
+Selector lambdas support optic-like projections for nested structures:
+
+| Projection | Meaning |
+|------------|---------|
+| `_.field` | Select a record field |
+| `_.field.nested` | Select a nested field |
+| `_.each` | Traverse into sequence elements |
+| `_.eachKey` | Traverse into map keys |
+| `_.eachValue` | Traverse into map values |
+
+## Path Helpers
+
+When you don't need compile-time type checking, use the `paths` object:
+
+```scala
+import MigrationBuilder.paths
+
+paths.field("name")               // Single field
+paths.field("address", "street")  // Nested field (address.street)
+paths.elements                    // Sequence elements
+paths.mapKeys                     // Map keys
+paths.mapValues                   // Map values
+```
+
+## Reversibility
+
+Every migration action stores enough information to be reversed. Call `.reverse` to get
+a `Migration[B, A]`:
+
+```scala
+val forward: Migration[PersonV1, PersonV2] = ...
+val backward: Migration[PersonV2, PersonV1] = forward.reverse
+
+// Reverse of addField is dropField, reverse of rename is rename back, etc.
+```
+
+> **Note:** Reverse transforms are resolved best-effort at build time. For `TransformValue`
+> and `ChangeType`, provide explicit reverse expressions for reliable round-tripping.
+
+## Composition
+
+Migrations compose sequentially with `++` or `.andThen`:
+
+```scala
+val v1ToV2: Migration[V1, V2] = ...
+val v2ToV3: Migration[V2, V3] = ...
+
+val v1ToV3: Migration[V1, V3] = v1ToV2 ++ v2ToV3
+// or equivalently:
+val v1ToV3: Migration[V1, V3] = v1ToV2.andThen(v2ToV3)
+```
+
+## Error Handling
+
+All migration operations return `Either[MigrationError, DynamicValue]`. Errors are accumulated
+(not short-circuiting) and carry path information:
+
+```scala
+migration.applyDynamic(value) match {
+  case Right(newValue) => // success
+  case Left(migrationError) =>
+    migrationError.errors.foreach { error =>
+      println(s"At ${error.path}: ${error.message}")
+    }
+}
+```
+
+Error types include:
+
+| Error | Description |
+|-------|-------------|
+| `FieldNotFound` | Required field missing from source |
+| `FieldAlreadyExists` | Field already exists when adding |
+| `NotARecord` | Expected a record, found something else |
+| `NotAVariant` | Expected a variant, found something else |
+| `NotASequence` | Expected a sequence, found something else |
+| `NotAMap` | Expected a map, found something else |
+| `CaseNotFound` | Variant case not found |
+| `TypeConversionFailed` | Primitive type conversion failed |
+| `ExprEvalFailed` | Expression evaluation failed |
+| `PathNavigationFailed` | Cannot navigate the specified path |
+| `DefaultValueMissing` | Default value not resolved for a required field |
+| `IndexOutOfBounds` | Sequence index out of range |
+| `KeyNotFound` | Map key not found |
+| `NumericOverflow` | Arithmetic overflow |
+| `ActionFailed` | General action failure |
+
+## Best Practices
+
+1. **Use `build` in production** to catch structural mismatches early; use `buildPartial` during prototyping
+2. **Provide explicit reverse expressions** for `transformField` and `changeFieldType` to ensure reliable round-tripping
+3. **Compose small migrations** rather than writing one large migration — this improves readability and testability
+4. **Test both directions** — apply forward, then reverse, and verify the round-trip
+5. **Serialize migrations** alongside schema versions for audit trails and reproducibility
+6. **Use `checkedBuilder` (Scala 3)** to catch missing fields at compile time instead of runtime
+
+## Compile-Time Field Tracking (Scala 3 Only)
+
+For Scala 3 projects, `TrackedMigrationBuilder` adds **type-level field name tracking** that validates
+migration completeness at compile time. If you forget to handle a source field or provide a target
+field, you get a compile error — not a runtime failure.
+
+### Quick Start
+
+```scala
+import zio.blocks.schema._
+import zio.blocks.schema.migration._
+import zio.blocks.schema.migration.MigrationBuilderSyntax._
+
+case class PersonV1(name: String, age: Int, ssn: String) derives Schema
+case class PersonV2(fullName: String, age: Int, email: String) derives Schema
+
+val migration = MigrationBuilderSyntax.checkedBuilder[PersonV1, PersonV2]
+  .renameField(_.name, _.fullName)   // handles source "name", provides target "fullName"
+  .dropField(_.ssn)                  // handles source "ssn"
+  .addField(_.email, "unknown")      // provides target "email"
+  .build                             // ✅ compiles — all fields accounted for
+```
+
+If you forget `.dropField(_.ssn)`, the compiler produces:
+
+```
+Migration is incomplete.
+
+Unhandled source fields: ssn
+
+Hints: Use .dropField or .renameField for: ssn
+
+Source fields: name, age, ssn
+Target fields: fullName, age, email
+Auto-mapped: age
+```
+
+### Entry Points
+
+| Entry point | Description |
+|-------------|-------------|
+| `MigrationBuilderSyntax.checkedBuilder[A, B]` | Create a tracked builder directly |
+| `Migration.newBuilder[A, B].tracked` | Convert an existing builder to tracked |
+
+### Build Modes
+
+| Method | Compile-time check | Runtime validation |
+|--------|-------------------|-------------------|
+| `.build` | ✅ `MigrationComplete` required | ✅ `MigrationValidator` runs |
+| `.buildChecked` | ✅ `MigrationComplete` required | ❌ Skipped |
+| `.buildPartial` | ❌ Not required | ❌ Skipped |
+
+### How It Works
+
+The builder carries two Tuple type parameters:
+- **`SH` (SourceHandled)**: Accumulates source field names handled via `dropField` / `renameField`
+- **`TP` (TargetProvided)**: Accumulates target field names provided via `addField` / `renameField`
+
+Fields with the same name in source and target are **auto-mapped** and don't need explicit handling.
+
+At `.build` time, the compiler summons `MigrationComplete[A, B, SH, TP]` evidence, which uses a
+macro to verify: `sourceFields - autoMapped ⊆ SH` and `targetFields - autoMapped ⊆ TP`.
+
+> **Note:** This feature requires Scala 3 (transparent inline macros + Tuple types). Scala 2
+> projects should use the standard `MigrationBuilder` with runtime validation via `.build`.

docs/reference/schema.md

@@ -221,9 +221,10 @@ Having the schema for `DynamicValue` allows seamless encoding/decoding between `
 
 ```scala
 import zio.blocks.schema._
+import zio.blocks.chunk.Chunk
 
 // Records have unquoted keys
-val record = DynamicValue.Record(Vector(
+val record = DynamicValue.Record(Chunk(
   "name" -> DynamicValue.Primitive(PrimitiveValue.String("Alice")),
   "age" -> DynamicValue.Primitive(PrimitiveValue.Int(30))
 ))
@@ -234,7 +235,7 @@ println(record)
 // }
 
 // Maps have quoted string keys
-val map = DynamicValue.Map(Vector(
+val map = DynamicValue.Map(Chunk(
   DynamicValue.Primitive(PrimitiveValue.String("key")) ->
     DynamicValue.Primitive(PrimitiveValue.String("value"))
 ))
@@ -244,7 +245,7 @@ println(map)
 // }
 
 // Variants use @ metadata
-val variant = DynamicValue.Variant("Some", DynamicValue.Record(Vector(
+val variant = DynamicValue.Variant("Some", DynamicValue.Record(Chunk(
   "value" -> DynamicValue.Primitive(PrimitiveValue.Int(42))
 )))
 println(variant)

docs/reference/validation.md

@@ -320,8 +320,10 @@ object Person {
 // Create a DynamicSchema for validation
 val dynamicSchema: DynamicSchema = Schema[Person].toDynamicSchema
 
+import zio.blocks.chunk.Chunk
+
 // Create a DynamicValue to validate
-val value = DynamicValue.Record(Vector(
+val value = DynamicValue.Record(Chunk(
   "name" -> DynamicValue.Primitive(PrimitiveValue.String("Alice")),
   "age" -> DynamicValue.Primitive(PrimitiveValue.Int(30))
 ))
@@ -345,7 +347,7 @@ val dynamicSchema: DynamicSchema = Schema[Person].toDynamicSchema
 val validatingSchema: Schema[DynamicValue] = dynamicSchema.toSchema
 
 // Now any decoding through this schema will validate structure
-val invalidValue = DynamicValue.Record(Vector(
+val invalidValue = DynamicValue.Record(Chunk(
   "name" -> DynamicValue.Primitive(PrimitiveValue.Int(42))  // wrong type!
 ))