Stop the dream cycle aborting when memory consolidation returns early (#495)

* Ignore compose agent local config and runtime data

deploy/docker-compose/agent-data/ is the bind-mounted runtime directory for the
compose agent — it holds the agent profile markdown, the long-term memory store,
the knowledge graph, and full conversation transcripts. It was untracked but not
ignored, so `git add -A` would stage the entire local agent state (517 files in a
working setup) into the repository.

Also ignores agent.extra.env, the optional env_file escape hatch for raw .NET
config keys, which sits next to the already-ignored .env and can likewise contain
provider credentials.

Neither belongs in source control: both are per-operator local state.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Stop the dream cycle aborting when memory consolidation returns early

The memory consolidation pass ran inline at the top of DreamAsync, and its two
early exits — fewer than two stored entries, or a null result from the LLM — used
a bare `return`. That returned from the whole dream cycle, not just the pass, so
every later pass was silently skipped.

The worst case is a freshly provisioned agent. With an empty memory store,
consolidation hits the `all.Count < 2` guard on every cycle, so memory mining
never runs and the store can never reach two entries. It is a deadlock: the agent
cannot mine its first memory, and nothing in the logs says why, because the cycle
reports as started but never reports as complete.

Extracted the pass into RunMemoryConsolidationPassAsync, returning its deleted and
saved counts, so its early exits return from the pass and the cycle continues.

Also in this change, both dream-subsystem fixes found alongside it:

- Dream:MemoryConsolidationEnabled (default true, so existing deployments are
  unaffected). Consolidation is the only pass that rewrites stored entries rather
  than only adding or deleting them, which makes it the only place a dream model
  can introduce detail no source entry contained. Turning it off costs duplicate
  merging, importance re-scoring and decay, all of which live inside the pass.

- Dream:ModelTier (default Balanced, matching the previous behaviour). One pass
  was hardcoded to ModelTier.Balanced while the rest of the cycle honoured the
  configured tier. Dream passes are structured extraction, so being able to point
  them at a different tier from the conversational one is useful in its own right.

Tests cover the toggle default, environment-style binding, and independence from
Dream:MemoryMiningEnabled. Full RockBot.Host.Tests suite passes (1080).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: rockfordlhotka

.gitignore

@@ -18,3 +18,7 @@ deploy/values.personal.yaml.live*
 
 # Docker Compose .env files contain API keys — never commit these
 deploy/docker-compose/.env
+# Local-only raw config overrides for the compose agent (optional env_file)
+deploy/docker-compose/agent.extra.env
+# Compose agent runtime data (profile, memory, transcripts) — local only, never commit
+deploy/docker-compose/agent-data/

src/RockBot.Host.Abstractions/DreamOptions.cs

@@ -18,6 +18,18 @@ public sealed class DreamOptions
     /// </summary>
     public string CronSchedule { get; set; } = "0 */12 * * *";
 
+    /// <summary>
+    /// Which LLM tier the dream passes run on. Defaults to <see cref="ModelTier.Balanced"/>,
+    /// matching the previous hardcoded behaviour.
+    /// <para>
+    /// Dream passes are structured-extraction work — read a transcript, return JSON — which is
+    /// a very different job from whatever the agent does conversationally. Pointing this at a
+    /// different tier lets an agent run its conversation on a model chosen for voice while
+    /// consolidating memory on one chosen for instruction-following and reliable JSON.
+    /// </para>
+    /// </summary>
+    public ModelTier ModelTier { get; set; } = ModelTier.Balanced;
+
     /// <summary>
     /// Path to the memory consolidation directive file, relative to <see cref="AgentProfileOptions.BasePath"/>.
     /// </summary>
@@ -65,6 +77,29 @@ public sealed class DreamOptions
     /// </summary>
     public string EpisodeDirectivePath { get; set; } = "episode-dream.md";
 
+    /// <summary>
+    /// Whether the memory consolidation pass is enabled — the pass that merges duplicate
+    /// long-term memory entries, re-scores importance, and prunes ephemeral ones.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// Worth turning off when entry fidelity matters more than tidiness. Consolidation is the
+    /// one pass that <em>rewrites</em> stored memories rather than only adding or deleting
+    /// them, so it is also the only place a weaker or more creative dream model can introduce
+    /// detail that was never in any source entry. Mining, episode and entity extraction all
+    /// derive from the conversation log and can be checked against it; a consolidated entry
+    /// has no such anchor, and any embellishment it picks up is indistinguishable from a real
+    /// fact on the next pass — which then feeds it back in as its own input.
+    /// </para>
+    /// <para>
+    /// With this off, duplicate and near-duplicate entries accumulate instead of being merged.
+    /// Importance re-scoring and the decay pass both stop as well, since both run inside
+    /// consolidation. That is the trade: a larger, redundant store with static importance,
+    /// whose entries are all traceable to something that was actually said.
+    /// </para>
+    /// </remarks>
+    public bool MemoryConsolidationEnabled { get; set; } = true;
+
     /// <summary>Whether the memory mining pass (requires <see cref="IConversationLog"/>) is enabled.</summary>
     public bool MemoryMiningEnabled { get; set; } = true;
 

src/RockBot.Host/DreamService.cs

@@ -496,143 +496,18 @@ private async Task DreamAsync()
         {
             var ct = slot.Token;
 
-            // Log retention runs first and unconditionally — append-only JSONL logs
-            // must be capped even on cycles where there's nothing to consolidate (the
-            // memory-count check below can early-return).
+            // Log retention runs first — append-only JSONL logs must be capped even on
+            // cycles where there is nothing to consolidate.
             await RunLogRetentionPassAsync(ct);
 
-            var all = await _memory.SearchAsync(new MemorySearchCriteria(MaxResults: 1000));
-
-            if (all.Count < 2)
-            {
-                _logger.LogInformation(
-                    "DreamService: only {Count} memory entries — nothing to consolidate; skipping",
-                    all.Count);
-                return;
-            }
-
-            _logger.LogDebug("DreamService: fetched {Count} memory entries for consolidation", all.Count);
-
-            // Apply importance decay to unreferenced entries before consolidation
-            await RunImportanceDecayPassAsync(all);
-
-            // Build user message: numbered list with IDs, categories, tags, content
-            var userMessage = new StringBuilder();
-            userMessage.AppendLine($"The agent currently has {all.Count} memory entries. Consolidate them:");
-            userMessage.AppendLine();
-
-            // Append recent feedback signals so the dream LLM has quality context
-            if (_feedbackStore is not null)
-            {
-                var recentFeedback = await _feedbackStore.QueryRecentAsync(
-                    since: DateTimeOffset.UtcNow.AddDays(-7),
-                    maxResults: 50);
-
-                if (recentFeedback.Count > 0)
-                {
-                    userMessage.AppendLine();
-                    userMessage.AppendLine("Recent feedback signals (last 7 days):");
-                    foreach (var fb in recentFeedback)
-                    {
-                        var detail = string.IsNullOrWhiteSpace(fb.Detail) ? string.Empty : $" (\"{fb.Detail}\")";
-                        userMessage.AppendLine($"- [{fb.SignalType}] session {fb.SessionId}: {fb.Summary}{detail}");
-                    }
-                    userMessage.AppendLine();
-                    _logger.LogDebug("DreamService: injected {Count} feedback signal(s) into dream prompt", recentFeedback.Count);
-                }
-            }
-
-            for (var i = 0; i < all.Count; i++)
-            {
-                var e = all[i];
-                var tags = e.Tags.Count > 0 ? string.Join(", ", e.Tags) : "(none)";
-                var subjectTime = FormatSubjectTimeForPrompt(e.Metadata);
-                userMessage.AppendLine(
-                    $"{i + 1}. [ID:{e.Id}] category={e.Category ?? "uncategorized"} " +
-                    $"importance={e.ImportanceScore:F2} reinforced={e.ReinforcementCount}× " +
-                    $"first={e.CreatedAt:yyyy-MM-dd} last={e.LastSeenAt:yyyy-MM-dd}" +
-                    $"{subjectTime} tags=[{tags}]");
-                userMessage.AppendLine($"   {e.Content}");
-            }
-
-            var result = await InvokeDreamPassAsync<DreamResultDto>(
-                "memory dream",
-                _dreamDirective!,
-                userMessage.ToString(),
-                ct);
-            if (result is null) return;
-
-            var deleted = 0;
-            var saved = 0;
-
-            // Union of explicit toDelete IDs and all sourceIds referenced by saved entries.
-            // This enforces the exhaustive-deletion contract even when the LLM omits some IDs
-            // from toDelete while still listing them in sourceIds.
-            var allToDelete = new HashSet<string>(
-                result.ToDelete ?? [],
-                StringComparer.OrdinalIgnoreCase);
-
-            foreach (var dto in result.ToSave ?? [])
-                foreach (var srcId in dto.SourceIds ?? [])
-                    allToDelete.Add(srcId);
-
-            foreach (var id in allToDelete)
-            {
-                await _memory.DeleteAsync(id);
-                deleted++;
-                _logger.LogDebug("DreamService: deleted entry {Id}", id);
-            }
-
-            // Build source-entry lookup (case-insensitive on ID) for merge arithmetic
-            var byId = new Dictionary<string, MemoryEntry>(StringComparer.OrdinalIgnoreCase);
-            foreach (var e in all)
-                byId[e.Id] = e;
-
-            foreach (var dto in result.ToSave ?? [])
-            {
-                if (string.IsNullOrWhiteSpace(dto.Content))
-                    continue;
-
-                var sourceIds = dto.SourceIds ?? [];
-                var sources = sourceIds
-                    .Where(id => byId.ContainsKey(id))
-                    .Select(id => byId[id])
-                    .ToList();
-
-                // CreatedAt = earliest source (first-seen preserved); UpdatedAt = now (record rewritten)
-                var firstSeen = sources.Count > 0 ? sources.Min(s => s.CreatedAt) : DateTimeOffset.UtcNow;
-
-                // LastSeenAt = most recent source reinforcement; never "now" on dream housekeeping
-                var lastSeen = sources.Count > 0 ? sources.Max(s => s.LastSeenAt) : DateTimeOffset.UtcNow;
-
-                // ReinforcementCount = sum of source counts; singleton merge (1 source) preserves its count
-                var reinforcementCount = sources.Count > 0 ? sources.Sum(s => s.ReinforcementCount) : 1;
-
-                // LLM-provided importance wins; otherwise carry forward max from sources
-                var importance = dto.Importance
-                    ?? (sources.Count > 0 ? sources.Max(s => s.ImportanceScore) : 0.5f);
-
-                var mergedMetadata = MergeSubjectTimeMetadata(sources);
-
-                var entry = new MemoryEntry(
-                    Id: Guid.NewGuid().ToString("N")[..12],
-                    Content: dto.Content.Trim(),
-                    Category: string.IsNullOrWhiteSpace(dto.Category) ? null : dto.Category.Trim(),
-                    Tags: dto.Tags ?? [],
-                    CreatedAt: firstSeen,
-                    UpdatedAt: DateTimeOffset.UtcNow,
-                    Metadata: mergedMetadata,
-                    ImportanceScore: Math.Clamp(importance, 0f, 1f))
-                {
-                    LastSeenAt = lastSeen,
-                    ReinforcementCount = reinforcementCount
-                };
-
-                await _memory.SaveAsync(entry);
-                saved++;
-                _logger.LogDebug("DreamService: saved entry {Id} ({Category}, importance={Importance:F2}, reinforced={Count}×): {Content}",
-                    entry.Id, entry.Category ?? "(none)", entry.ImportanceScore, entry.ReinforcementCount, entry.Content);
-            }
+            // Consolidation is one pass among many and must not gate the rest of the
+            // cycle. It lives in its own method precisely so that its early exits (too
+            // few entries to merge, or a failed LLM call) return from *it* rather than
+            // aborting the cycle. Inlining it here previously deadlocked a fresh agent:
+            // consolidation needs >=2 memory entries, memory mining is what creates
+            // them, and mining runs after consolidation — so an empty store skipped the
+            // cycle at this point and memory could never become non-empty.
+            var (deleted, saved) = await RunMemoryConsolidationPassAsync(ct);
 
             if (_skillStore is not null)
             { ct.ThrowIfCancellationRequested(); await RunSkillGapDetectionPassAsync(ct); }
@@ -697,6 +572,161 @@ private async Task DreamAsync()
         }
     }
 
+    /// <summary>
+    /// Merges and prunes long-term memory entries via the "memory dream" pass.
+    /// Returns the number of entries deleted and saved.
+    /// </summary>
+    /// <remarks>
+    /// Deliberately a separate method rather than inline in <c>DreamAsync</c>: both of
+    /// its early exits — fewer than two entries to merge, and a failed LLM call — used
+    /// to <c>return</c> straight out of the dream cycle, silently skipping every later
+    /// pass including memory mining. That deadlocked a fresh agent, whose memory store
+    /// only becomes non-empty once mining has run.
+    /// </remarks>
+    private async Task<(int Deleted, int Saved)> RunMemoryConsolidationPassAsync(CancellationToken ct)
+    {
+        if (!_options.MemoryConsolidationEnabled)
+        {
+            _logger.LogInformation("DreamService: memory consolidation disabled; skipping");
+            return (0, 0);
+        }
+
+        var all = await _memory.SearchAsync(new MemorySearchCriteria(MaxResults: 1000));
+
+        if (all.Count < 2)
+        {
+            _logger.LogInformation(
+                "DreamService: only {Count} memory entries — nothing to consolidate; skipping consolidation",
+                all.Count);
+            return (0, 0);
+        }
+
+        _logger.LogDebug("DreamService: fetched {Count} memory entries for consolidation", all.Count);
+
+        // Apply importance decay to unreferenced entries before consolidation
+        await RunImportanceDecayPassAsync(all);
+
+        // Build user message: numbered list with IDs, categories, tags, content
+        var userMessage = new StringBuilder();
+        userMessage.AppendLine($"The agent currently has {all.Count} memory entries. Consolidate them:");
+        userMessage.AppendLine();
+
+        // Append recent feedback signals so the dream LLM has quality context
+        if (_feedbackStore is not null)
+        {
+            var recentFeedback = await _feedbackStore.QueryRecentAsync(
+                since: DateTimeOffset.UtcNow.AddDays(-7),
+                maxResults: 50);
+
+            if (recentFeedback.Count > 0)
+            {
+                userMessage.AppendLine();
+                userMessage.AppendLine("Recent feedback signals (last 7 days):");
+                foreach (var fb in recentFeedback)
+                {
+                    var detail = string.IsNullOrWhiteSpace(fb.Detail) ? string.Empty : $" (\"{fb.Detail}\")";
+                    userMessage.AppendLine($"- [{fb.SignalType}] session {fb.SessionId}: {fb.Summary}{detail}");
+                }
+                userMessage.AppendLine();
+                _logger.LogDebug("DreamService: injected {Count} feedback signal(s) into dream prompt", recentFeedback.Count);
+            }
+        }
+
+        for (var i = 0; i < all.Count; i++)
+        {
+            var e = all[i];
+            var tags = e.Tags.Count > 0 ? string.Join(", ", e.Tags) : "(none)";
+            var subjectTime = FormatSubjectTimeForPrompt(e.Metadata);
+            userMessage.AppendLine(
+                $"{i + 1}. [ID:{e.Id}] category={e.Category ?? "uncategorized"} " +
+                $"importance={e.ImportanceScore:F2} reinforced={e.ReinforcementCount}× " +
+                $"first={e.CreatedAt:yyyy-MM-dd} last={e.LastSeenAt:yyyy-MM-dd}" +
+                $"{subjectTime} tags=[{tags}]");
+            userMessage.AppendLine($"   {e.Content}");
+        }
+
+        var result = await InvokeDreamPassAsync<DreamResultDto>(
+            "memory dream",
+            _dreamDirective!,
+            userMessage.ToString(),
+            ct);
+        if (result is null) return (0, 0);
+
+        var deleted = 0;
+        var saved = 0;
+
+        // Union of explicit toDelete IDs and all sourceIds referenced by saved entries.
+        // This enforces the exhaustive-deletion contract even when the LLM omits some IDs
+        // from toDelete while still listing them in sourceIds.
+        var allToDelete = new HashSet<string>(
+            result.ToDelete ?? [],
+            StringComparer.OrdinalIgnoreCase);
+
+        foreach (var dto in result.ToSave ?? [])
+            foreach (var srcId in dto.SourceIds ?? [])
+                allToDelete.Add(srcId);
+
+        foreach (var id in allToDelete)
+        {
+            await _memory.DeleteAsync(id);
+            deleted++;
+            _logger.LogDebug("DreamService: deleted entry {Id}", id);
+        }
+
+        // Build source-entry lookup (case-insensitive on ID) for merge arithmetic
+        var byId = new Dictionary<string, MemoryEntry>(StringComparer.OrdinalIgnoreCase);
+        foreach (var e in all)
+            byId[e.Id] = e;
+
+        foreach (var dto in result.ToSave ?? [])
+        {
+            if (string.IsNullOrWhiteSpace(dto.Content))
+                continue;
+
+            var sourceIds = dto.SourceIds ?? [];
+            var sources = sourceIds
+                .Where(id => byId.ContainsKey(id))
+                .Select(id => byId[id])
+                .ToList();
+
+            // CreatedAt = earliest source (first-seen preserved); UpdatedAt = now (record rewritten)
+            var firstSeen = sources.Count > 0 ? sources.Min(s => s.CreatedAt) : DateTimeOffset.UtcNow;
+
+            // LastSeenAt = most recent source reinforcement; never "now" on dream housekeeping
+            var lastSeen = sources.Count > 0 ? sources.Max(s => s.LastSeenAt) : DateTimeOffset.UtcNow;
+
+            // ReinforcementCount = sum of source counts; singleton merge (1 source) preserves its count
+            var reinforcementCount = sources.Count > 0 ? sources.Sum(s => s.ReinforcementCount) : 1;
+
+            // LLM-provided importance wins; otherwise carry forward max from sources
+            var importance = dto.Importance
+                ?? (sources.Count > 0 ? sources.Max(s => s.ImportanceScore) : 0.5f);
+
+            var mergedMetadata = MergeSubjectTimeMetadata(sources);
+
+            var entry = new MemoryEntry(
+                Id: Guid.NewGuid().ToString("N")[..12],
+                Content: dto.Content.Trim(),
+                Category: string.IsNullOrWhiteSpace(dto.Category) ? null : dto.Category.Trim(),
+                Tags: dto.Tags ?? [],
+                CreatedAt: firstSeen,
+                UpdatedAt: DateTimeOffset.UtcNow,
+                Metadata: mergedMetadata,
+                ImportanceScore: Math.Clamp(importance, 0f, 1f))
+            {
+                LastSeenAt = lastSeen,
+                ReinforcementCount = reinforcementCount
+            };
+
+            await _memory.SaveAsync(entry);
+            saved++;
+            _logger.LogDebug("DreamService: saved entry {Id} ({Category}, importance={Importance:F2}, reinforced={Count}×): {Content}",
+                entry.Id, entry.Category ?? "(none)", entry.ImportanceScore, entry.ReinforcementCount, entry.Content);
+        }
+
+        return (deleted, saved);
+    }
+
     private async Task ConsolidateSkillsAsync(CancellationToken ct)
     {
         var all = await _skillStore!.ListAsync();
@@ -3894,7 +3924,7 @@ private async Task RunPassAsync(string passName, Func<Task> body)
 
         var response = await _llmClient.GetResponseAsync(
             messages,
-            ModelTier.Balanced,
+            _options.ModelTier,
             new ChatOptions { ResponseFormat = ChatResponseFormat.Json },
             ct);