If local .db files are gitignored then looms will be hard or impossible to test

[Gromski]

Diagnostic Information

Property	Value
CLI Version	0.5.3
Node.js Version	v24.5.0
OS	darwin
OS Version	25.1.0
Architecture	x64
Capabilities	web
Claude CLI Version	2.0.73 (Claude Code)

I'm running a db locally with the .db files stored within the project structure. However they're gitignored, which means that they don't transfer to the loom when fixing an issue. As a result I can't test the changes in the loom without merging the fix first

---

Enhancement Request Analysis

Questions for Reporter

Question	Answer
What type of database are you using (SQLite, PostgreSQL via Docker, etc.) and how is it configured in your main branch?	@Gromski, please edit this issue to answer these questions.
Are your .db files explicitly listed in .gitignore or matched by a pattern (e.g., *.db)? Could you share the relevant .gitignore entries?
Would you prefer iloom to automatically copy database files to looms, or would you like a configuration option to control this behavior?
Are there any security concerns with copying potentially sensitive database content to looms that we should consider?

Problem Summary
When using local database files (.db, .sqlite, etc.) that are gitignored, these files don't get copied to iloom workspaces (looms), preventing proper testing of database-dependent features without first merging changes back to the main branch. This breaks the isolated testing environment that iloom provides.

User Impact
Developers working with local file-based databases (SQLite, etc.) cannot fully test their changes in isolated looms, forcing them to either merge untested code or manually copy database files to each workspace, defeating iloom's automation benefits.

Enhancement Goal
Enable iloom to handle gitignored database files intelligently, allowing developers to test database-dependent changes in isolated workspaces while maintaining security and not accidentally committing sensitive data.

Next Steps

• Reporter to answer questions above for full context
• Investigate adding a copyGitIgnoredPatterns configuration option in .iloom/settings.json
• Consider implementing automatic detection of common database file patterns with opt-in copying

---

📋 Complete Context & Details (click to expand)

Current Behavior
When il start creates a new loom workspace, it:

1. Creates a Git worktree (which only includes tracked files)
2. Copies environment files (.env, .env.local, etc.) from the main branch
3. Sets up database branching for cloud databases (Neon)
4. Does NOT copy gitignored database files (.db, .sqlite, etc.)

This means local file-based databases remain in the main branch directory and aren't accessible from the loom workspace.

Proposed Solution
Add configuration support for copying specific gitignored files/patterns to looms:

1. Configuration in .iloom/settings.json:

{
  "workspace": {
    "copyGitIgnoredPatterns": [
      "*.db",
      "*.sqlite",
      "data/*.db"
    ]
  }
}

2. Implementation in LoomManager:

• During workspace creation, after worktree setup
• Check for configured patterns
• Copy matching files from main branch to loom workspace
• Preserve file permissions and structure
• Log copied files for transparency

3. Security Considerations:

• Never copy these files during il finish (one-way copy only)
• Add warning if large database files detected (>100MB)
• Document that these files should remain gitignored
• Consider adding .iloom/workspace.ignore for workspace-specific ignores

Benefits

• Maintains iloom's isolated testing promise for all project types
• Supports local development workflows without cloud database dependencies
• Preserves security by keeping database files gitignored
• Provides flexibility through configuration rather than hard-coding behavior
• Enables true parallel development with database-dependent features

[acreeger]

@Gromski great insight! Would it be helpful to solve this by specifiying custom files that need to be copied to new looms?

I think this looks like the best path without going too deep into specific implementations:

• Investigate adding a copyGitIgnoredPatterns configuration option in .iloom/settings.json

[acreeger]

Clarifications from Maintainer

Questions and Answers:

Question	Answer
Database type	Unknown - not answered by reporter (but implementation should be database-agnostic)
Gitignore pattern	Unknown - not answered by reporter (implementation should support any pattern)
Preferred behavior	Configuration option - Add copyGitIgnoredPatterns setting in .iloom/settings.json
Security concerns	No major concerns - files stay on local machine. Should make it clear when copying files so user is aware of what's taking time.

Key Decision: Implement via configuration option (copyGitIgnoredPatterns) rather than automatic detection, with clear logging when files are copied.

[acreeger]

Complexity Assessment

Classification: SIMPLE

Metrics:

• Estimated files affected: 2
• Estimated lines of code: 150-200
• Breaking changes: No
• Database migrations: No
• Cross-cutting changes: No
• File architecture quality: Good - modifying well-organized classes with established patterns
• Overall risk level: Low

Reasoning: This is a straightforward feature addition that adds a new configuration option (copyGitIgnoredPatterns) to the IloomSettingsSchema and implements a single copying method in LoomManager following the existing pattern used for environment files, iloom settings, and claude settings. The changes are isolated, low-risk, and follow established patterns in the codebase.

[acreeger]

Combined Analysis & Plan - Issue #400

Executive Summary

Add a copyGitIgnoredPatterns configuration option to .iloom/settings.json that allows users to specify glob patterns for gitignored files (like .db or *.sqlite) to copy from the main repo to looms during workspace creation. This enables testing of database-dependent features without merging first.

Implementation Overview

High-Level Execution Phases

1. Schema Update: Add copyGitIgnoredPatterns to IloomSettingsSchema and IloomSettingsSchemaNoDefaults in SettingsManager.ts
2. Copying Logic: Add copyGitIgnoredFiles() method to LoomManager.ts following existing copyEnvironmentFiles() pattern
3. Integration: Call new method during workspace creation (both new and reused looms)
4. Testing: Add unit tests for the new functionality

Quick Stats

• 2 files to modify
• 0 new files to create
• 0 files to delete
• Dependencies: None (will use existing fs-extra for file operations and basic glob matching)

---

Complete Analysis & Implementation Details (click to expand)

Research Findings

Problem Space

• Problem: Gitignored database files (.db, .sqlite) don't transfer to looms, preventing testing without merging
• Architectural context: LoomManager already copies env files and settings using EnvironmentManager.copyIfExists()
• Edge cases: Large files, security concerns (both addressed by maintainer as non-issues for local files)

Codebase Research

• Entry point: /src/lib/LoomManager.ts:112-118 - where copyEnvironmentFiles/copyIloomSettings/copyClaudeSettings are called
• Dependencies: EnvironmentManager.copyIfExists() at line 143-155, fs-extra for file operations
• Similar patterns: copyEnvironmentFiles() at lines 684-726, copyIloomSettings() at lines 733-753
• Settings schema: IloomSettingsSchema at /src/lib/SettingsManager.ts:218-377

Affected Files

• /src/lib/SettingsManager.ts:218-377 - Schema definition needs new field
• /src/lib/LoomManager.ts:112-118, 684-726 - Add copying method and call it

Integration Points

• LoomManager.createIloom() calls copy methods at lines 112-118
• LoomManager.reuseIloom() calls copy methods at lines 1022-1024

---

Implementation Plan

Automated Test Cases to Create

Test File: /src/lib/LoomManager.test.ts (MODIFY)

Click to expand complete test structure (15 lines)

describe('copyGitIgnoredFiles', () => {
  it('should copy files matching configured patterns from main to worktree')
  it('should skip files that do not exist in main workspace')
  it('should skip files already in worktree')
  it('should handle multiple patterns')
  it('should log copied files for transparency')
  it('should warn but not fail on copy errors')
})

Files to Modify

1. /src/lib/SettingsManager.ts:218-377

Change: Add copyGitIgnoredPatterns field to both IloomSettingsSchema and IloomSettingsSchemaNoDefaults

// Add after line 269 (after protectedBranches):
copyGitIgnoredPatterns: z
  .array(z.string().min(1, 'Pattern cannot be empty'))
  .optional()
  .describe('Glob patterns for gitignored files to copy to looms (e.g., ["*.db", "data/*.sqlite"])'),

Same pattern needs to be added to IloomSettingsSchemaNoDefaults after line 438.

2. /src/lib/LoomManager.ts:112-118

Change: Add call to new copyGitIgnoredFiles() method after existing copy calls

// After line 118 (after copyClaudeSettings):
// 7.6. Copy gitignored files matching configured patterns
await this.copyGitIgnoredFiles(worktreePath)

3. /src/lib/LoomManager.ts (new method)

Change: Add new private method copyGitIgnoredFiles() following copyEnvironmentFiles pattern

Click to expand implementation pseudocode (25 lines)

/**
 * Copy gitignored files matching configured patterns from main repo to worktree
 * Uses copyGitIgnoredPatterns from settings to determine which files to copy
 * Only copies files that exist and are NOT tracked by git
 */
private async copyGitIgnoredFiles(worktreePath: string): Promise<void> {
  // Load settings to get patterns
  const settingsData = await this.settings.loadSettings()
  const patterns = settingsData.copyGitIgnoredPatterns
  
  // Exit early if no patterns configured
  if (!patterns || patterns.length === 0) {
    return
  }
  
  const mainWorkspacePath = this.gitWorktree.workingDirectory
  
  // For each pattern, find matching files and copy
  for (const pattern of patterns) {
    // Use simple pattern matching (supports *.ext and path/*.ext)
    // Find files matching pattern in main workspace
    // For each file: check exists, check not tracked, check not in worktree
    // Copy using this.environment.copyIfExists()
    // Log each copied file
  }
}

4. /src/lib/LoomManager.ts:1022-1024

Change: Add call to copyGitIgnoredFiles in reuseIloom() after other copy methods

// After line 1024 (after copyClaudeSettings):
await this.copyGitIgnoredFiles(worktreePath)

Detailed Execution Order

Phase 1: Schema Update

1. Add copyGitIgnoredPatterns to IloomSettingsSchema at line ~270 -> Verify: TypeScript compiles
2. Add copyGitIgnoredPatterns to IloomSettingsSchemaNoDefaults at line ~439 -> Verify: TypeScript compiles

Phase 2: Implementation

1. Add copyGitIgnoredFiles() method after copyClaudeSettings() (~line 782) -> Verify: Method exists
2. Call method in createIloom() after line 118 -> Verify: Method called during creation
3. Call method in reuseIloom() after line 1024 -> Verify: Method called during reuse

Phase 3: Testing

1. Add test cases to LoomManager.test.ts -> Verify: Tests pass
2. Run pnpm build -> Verify: Build succeeds

Dependencies and Configuration

None - uses existing fs-extra for file operations.

NOTE: For simple glob patterns like *.db or data/*.sqlite, we can use basic string matching with path.basename() and path.dirname(). For complex patterns, we could add fast-glob but simple patterns should cover 90% of use cases.

[acreeger]

Implementation Complete - Issue #400

Summary

Added copyGitIgnoredPatterns configuration option to .iloom/settings.json that allows users to specify glob patterns for gitignored files (like *.db, *.sqlite) to copy from the main repo to looms during workspace creation. This enables testing of database-dependent features without merging first.

Changes Made

• SettingsManager.ts: Added copyGitIgnoredPatterns field to both IloomSettingsSchema and IloomSettingsSchemaNoDefaults
• LoomManager.ts: Implemented copyGitIgnoredFiles(), findFilesMatchingPattern(), and matchesPattern() methods; integrated into both createIloom() and reuseIloom() workflows
• LoomManager.test.ts: Added 6 test cases covering pattern matching, git tracking checks, multiple patterns, error handling, and directory existence

Validation Results

• Tests: 3071 passed / 3094 total (23 skipped)
• Typecheck: Passed
• Lint: Passed

---

Detailed Changes by File (click to expand)

Files Modified

/src/lib/SettingsManager.ts

Changes: Added copyGitIgnoredPatterns optional array field to settings schemas

• Added to IloomSettingsSchema after protectedBranches (lines 274-277)
• Added to IloomSettingsSchemaNoDefaults after protectedBranches (lines 443-446)

/src/lib/LoomManager.ts

Changes: Implemented gitignored file copying functionality

• Added copyGitIgnoredFiles() method (lines 783-841) - copies files matching configured patterns
• Added findFilesMatchingPattern() method (lines 843-880) - finds files matching glob patterns in a directory
• Added matchesPattern() method (lines 882-909) - simple glob pattern matching (.ext, name., exact)
• Called copyGitIgnoredFiles() in createIloom() at line 121
• Called copyGitIgnoredFiles() in reuseIloom() at line 1158

/src/lib/LoomManager.test.ts

Changes: Added test suite for copyGitIgnoredFiles functionality

• Test: should skip copying when no patterns are configured
• Test: should copy files matching configured patterns
• Test: should skip files that are tracked by git
• Test: should handle multiple patterns
• Test: should warn but not fail on copy errors
• Test: should skip if pattern directory does not exist

Usage Example

{
  "copyGitIgnoredPatterns": [
    "*.db",
    "*.sqlite",
    "data/*.db"
  ]
}

Dependencies Added

None - uses existing fs-extra for file operations.

[acreeger]

Implementation Plan for Issue #400 - Replace Custom Glob with fast-glob

Summary

Replace the custom findFilesMatchingPattern and matchesPattern methods in LoomManager.ts (lines 854-912) with the fast-glob library to support advanced glob patterns like **/*.db, data/**/*.sqlite, and {data,backup}/*.db that the current implementation cannot handle.

Questions and Key Decisions

Question	Answer	Rationale
Which glob library to use?	fast-glob	High performance, well-maintained, supports all required patterns, async API with cwd option
How to handle Windows paths?	Use cwd option	fast-glob requires forward slashes in patterns; using cwd option avoids path conversion issues
Should we keep simple pattern fallback?	No	fast-glob handles all patterns including simple ones like *.db

High-Level Execution Phases

1. Add Dependency: Install fast-glob package
2. Replace findFilesMatchingPattern: Replace custom method with fast-glob call using cwd option
3. Remove matchesPattern: Delete unused helper method
4. Update Tests: Modify tests to mock fast-glob instead of fs.readdir
5. Update copyGitIgnoredFiles: Simplify to use new glob method
6. Validate Build: Run typecheck, lint, and tests

Quick Stats

• 0 files for deletion
• 2 files to modify (LoomManager.ts, LoomManager.test.ts)
• 0 new files to create
• Dependencies: fast-glob (new)
• Estimated complexity: Simple

---

Complete Implementation Guide (click to expand for step-by-step details)

Automated Test Cases to Create

Test File: /src/lib/LoomManager.test.ts (MODIFY)

Purpose: Update existing tests to mock fast-glob instead of fs.readdir, and add tests for advanced patterns

Click to expand complete test structure (20 lines)

// Update existing copyGitIgnoredFiles tests to:
// 1. Mock fast-glob (fg.glob) instead of fs.readdir
// 2. Add test for recursive ** patterns
// 3. Add test for brace expansion patterns

describe('copyGitIgnoredFiles', () => {
  it('should skip copying when no patterns are configured')  // Keep existing
  it('should copy files matching configured patterns')       // Update mock from fs.readdir to fg.glob
  it('should skip files that are tracked by git')            // Update mock
  it('should handle multiple patterns')                      // Update mock
  it('should warn but not fail on copy errors')              // Update mock
  it('should skip if pattern directory does not exist')      // Remove - fast-glob handles this
  
  // NEW tests
  it('should support recursive ** patterns like **/*.db')
  it('should support brace expansion like {data,backup}/*.db')
})

Files to Delete

None.

Files to Modify

1. /Users/adam/Documents/Projects/iloom-cli/fix-issue-400__db-gitignore-testing/package.json

Change: Add fast-glob as a production dependency

// In dependencies section, add:
"fast-glob": "^3.3.2"

2. /Users/adam/Documents/Projects/iloom-cli/fix-issue-400__db-gitignore-testing/src/lib/LoomManager.ts

Change 1: Add import for fast-glob at top of file (after line 1)

import fg from 'fast-glob'

Change 2: Replace findFilesMatchingPattern method (lines 854-883) with fast-glob implementation

The current implementation:

• Only reads a single directory level
• Only supports simple patterns: *.ext, name.*, exact match
• Cannot handle ** recursive patterns, brace expansion, or complex globs

Click to expand replacement pseudocode (15 lines)

/**
 * Find files matching a glob pattern using fast-glob
 * Supports full glob syntax: *.db, **/*.db, data/**/*.sqlite, {data,backup}/*.db
 */
private async findFilesMatchingPattern(basePath: string, pattern: string): Promise<string[]> {
  // Use fast-glob with cwd option to search from basePath
  // This avoids path manipulation issues on Windows
  // onlyFiles: true - we only want files, not directories
  // dot: true - match dot files if pattern includes them
  // Returns relative paths from basePath
  const matches = await fg.glob(pattern, {
    cwd: basePath,
    onlyFiles: true,
    dot: true,
  })
  return matches
}

Change 3: Delete matchesPattern method (lines 893-912)

This method is no longer needed as fast-glob handles all pattern matching internally.

3. /Users/adam/Documents/Projects/iloom-cli/fix-issue-400__db-gitignore-testing/src/lib/LoomManager.test.ts

Change: Update test mocks from fs.readdir to fast-glob

Current tests mock fs.readdir to return file entries. With fast-glob, tests should mock fg.glob to return matched paths directly.

Click to expand test update pseudocode (25 lines)

// Add mock at top of test file
vi.mock('fast-glob', () => ({
  default: {
    glob: vi.fn()
  }
}))

import fg from 'fast-glob'

// In each test, instead of:
const mockFsReaddir = vi.fn().mockResolvedValue([
  { name: 'app.db', isFile: () => true },
])
vi.mocked(fs.readdir).mockImplementation(mockFsReaddir)

// Use:
vi.mocked(fg.glob).mockResolvedValue(['app.db', 'cache.db'])

// For recursive pattern tests:
vi.mocked(fg.glob).mockResolvedValue([
  'data/app.db',
  'data/nested/other.db'
])

// For brace expansion tests:
vi.mocked(fg.glob).mockResolvedValue([
  'data/app.db',
  'backup/old.db'
])

Detailed Execution Order

Phase 1: Add Dependency

1. Run pnpm add fast-glob -> Verify: package.json updated, pnpm-lock.yaml updated

Phase 2: Update LoomManager.ts

1. Add import fg from 'fast-glob' after line 1 -> Verify: No import errors
2. Replace findFilesMatchingPattern method (lines 854-883) with fast-glob implementation -> Verify: TypeScript compiles
3. Delete matchesPattern method (lines 893-912) -> Verify: No references remain

Phase 3: Update Tests

1. Add fast-glob mock at top of LoomManager.test.ts -> Verify: Mock imports correctly
2. Update "should copy files matching configured patterns" test to mock fg.glob instead of fs.readdir -> Verify: Test passes
3. Update "should skip files that are tracked by git" test -> Verify: Test passes
4. Update "should handle multiple patterns" test -> Verify: Test passes
5. Update "should warn but not fail on copy errors" test to make fg.glob throw -> Verify: Test passes
6. Update/remove "should skip if pattern directory does not exist" test (fast-glob handles gracefully) -> Verify: Test removed or updated
7. Add new test for **/*.db recursive patterns -> Verify: Test passes
8. Add new test for {data,backup}/*.db brace expansion -> Verify: Test passes

Phase 4: Validate

1. Run pnpm build -> Verify: Build succeeds
2. Run pnpm test -> Verify: All tests pass
3. Run pnpm lint -> Verify: No lint errors

Dependencies and Configuration

Package	Version	Purpose
fast-glob	^3.3.2	Full glob pattern matching with async API

Note: fast-glob has no additional dependencies that conflict with the existing project.

Backwards Compatibility

The fast-glob library fully supports all patterns that the current implementation handles:

• *.db - matches files ending in .db in current directory
• data/*.sqlite - matches files ending in .sqlite in data directory
• Exact file names

Additionally, fast-glob enables these new patterns:

• **/*.db - recursively match all .db files
• data/**/*.db - recursively match .db files under data/
• {data,backup}/*.db - match .db files in either data/ or backup/
• *.{db,sqlite} - match files with either .db or .sqlite extension

[acreeger]

Implementation Complete - Issue #400

Summary

Replaced the custom findFilesMatchingPattern and matchesPattern methods in LoomManager.ts with the fast-glob library. This enables support for advanced glob patterns like **/*.db, data/**/*.sqlite, and {data,backup}/*.db that the previous implementation couldn't handle.

Changes Made

• package.json: Added fast-glob v3.3.3 as a production dependency
• LoomManager.ts: Replaced ~60 lines of custom pattern matching with ~10 lines using fast-glob
• LoomManager.test.ts: Updated tests to mock fg.glob instead of fs.readdir, added 2 new tests for advanced patterns

Validation Results

• Tests: 3072 passed / 3095 total (23 skipped)
• Typecheck: Passed
• Lint: Passed
• Build: Passed

---

Detailed Changes by File (click to expand)

Files Modified

/src/lib/LoomManager.ts

Changes: Replaced custom glob implementation with fast-glob

• Added import fg from 'fast-glob' at top of file
• Replaced findFilesMatchingPattern method (lines 854-883) with simplified fast-glob call
• Deleted matchesPattern helper method (lines 893-912) - no longer needed

/src/lib/LoomManager.test.ts

Changes: Updated tests to use fast-glob mocks

• Added import fg from 'fast-glob' and mock for fast-glob
• Updated 6 existing copyGitIgnoredFiles tests to mock fg.glob instead of fs.readdir
• Added new test: "should support recursive ** patterns like **/*.db"
• Added new test: "should support brace expansion like {data,backup}/*.db"

/package.json

Changes: Added fast-glob dependency

• Added "fast-glob": "^3.3.3" to dependencies

Dependencies Added

• fast-glob@3.3.3: Full glob pattern matching with async API

New Pattern Support

The implementation now supports:

• **/*.db - recursively match all .db files
• data/**/*.db - recursively match .db files under data/
• {data,backup}/*.db - match .db files in either data/ or backup/
• *.{db,sqlite} - match files with either .db or .sqlite extension

[acreeger]

Implementation Progress - Refactor copyGitIgnoredFiles

Starting work on refactoring copyGitIgnoredFiles to combine all glob patterns into a single fast-glob call.

Tasks

• Refactor copyGitIgnoredFiles to pass all patterns to a single fg.glob() call
• Remove findFilesMatchingPattern method (only used by copyGitIgnoredFiles)
• Update tests for the new implementation
• Run validation (typecheck, lint, tests, build)

ETA: ~10 minutes

[Gromski]

@acreeger - I was wondering about how best to do this without the risk of publishing things to GH that you don’t want to share. I suspect, as you say, that when you attach iloom to a project you might want an option to copy certain gitignored files into each loom, e.g. local db, .env, etc