OPERATIONS.md

LATTICE.DB Operations Guide

This guide covers deploying, monitoring, and scaling LATTICE.DB.

Deployment

Embedded Mode

For applications that embed LATTICE.DB:

db, err := lattice.Open("./data", &lattice.Options{
    EncryptionKey: loadKeyFromSecureStorage(),
})
if err != nil {
    log.Fatal(err)
}
defer db.Close()

Considerations

File Location:

• Use app-specific directory (not user documents)
• Ensure sufficient disk space
• Consider backup strategy

Concurrent Access:

• Multiple goroutines can safely share a DB instance
• Each goroutine should use separate transactions
• Writable transactions serializably at commit

Resource Limits:

• Monitor memory usage with large datasets
• Set appropriate cache size based on working set
• Close database when application exits

Monitoring

Key Metrics

1. Transaction Throughput: Transactions per second
2. Latency: P50, P95, P99 commit latency
3. WAL Size: Current WAL file size
4. Cache Hit Rate: Percentage of cache hits
5. Sync Status: Last sync time, error count

Metrics Collection

// Get database stats
stats := db.Stats()
fmt.Printf("Transactions: %d\n", stats.TransactionCount)
fmt.Printf("Cache hits: %d\n", stats.CacheHits)
fmt.Printf("WAL size: %d\n", stats.WALSize)

Health Checks

// Verify database integrity
result, err := db.Verify()
if err != nil {
    // Database error
    return err
}
if !result.Passed() {
    // Database corruption detected
    log.Printf("Corruption: %v", result.Errors)
}

Backup and Restore

Backup

LATTICE.DB files can be backed up using standard tools:

# Hot backup (database can be in use)
cp -R /path/to/data /backup/location/data-$(date +%Y%m%d)

# Or use rsync for incremental backups
rsync -av /path/to/data/ /backup/location/

Restore

# Stop application
# Restore backup
cp -R /backup/location/data-20250101 /path/to/data
# Start application

Important:

• Ensure database is not running during restore
• Verify file permissions
• Consider running verification after restore

Maintenance

Verification

Run periodic verification:

lattice verify --data=/path/to/data

Scrubbing

Remove orphaned data:

lattice scrub --data=/path/to/data

Use --dry-run first to preview changes.

Compaction

WAL segments are automatically compacted, but you can trigger manual compaction:

lattice compact --data=/path/to/data

Scaling

Vertical Scaling

More Memory:

• Increase cache size
• Reduces disk I/O
• Better performance for large working sets

Faster Storage:

• Use SSD instead of HDD
• Reduces WAL latency
• Improves commit performance

Horizontal Scaling

LATTICE.DB scales horizontally via the sync server:

+-----------+     +-----------+     +-----------+
|  Device 1 | <---> |   Sync    | <---> |  Device 2 |
+-----------+     +-----------+     +-----------+
                           ^
                           |
                     +-----------+
                     |  Device 3 |
                     +-----------+

Server Considerations:

• Stateless operation (can load balance)
• Blob storage for encrypted chunks
• Rate limiting per device
• Session timeout management

Data Partitioning

For very large datasets, consider:

• Separate collections per tenant
• Separate database files per service
• Application-level sharding

Troubleshooting

Slow Commits

Symptoms: High commit latency

Causes:

• WAL sync bottleneck
• Large transactions
• Cache misses

Solutions:

• Batch operations in single transaction
• Increase WAL segment size
• Add more memory to cache

High Memory Usage

Symptoms: Memory grows over time

Causes:

• Uncommitted transactions holding versions
• Large transactions
• Cache too large

Solutions:

• Commit transactions more frequently
• Reduce transaction size
• Reduce cache size
• Restart application (clears versions)

Sync Conflicts

Symptoms: Many merge conflicts

Causes:

• Same documents edited concurrently
• High contention on hot documents

Solutions:

• Use document-level sharding
• Implement application-level conflict resolution
• Use collaborative editing patterns

Corruption

Symptoms: Verification fails

Steps:

1. Run lattice verify for detailed report
2. If WAL corruption, truncate WAL to last valid record
3. If data corruption, run lattice repair
4. Restore from backup if repair fails

Performance Tuning

Transaction Batching

// Instead of:
for _, item := range items {
    tx, _ := db.Begin(true)
    tx.Insert("items", item)
    tx.Commit()
}

// Do:
tx, _ := db.Begin(true)
for _, item := range items {
    tx.Insert("items", item)
}
tx.Commit()

Query Optimization

// Use indexes
schema.Fields["email"].Indexed = true

// Limit results
query.Limit = 100

// Use specific fields
query.Fields = []string{"name", "email"}

Cache Sizing

// Estimate cache size based on working set
workingSetPages := 10000 // ~40MB at 4KB pages
opts := &lattice.Options{
    CacheSize: workingSetPages,
}

Security

Key Management

Never hardcode keys:

// BAD
key := []byte{0x01, 0x02, ...}

// GOOD
key := os.Getenv("LATTICE_KEY")
keyBytes, _ := hex.DecodeString(key)

Use key derivation:

// Derive from password with salt
masterKey := crypto.DeriveKey(password, salt)

At-Rest Encryption

Enable encryption for sensitive data:

key := loadKeyFromSecureStorage()
db, _ := lattice.Open("./data", &lattice.Options{
    EncryptionKey: key,
})

Sync Security

• Always use HTTPS in production
• Validate server certificates
• Rotate device keys periodically
• Monitor for unauthorized access

Disaster Recovery

Backup Strategy

1. Daily Incremental Backups: WAL segments only
2. Weekly Full Backups: Entire data directory
3. Off-site Storage: Cloud or remote server
4. Backup Verification: Periodically test restore

Recovery Procedure

1. Stop application
2. Identify last good backup
3. Restore data directory
4. Run verification
5. Start application
6. Sync to catch up

Alerts

Recommended Alerts

1. Verification Failed: Database corruption detected
2. High WAL Size: WAL not compacting
3. Sync Failure: Can't reach sync server
4. High Latency: Performance degradation
5. Low Disk Space: <10% remaining