fix: resolve race condition in reference counting

.tool-versions

@@ -0,0 +1,2 @@
+golang 1.23.0
+nodejs 24.2.0

CLAUDE.md

@@ -0,0 +1,136 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Overview
+
+Asherah is an application-layer encryption SDK that provides advanced encryption features and defense in depth against compromise. It supports multiple languages (Java, Go, C#/.NET) and includes a gRPC server layer.
+
+**Important**: The Go implementation is considered the canonical implementation. The C# and Java implementations are somewhat legacy, although they are still used in production.
+
+## Build and Test Commands
+
+### Java
+```bash
+# Build
+cd java/app-encryption
+mvn compile
+
+# Run tests
+mvn test
+
+# Run integration tests
+mvn verify -Dskip.surefire.tests
+
+# Run tests with coverage
+mvn test jacoco:report
+
+# Run checkstyle
+mvn checkstyle:check
+
+# Run a single test
+mvn test -Dtest=TestClassName#testMethodName
+```
+
+### Go
+```bash
+# Build
+cd go/appencryption
+go build ./...
+
+# Run tests
+go test -race ./...
+
+# Run tests with coverage
+go test -race -coverprofile coverage.out ./...
+
+# Run integration tests (requires gotestsum)
+gotestsum -f testname --junitfile junit_integration_results.xml -- -p 1 -race -coverprofile coverage.out -v `go list ./integrationtest/... | grep -v traces`
+
+# Run linting
+golangci-lint run --config .golangci.yml
+
+# Run a single test
+go test -race -run TestName ./...
+```
+
+### C# / .NET
+```bash
+# Build
+cd csharp/AppEncryption
+dotnet build --configuration Release
+
+# Run tests
+dotnet test --configuration Release
+
+# Run tests with coverage
+dotnet test --configuration Release --test-adapter-path:. --logger:"junit;LogFilePath=test-result.xml" /p:CollectCoverage=true /p:Exclude=\"[xunit*]*,[*.Tests]*\" /p:CoverletOutputFormat=opencover
+```
+
+### Cross-Language Scripts
+```bash
+# Run cross-language encryption tests
+./tests/cross-language/scripts/encrypt_all.sh
+
+# Run cross-language decryption tests
+./tests/cross-language/scripts/decrypt_all.sh
+```
+
+## Code Architecture
+
+### Key Hierarchy
+The SDK implements a hierarchical key model with envelope encryption:
+- **Master Key (MK)**: Root key managed by HSM/KMS
+- **System Key (SK)**: KEK encrypted by MK, scoped to service/subsystem
+- **Intermediate Key (IK)**: KEK encrypted by SK, scoped to user/account
+- **Data Row Key (DRK)**: DEK encrypted by IK, generated per encryption operation
+
+### Core Components
+
+1. **Session Factory**: Entry point for creating encryption sessions
+   - Configured with metastore, KMS, and crypto policy
+   - Manages session lifecycle and caching
+
+2. **Metastore**: Pluggable key storage layer
+   - In-memory, RDBMS, and DynamoDB implementations
+   - Stores encrypted intermediate and system keys
+
+3. **Key Management Service (KMS)**: Pluggable master key provider
+   - AWS KMS, static (testing), and custom implementations
+   - Manages root key encryption/decryption
+
+4. **Crypto Policy**: Configures key rotation and caching behavior
+   - Key expiration intervals
+   - Cache TTL settings
+   - Rotation strategies
+
+### Language-Specific Patterns
+
+**Java**: Uses builder pattern extensively, heavy use of interfaces
+- Main package: `com.godaddy.asherah.appencryption`
+- Test naming: `*Test.java`, `*IT.java` for integration tests
+
+**Go**: Idiomatic Go with interfaces and struct embedding
+- Main package: `github.com/godaddy/asherah/go/appencryption`
+- Uses context.Context for cancellation and deadlines
+
+**C#**: Modern .NET patterns with async/await
+- Namespace: `GoDaddy.Asherah.AppEncryption`
+- Uses dependency injection patterns
+
+## Important Implementation Notes
+
+1. **Thread Safety**: All session factories and sessions are thread-safe
+2. **Memory Protection**: Uses secure memory for key storage (off-heap in Java, locked memory in Go/C#)
+3. **Key Caching**: Implements multi-level caching with configurable TTLs
+4. **Partition Scoping**: All encryption operations are scoped to a partition ID
+5. **Error Handling**: Fail-fast approach - operations fail completely rather than partially
+
+## Testing Approach
+
+- Unit tests for individual components
+- Integration tests with real metastores and KMS providers
+- Parameterized tests covering all key state combinations
+- Cross-language compatibility tests
+- Multi-threaded stress tests
+- Benchmark tests for performance validation

go/appencryption/ORPHAN_KEY_FIX_SUMMARY.md

@@ -0,0 +1,61 @@
+# Cache Eviction Orphan Key Fix Summary
+
+## Problem
+The cache eviction mechanism had a critical flaw where keys with active references would become "orphaned" - removed from the cache but not properly closed, leading to memory leaks.
+
+## Root Cause
+In `pkg/cache/cache.go`, the cache removes entries from its map BEFORE calling the eviction callback. If a key still has active references (ref count > 0), the `Close()` method returns early without actually closing the key. This creates orphaned keys that are:
+- No longer in the cache (cannot be retrieved)
+- Still consuming memory (not closed)
+- Lost forever (no way to track or clean them up)
+
+## Solution: Minimal Change Approach
+Added orphaned key tracking to `key_cache.go`:
+
+1. **Modified `Close()` to return bool** - indicates whether the key was actually closed
+2. **Track orphaned keys** - maintain a separate list of keys that failed to close during eviction
+3. **Periodic cleanup** - attempt to close orphaned keys every 100 cache accesses
+4. **Cleanup on cache close** - ensure orphaned keys are cleaned up when cache is closed
+
+## Implementation Details
+
+### Changes to `key_cache.go`:
+
+1. Added orphan tracking fields:
+```go
+orphaned []*cachedCryptoKey
+orphanedMu sync.Mutex
+```
+
+2. Modified eviction callback to track orphans:
+```go
+if !value.key.Close() {
+    c.orphanedMu.Lock()
+    c.orphaned = append(c.orphaned, value.key)
+    c.orphanedMu.Unlock()
+}
+```
+
+3. Added cleanup function:
+```go
+func (c *keyCache) cleanOrphaned() {
+    // Attempts to close orphaned keys
+    // Keeps only those still referenced
+}
+```
+
+4. Integrated cleanup into cache lifecycle:
+- Called periodically during `GetOrLoad`
+- Called during `Close()`
+
+## Benefits
+- **Prevents memory leaks** - orphaned keys are eventually cleaned up
+- **Minimal change** - doesn't require modifying third-party cache library
+- **Thread-safe** - uses mutex to protect orphaned list
+- **No performance impact** - cleanup is infrequent and synchronous
+
+## Testing
+- Verified orphaned keys are tracked when eviction fails
+- Confirmed cleanup removes keys once references are released
+- Ensured thread safety with concurrent access
+- All existing tests continue to pass

go/appencryption/PR_DESCRIPTION.md

@@ -0,0 +1,182 @@
+# Fix Race Condition in Reference Counting and Memory Leak from Cache Eviction
+
+## Summary
+
+This PR fixes two critical issues:
+1. A race condition in reference counting that could cause incorrect log messages
+2. A memory leak where evicted keys with active references become orphaned
+
+## Issue 1: Race Condition in Reference Counting
+
+### The Problem
+
+The original code had a Time-of-Check-Time-of-Use (TOCTOU) race condition:
+
+```go
+func (c *cachedCryptoKey) Close() {
+    newRefCount := c.refs.Add(-1)
+
+    if newRefCount == 0 {
+        if c.refs.Load() > 0 {  // RACE: refs could change between Add and Load
+            log.Debugf("cachedCryptoKey refcount is %d, not closing key", c.refs.Load())
+            return
+        }
+        log.Debugf("closing cached key: %s", c.CryptoKey)
+        c.CryptoKey.Close()
+    }
+}
+```
+
+**Race scenario:**
+1. Thread A calls Close(), `Add(-1)` returns 0 (was 1→0)
+2. Thread B calls `increment()`, refs becomes 1
+3. Thread A calls `Load()`, sees 1, logs "not closing"
+4. Result: Confusing log saying "refcount is 1, not closing" when we just decremented from 1→0
+
+### The Fix
+
+Capture the atomic result directly:
+
+```go
+func (c *cachedCryptoKey) Close() bool {
+    newRefCount := c.refs.Add(-1)
+    if newRefCount > 0 {
+        return false
+    }
+
+    // newRefCount is 0, which means the ref count was 1 before decrement
+    log.Debugf("closing cached key: %s, final ref count was 1", c.CryptoKey)
+    c.CryptoKey.Close()
+    return true
+}
+```
+
+This eliminates the race by using only the atomic operation's result.
+
+## Issue 2: Memory Leak from Orphaned Keys
+
+### The Problem
+
+The cache eviction mechanism has a fundamental flaw that causes memory leaks:
+
+```go
+// In pkg/cache/cache.go
+func (c *cache[K, V]) evictItem(item *cacheItem[K, V]) {
+    delete(c.byKey, item.key)  // Step 1: Remove from map
+    c.size--
+    c.policy.Remove(item)
+
+    // Step 2: Call eviction callback (which calls key.Close())
+    c.onEvictCallback(item.key, item.value)
+}
+```
+
+**The issue:** The cache removes entries from its map BEFORE checking if they can be closed.
+
+**Leak scenario:**
+1. Thread A gets key from cache (ref count: 1→2)
+2. Cache decides to evict the key
+3. Cache removes key from map (no new references possible)
+4. Cache calls `Close()` on key (ref count: 2→1)
+5. `Close()` returns early because ref count > 0
+6. Key is now orphaned: not in cache, but still allocated in memory
+7. Memory leaks until Thread A eventually closes its reference
+
+### The Solution
+
+Track orphaned keys and clean them up periodically:
+
+```go
+type keyCache struct {
+    // ... existing fields ...
+
+    // orphaned tracks keys that were evicted from cache but still have references
+    orphaned []*cachedCryptoKey
+    orphanedMu sync.Mutex
+
+    // cleanup management
+    cleanupStop chan struct{}
+    cleanupDone sync.WaitGroup
+}
+
+// In eviction callback
+onEvict := func(key string, value cacheEntry) {
+    if !value.key.Close() {
+        // Key still has active references, track it
+        c.orphanedMu.Lock()
+        c.orphaned = append(c.orphaned, value.key)
+        c.orphanedMu.Unlock()
+    }
+}
+```
+
+**Background cleanup goroutine (runs every 30 seconds):**
+
+```go
+func (c *keyCache) cleanOrphaned() {
+    // Swap the list to minimize lock time
+    c.orphanedMu.Lock()
+    toClean := c.orphaned
+    c.orphaned = make([]*cachedCryptoKey, 0)
+    c.orphanedMu.Unlock()
+
+    // Process outside the lock
+    remaining := make([]*cachedCryptoKey, 0)
+    for _, key := range toClean {
+        if !key.Close() {
+            remaining = append(remaining, key)
+        }
+    }
+
+    // Put back the ones we couldn't close
+    if len(remaining) > 0 {
+        c.orphanedMu.Lock()
+        c.orphaned = append(c.orphaned, remaining...)
+        c.orphanedMu.Unlock()
+    }
+}
+```
+
+## Why This Approach?
+
+### Minimal Change
+- Doesn't require modifying the third-party cache library
+- Only changes our wrapper code
+- Maintains backward compatibility
+
+### Performance Conscious
+- Eviction callbacks just append to a list (fast)
+- No operations in the hot path
+- Background cleanup every 30 seconds
+- List swapping minimizes lock contention
+
+### Correct Memory Management
+- Orphaned keys are tracked, not lost
+- Eventually freed when references are released
+- No permanent memory leaks
+- Bounded by number of concurrent operations
+
+## Testing
+
+All existing tests pass. The race condition fix has been validated by:
+1. The atomic operation guarantees correct behavior
+2. No separate Load() operation that could race
+
+The orphan cleanup has been validated by:
+1. Orphaned keys are tracked when eviction fails
+2. Background cleanup attempts to free them periodically
+3. Eventually all keys are freed when references are released
+
+## Alternative Approaches Considered
+
+1. **Modify cache library to retry eviction** - Too invasive, requires forking
+2. **Put keys back in cache if eviction fails** - Complex, could prevent new entries
+3. **Synchronous cleanup in hot path** - Would add variable latency
+4. **Using channels instead of list** - Can't re-queue keys that still have refs
+
+## Impact
+
+- **Fixes confusing/incorrect log messages** from the race condition
+- **Prevents memory leaks** in production systems with cache pressure
+- **No performance impact** - cleanup happens in background
+- **Graceful degradation** - if a key can't be freed, it's retried later