feat(aggregator): implement configurable outlier detection and filtering pipeline

[Oluwaseyi89]

🚀 Neko Pull Request

Mark with an x all the checkboxes that apply (like [x])

• Closes Outlier Detection and Filtering Serviceo #18
• Added tests (if necessary)
• Run tests
• Run formatting
• Evidence attached
• Commented the code

---

📌 Type of Change

• Documentation (updates to README, docs, or comments)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

---

📝 Changes description

Implements a configurable outlier detection and filtering service to identify anomalous price data before aggregation. The service uses multiple statistical techniques to detect outliers and marks them with metadata while preserving them for analysis.

Core Implementation

Outlier Detection Strategies (apps/aggregator/src/strategies/outlier-detectors/):

• Range Detector: Filters negative prices and >50% changes in seconds (configurable thresholds)
• IQR Detector: Interquartile Range method with configurable multiplier (default: 1.5)
• Z-Score Detector: Standard deviation method with configurable threshold (default: 3.0)
• MAD Detector: Median Absolute Deviation method, robust to extreme values

OutlierDetectionService (apps/aggregator/src/services/outlier-detection.service.ts):

• Batch detection for price arrays
• Stream processing via RxJS operators
• Source quality metrics (% outliers per source)
• Configurable thresholds via environment variables
• Detailed outlier metadata (detector used, reason, severity)

Integration:

• Optional pre-filtering in aggregation pipeline
• Registered in app.module.ts as a provider
• Comprehensive unit tests with >85% coverage

Environment Variables Added

# Outlier Detection Configuration
OUTLIER_ENABLED=true
OUTLIER_RANGE_CHANGE_THRESHOLD=0.5
OUTLIER_IQR_MULTIPLIER=1.5
OUTLIER_ZSCORE_THRESHOLD=3.0
OUTLIER_MAD_THRESHOLD=3.5

📁 Files Added

File	Description
apps/aggregator/src/interfaces/outlier-result.interface.ts	Detection result types
apps/aggregator/src/services/outlier-detection.service.ts	Main service
apps/aggregator/src/services/outlier-detection.service.spec.ts	Unit tests
apps/aggregator/src/strategies/outlier-detectors/	All detector implementations
apps/aggregator/tsconfig.spec.json	Test configuration

---

📸 Evidence (A Loom/Cap video is required as evidence, we WON'T merge if there's no proof)

https://www.loom.com/share/81affce95e8e4ace83ba887111a58869

---

⏰ Time spent breakdown

Task	Time
Core service implementation	4 hours
Detector strategies (4 detectors)	3 hours
Integration and testing	2 hours
Documentation and cleanup	1 hour
Total	10 hours

---

🌌 Comments

• All detectors are configurable via environment variables with sensible defaults
• Service gracefully handles empty or insufficient data (returns empty array)
• Outliers are marked with metadata (not discarded) for debugging and analysis
• Source quality metrics help identify unreliable data providers
• Integration is optional and can be disabled via OUTLIER_ENABLED=false
• Tests cover edge cases including empty inputs, single values, and all-outlier scenarios

Closes #18

[drips-wave]

@Oluwaseyi89 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[Oluwaseyi89]

@Villarley please, review.

[Oluwaseyi89]

@Villarley please, review.