SceneBaseliner anomaly detection checks against modeled baseline rather than original data

[tdelacour]

Maybe I am misunderstanding the intended behavior here, but I had expected anomaly detection to be about detecting when my timeseries data crosses either the upper or lower bounds returned by the baseliner model. In the current implementation, the check is done against the values (e.g. the baseline) that the model itself returns. This means, at least in my experience, that there is never an anomaly. Can the intended behavior be clarified?

Code link: https://github.com/grafana/scenes-ml/blob/main/packages/scenes-ml/src/components/SceneBaseliner.tsx#L446

[sd2k]

Oh man yeah you're totally right, we should be checking y against the lower and upper bounds there. Do you want to send a PR to fix? I can get round to it soon if not.

Thanks for spotting this!

[tdelacour]

@sd2k yeah should be able to sometime today. Can I check an assumption first though? My quick hack that I implemented locally was to index y based on the assumption that the series returned form the model aligned with the last N points in y (where N is the length of the model series). I'm pretty sure that's how this is working, but I didn't do a super deep dive into the code or any unit testing. That's based on my understanding that the data window we are pulling for y includes some amount of extra historical data, and the model is only baselining against the last bit.

Just want to make sure that I'm aligning indexes correctly when swapping out the series arrays.

Appreciate the lightning-quick response on this issue btw :)

[sd2k]

Bleh, I have a nasty feeling that we can't assume that the indexes of the inputs (y and timeField.values) match the indexes of the output (times, values, etc), since we do a bunch of processing to each... I'm not sure what the best solution is here.

I wonder if we could generate an 'aligned' y at the point where we create the output times, using forward-fill to select the most relevant value of yAligned for each timestamp. That should be quite quick and efficient, although the forward-fill could be a bit fiddly.

(FWIW I asked Claude about this, the verbose and slightly sycophantic response is below 😅)

Anomaly Detection Data Indexing Challenge

User

The following items were attached by the user. They are up-to-date and don't need to be re-read. ```tsx scenes-ml/packages/scenes-ml/src/components/SceneBaseliner.tsx:444-455 if (onAnomalyDetected !== undefined && lower && upper) { for (let idx = 0; idx < values.length; idx++) { const value = values[idx]; const lowerValue = lower[idx]; const upperValue = upper[idx]; if (value < lowerValue) { onAnomalyDetected({ direction: 'lower', idx, time: times[idx], field: numField }); } else if (value > upperValue) { onAnomalyDetected({ direction: 'upper', idx, time: times[idx], field: numField }); } } } ```

This function incorrectly compares value (the point estimate from the model) with the lower and upper bounds, when we should be comparing the actual observed data (in the y variable). However, I don't think we can assume that the indexes of y and lower/upper match. Could you analyse this file and provide your thoughts, keeping in mind the way that we calculate the step between timestamps, and the way we generate point estimates from the model.

Agent

I'll analyze the SceneBaseliner.tsx file to understand how timestamps, point estimates, and observed data are handled to provide insights on the anomaly detection issue you've pointed out.

Use Tool: read_file (toolu_01TxtfmGbmVZFcuVm1X3EoVe)

{
  "path": "scenes-ml/packages/scenes-ml/src/components/SceneBaseliner.tsx"
}

Tool Results: toolu_01TxtfmGbmVZFcuVm1X3EoVe

Tool result too long. The first 1876 bytes:

This file was too big to read all at once.

Here is an outline of its symbols:

type ModelType [L31]
type AnomalyDirection [L37]
interface Anomaly [L40-49]
interface SceneBaselinerState [L51-77]
const DEFAULT_INTERVAL [L80]
const DEFAULT_TRAINING_FACTOR_OPTIONS [L83-87]
const DEFAULT_TRAINING_FACTOR_OPTION [L89-92]
class SceneBaseliner [L94-227]
public static Component [L97]
protected _urlSync [L98]
public latestData [L99]
public constructor( ) [L101-103]
public getExtraQueries( ) [L106-128]
public shouldRerun( ) [L131-144]
public getUrlState() [L147-153]
public onModelTypeChanged( ) [L155-157]
public onIntervalChanged( ) [L159-161]
public onDiscoverSeasonalitiesChanged( ) [L163-165]
public onFactorChanged( ) [L167-169]
public onClearFactor() [L171-173]
public onPinnedChanged( ) [L175-177]
public setLatestData( ) [L179-181]
public updateFromUrl( ) [L184-226]
const baselineProcessor [L233-274]
const defaultSeasonalities [L277-282]
const defaultSeasonalitiesArray [L283]
function canAddBaseline( ) [L295-302]
function determineSeasonLengths( ) [L315-322]
function createTimes( ) [L326-328]
interface AugursPredictionTransformationOptions [L331-348]
function createBaselinesForFrame( ) [L350-460]
function createFields( ) [L465-530]
function SceneBaselinerRenderer( ) [L532-624]
function getStyles( ) [L626-653]
& > div [L629-631]
& > div [L639-646]
.rc-slider [L640-642]
.rc-slider + div [L643-645]
interface ModelBase [L657-660]
interface ProphetModel [L662-665]
interface ETSModel [L666-669]
type Model [L671]
function fitModel( ) [L673-684]
function predictInSample( ) [L686-700]
function predictOutOfSample( ) [L702-717]

Showing symbols 1-45 (total symbols: 45)

Using the line numbers in this outline, you can call this tool again
while specifying the start_line and end_line fields to see the
implementations of symbols in the outline.

Agent

Let me look at the createBaselinesForFrame function which contains the anomaly detection logic you mentioned.

Use Tool: read_file (toolu_01YDNdYrPvx4y54CvNtWavtj)

{
  "path": "scenes-ml/packages/scenes-ml/src/components/SceneBaseliner.tsx",
  "start_line": 350,
  "end_line": 460
}

Tool Results: toolu_01YDNdYrPvx4y54CvNtWavtj

Tool result too long. The first 4591 bytes:

function createBaselinesForFrame(
modelType: ModelType,
frame: DataFrame,
interval?: number,
extraSeasonalities?: Duration[],
discoverSeasonalities = false,
timeRange?: TimeRange,
onAnomalyDetected?: (anomaly: Anomaly) => void,
): DataFrame {
const canAdd = canAddBaseline(frame);
if (!canAdd) {
return frame;
}

const numField = frame.fields[canAdd.numFieldIdx];
const timeField = frame.fields[canAdd.timeFieldIdx];

// Figure out the range and frequency of the data.
const inSampleRange = timeField.values.at(-1) - timeField.values.at(0);
const freq = timeField.values.at(1) - timeField.values.at(0);

// Convert the data to a Float64Array so it can be sent to the model.
// const y = new Float64Array(numField.values);
const y = numField.values;

const extraSeasonLengths = discoverSeasonalities ? Array.from(seasonalities(new Float64Array(y))) : [];
const seasonLengths = new Uint32Array([
...determineSeasonLengths(inSampleRange, freq, extraSeasonalities),
...extraSeasonLengths,
]);

// We can only do seasonal predictions for now :/
// Realistically that means we either need our data range to be > 2h or
// we need to have a detected seasonal pattern in the data.
if (modelType === 'ets' && seasonLengths.length === 0) {
return frame;
}

// Create and fit the model.
const model = fitModel(modelType, timeField.values, y, seasonLengths, interval);

// Get predictions for in-sample data (i.e. the same data we trained on).
let { values, lower, upper } = predictInSample(model, interval);
// const inSample = model.predictInSample(interval);
// let values = Array.from(inSample.point);
// let lower = inSample.intervals ? Array.from(inSample.intervals.lower) : undefined;
// let upper = inSample.intervals ? Array.from(inSample.intervals.upper) : undefined;

// Create an output time field with the right number of entries.
let totalSteps = inSampleRange / freq + 1;
let times = createTimes(totalSteps, freq, timeField.values.at(0));

// If we've been given a time range, we can filter our in-sample
// predictions to only include data within that range. If the range
// extends beyond the end of the data, we can also add out-of-sample
// predictions.
if (timeRange !== undefined) {
const { from, to } = timeRange;
// Find the indexes of the first and last times in our times array that we should keep.
const fromIdx = times.findIndex((t) => t >= from.valueOf());
let toIdx = times.findIndex((t) => t >= to.valueOf());
if (toIdx === -1) {
toIdx = times.length;
}
const inSampleSteps = toIdx - fromIdx;

// The final number of steps will now be the difference between to and from, divided by the
// frequency of the data. Add one to account because the range is inclusive.
totalSteps = (to.valueOf() - from.valueOf()) / freq + 1;
const outOfSampleSteps = totalSteps - inSampleSteps;

// Slice the in-sample data to only include the data we want.
times = times.slice(fromIdx, toIdx);
values = values.slice(fromIdx, toIdx);
lower = lower ? lower.slice(fromIdx, toIdx) : undefined;
upper = upper ? upper.slice(fromIdx, toIdx) : undefined;

// Add out-of-sample predictions.
if (outOfSampleSteps > 0) {
  const outOfSample = predictOutOfSample(
    model, outOfSampleSteps, interval, times[0], freq
  )
  // const outOfSample = model.predict(outOfSampleSteps, interval);
  // Recreate the times array since we're going to have data for the
  // full time range now.
  times = createTimes(totalSteps, freq, times[0]);
  values = values.concat(Array.from(outOfSample.values));
  if (lower && upper && outOfSample.lower && outOfSample.upper) {
    lower = lower.concat(Array.from(outOfSample.lower));
    upper = upper.concat(Array.from(outOfSample.upper));
  }
}

}

if (onAnomalyDetected !== undefined && lower && upper) {
for (let idx = 0; idx < values.length; idx++) {
const value = values[idx];
const lowerValue = lower[idx];
const upperValue = upper[idx];
if (value < lowerValue) {
onAnomalyDetected({ direction: 'lower', idx, time: times[idx], field: numField });
} else if (value > upperValue) {
onAnomalyDetected({ direction: 'upper', idx, time: times[idx], field: numField });
}
}
}

const name = numField.config.displayNameFromDS ?? frame.name ?? numField.name;
const fields = createFields(name, timeField, times, values, lower, upper);
return { fields, length: times.length };
}

Agent

Now let me look at how the model predictions are generated to better understand the available data.

Use Tool: read_file (toolu_01KvhtWhL535PE5GxgwzBAdf)

{
  "path": "scenes-ml/packages/scenes-ml/src/components/SceneBaseliner.tsx",
  "start_line": 673,
  "end_line": 717
}

Tool Results: toolu_01KvhtWhL535PE5GxgwzBAdf

Tool result too long. The first 2026 bytes:

function fitModel(type: ModelType, ds: number[], y: number[], seasonLengths: Uint32Array, level?: number): Model {
switch (type) {
case 'ets':
const ets = new MSTL('ets', seasonLengths, { impute: true });
ets.fit(y);
return { type, model: ets };
case 'prophet':
const prophet = new Prophet({ optimizer, intervalWidth: level });
prophet.fit({ ds, y });
return { type, model: prophet };
}
}

function predictInSample({ model, type }: Model, level?: number) {
if (type === "ets") {
const inSample = model.predictInSample(level);
const values = Array.from(inSample.point);
const lower = inSample.intervals ? Array.from(inSample.intervals.lower) : undefined;
const upper = inSample.intervals ? Array.from(inSample.intervals.upper) : undefined;
return { values, lower, upper };
} else {
const inSample = model.predict();
const values = Array.from(inSample.yhat.point);
const lower = inSample.yhat.intervals ? Array.from(inSample.yhat.intervals.lower) : undefined;
const upper = inSample.yhat.intervals ? Array.from(inSample.yhat.intervals.upper) : undefined;
return { values, lower, upper }
}
}

function predictOutOfSample({ model, type }: Model, steps: number, level?: number, last?: number, interval?: number) {
if (type === "ets") {
const preds = model.predict(steps, level);
const values = Array.from(preds.point);
const lower = preds.intervals ? Array.from(preds.intervals.lower) : undefined;
const upper = preds.intervals ? Array.from(preds.intervals.upper) : undefined;
return { values, lower, upper };
} else {
const trainingData = createTimes(steps, interval!, last!);
const preds = model.predict({ ds: trainingData });
const values = Array.from(preds.yhat.point);
const lower = preds.yhat.intervals ? Array.from(preds.yhat.intervals.lower) : undefined;
const upper = preds.yhat.intervals ? Array.from(preds.yhat.intervals.upper) : undefined;
return { values, lower, upper }
}
}

Agent

Let me also look at the createFields function to understand the structure of the returned data frame.

Use Tool: read_file (toolu_01Cpk8ac5pWwGrbowx1RmZs5)

{
  "path": "scenes-ml/packages/scenes-ml/src/components/SceneBaseliner.tsx",
  "start_line": 465,
  "end_line": 530
}

Tool Results: toolu_01Cpk8ac5pWwGrbowx1RmZs5

Tool result too long. The first 1585 bytes:

function createFields(name: string, timeField: Field, times: number[], point: number[], lower?: number[], upper?: number[]) {
// Always add the point estimates, but only add the intervals if they exist.
const fields: Field[] = [
// Recreate the time field with the new times.
{
...timeField,
values: times,
},
{
name: ${name} - baseline,
type: FieldType.number,
values: point,
config: {
displayNameFromDS: ${name} - baseline,
color: {
fixedColor: 'blue',
mode: FieldColorModeId.Fixed,
},
},
},
];
if (lower && upper) {
fields.push({
name: ${name} - lower,
type: FieldType.number,
values: lower,
config: {
displayNameFromDS: ${name} - lower,
color: {
fixedColor: 'blue',
mode: FieldColorModeId.Fixed,
},
custom: {
lineWidth: 1,
hideFrom: {
viz: false,
tooltip: false,
legend: true,
},
}
},
});
fields.push({
name: ${name} - upper,
type: FieldType.number,
values: upper,
config: {
displayNameFromDS: ${name} - upper,
color: {
fixedColor: 'blue',
mode: FieldColorModeId.Fixed,
},
custom: {
fillBelowTo: ${name} - lower,
lineWidth: 1,
hideFrom: {
viz: false,
tooltip: false,
legend: true,
},
}
},
});
}
return fields;
}

Agent

Use Tool: thinking (toolu_01QiFi8UCP3xZE78hj3DQZ5X)

{
  "content": "Let me analyze what's happening in the anomaly detection logic based on the code I've reviewed:\n\n1. The code takes in a data frame with time series data and creates a prediction model (either ETS or Prophet).\n\n2. The model generates:\n   - 'values' (point estimates from the model)\n   - 'lower' (lower bound of confidence interval)\n   - 'upper' (upper bound of confidence interval)\n\n3. The anomaly detection logic (lines 444-455) compares the point estimates ('values') against the confidence intervals ('lower' and 'upper').\n\n4. The issue identified is that we should be comparing the original observed data ('y') against these confidence intervals, not the point estimates themselves.\n\n5. Looking at the data flow:\n   - 'y' contains the original observed values from numField.values\n   - The model is fit using these values and generates point estimates and confidence intervals\n   - The time array 'times' might be modified based on time range filtering\n   - After potential filtering and out-of-sample prediction, the code uses 'values', 'lower', and 'upper' for anomaly detection\n\n6. The problem with comparing against the original 'y' values is that the indexes might not align after filtering and out-of-sample prediction, as the user mentioned.\n\nI need to figure out how to correctly access the observed data at each point for comparison with the confidence intervals."
}

Tool Results: toolu_01QiFi8UCP3xZE78hj3DQZ5X

Tool result too long. The first 18 bytes:

Finished thinking.

Agent

Based on my analysis of the SceneBaseliner.tsx file, I can provide insights on the anomaly detection issue you've highlighted.

Current Implementation Issue

You're absolutely correct that the current implementation has a logical error. In the anomaly detection code (lines 444-455), it compares the model's point estimates (values) against the confidence intervals (lower and upper), when it should be comparing the actual observed data (y) against these confidence intervals.

if (onAnomalyDetected !== undefined && lower && upper) {
  for (let idx = 0; idx < values.length; idx++) {
    const value = values[idx];
    const lowerValue = lower[idx];
    const upperValue = upper[idx];
    if (value < lowerValue) {
      onAnomalyDetected({ direction: 'lower', idx, time: times[idx], field: numField });
    } else if (value > upperValue) {
      onAnomalyDetected({ direction: 'upper', idx, time: times[idx], field: numField });
    }
  }
}

The Indexing Challenge

Your concern about indexing mismatches between y and the confidence intervals is valid. Here's why this is complex:

1. The original time series data (y and original timestamps) might be filtered or modified based on the timeRange parameter
2. The code might add out-of-sample predictions that extend beyond the original data
3. The arrays times, values, lower, and upper are manipulated together to remain in sync

Proposed Solution Approach

To correctly perform anomaly detection, we need to:

1. Track the original data points throughout the processing
2. Align them with our confidence intervals for comparison

Here's a potential solution approach:

// After processing the data and before the anomaly detection block
// Create a map of timestamps to original values (for lookup during anomaly detection)
const timeToOriginalValue = new Map();
for (let i = 0; i < timeField.values.length; i++) {
  timeToOriginalValue.set(timeField.values[i], numField.values[i]);
}

// Then in the anomaly detection block
if (onAnomalyDetected !== undefined && lower && upper) {
  for (let idx = 0; idx < values.length; idx++) {
    const timestamp = times[idx];
    const lowerValue = lower[idx];
    const upperValue = upper[idx];
    
    // Only check for anomalies if we have an original value for this timestamp
    // (ignores out-of-sample prediction points)
    if (timeToOriginalValue.has(timestamp)) {
      const originalValue = timeToOriginalValue.get(timestamp);
      if (originalValue < lowerValue) {
        onAnomalyDetected({ direction: 'lower', idx, time: timestamp, field: numField });
      } else if (originalValue > upperValue) {
        onAnomalyDetected({ direction: 'upper', idx, time: timestamp, field: numField });
      }
    }
  }
}

Key Considerations

1. Timestamp Precision: We need to be careful with floating-point timestamp comparisons. It might be better to use a tolerance or round timestamps to a specific precision when creating the map.

2. Performance: For very large datasets, the Map approach could have memory implications, but it provides the most accurate way to match original values with their corresponding confidence intervals.

3. Out-of-Sample Data: The solution above only detects anomalies for points that exist in the original dataset. If you want to extend anomaly detection to predicted points, you'd need a different approach.

4. Time Step Handling: The code already has logic for handling the time step between points, which is important for aligning the data correctly after filtering.

Would you like me to refine this solution further, or would you prefer to see alternative approaches to this problem?

User

Can you think of some alternative approaches that don't involve a map?

Agent

Certainly, I can think of a few alternative approaches that don't involve using a map. Let's explore some options:

Alternative 1: Track Original Index Mapping

Instead of storing a map of timestamps to values, we could track the mapping between the processed indices and the original indices:

// During data processing, maintain an array that maps processed indices to original indices
const originalIndices: number[] = [];

// When filtering times/values based on timeRange
if (timeRange !== undefined) {
  const { from, to } = timeRange;
  const fromIdx = times.findIndex((t) => t >= from.valueOf());
  let toIdx = times.findIndex((t) => t >= to.valueOf());
  if (toIdx === -1) {
    toIdx = times.length;
  }
  
  // Store mapping of new indices to original indices
  for (let i = fromIdx; i < toIdx; i++) {
    originalIndices.push(i);
  }
  
  // Then process data as before...
  // (slicing times, values, lower, upper)
  
  // For out-of-sample predictions, use null or -1 to indicate no original index
  for (let i = 0; i < outOfSampleSteps; i++) {
    originalIndices.push(-1);
  }
}

// In the anomaly detection block
if (onAnomalyDetected !== undefined && lower && upper) {
  for (let idx = 0; idx < values.length; idx++) {
    const origIdx = originalIndices[idx];
    if (origIdx >= 0) { // Check if this is an in-sample point
      const originalValue = y[origIdx];
      if (originalValue < lower[idx]) {
        onAnomalyDetected({ direction: 'lower', idx, time: times[idx], field: numField });
      } else if (originalValue > upper[idx]) {
        onAnomalyDetected({ direction: 'upper', idx, time: times[idx], field: numField });
      }
    }
  }
}

Alternative 2: Include Original Values in the Output Frame

Another approach is to include the original values in the output data frame and use that for comparison:

// Modify the createFields function to include the original data
function createFields(name: string, timeField: Field, times: number[], point: number[], originalData: number[] | null, lower?: number[], upper?: number[]) {
  const fields: Field[] = [
    // Time field and baseline fields as before
    // ...
    
    // Add original data field if available
    ...(originalData ? [{
      name: `${name} - original`,
      type: FieldType.number,
      values: originalData,
      config: {
        displayNameFromDS: name,
        // styling config
      },
    }] : []),
  ];
  
  // Add lower and upper bounds as before
  // ...
  
  return fields;
}

// Then in createBaselinesForFrame, prepare the original data array
let originalValues: number[] = [];
if (timeRange !== undefined) {
  // After filtering and possibly adding out-of-sample predictions
  originalValues = new Array(times.length).fill(null);
  
  // For each timestamp in the processed data
  for (let i = 0; i < times.length; i++) {
    // Try to find this timestamp in the original timeField
    const origIdx = timeField.values.findIndex(t => t === times[i]);
    if (origIdx >= 0) {
      originalValues[i] = numField.values[origIdx];
    }
  }
}

// Then for anomaly detection
if (onAnomalyDetected !== undefined && lower && upper) {
  for (let idx = 0; idx < originalValues.length; idx++) {
    if (originalValues[idx] !== null) {
      if (originalValues[idx] < lower[idx]) {
        onAnomalyDetected({ direction: 'lower', idx, time: times[idx], field: numField });
      } else if (originalValues[idx] > upper[idx]) {
        onAnomalyDetected({ direction: 'upper', idx, time: times[idx], field: numField });
      }
    }
  }
}

Alternative 3: Direct Binary Search on Original Timestamps

Use binary search to efficiently find the matching original value for each timestamp:

// Helper function to find the index of a timestamp in the original data
function findOriginalIndex(timestamp: number, originalTimes: number[]): number {
  // Binary search since timestamps are ordered
  let left = 0;
  let right = originalTimes.length - 1;
  
  while (left <= right) {
    const mid = Math.floor((left + right) / 2);
    if (Math.abs(originalTimes[mid] - timestamp) < 0.001) { // Small epsilon for float comparison
      return mid;
    }
    if (originalTimes[mid] < timestamp) {
      left = mid + 1;
    } else {
      right = mid - 1;
    }
  }
  return -1; // Not found
}

// In the anomaly detection block
if (onAnomalyDetected !== undefined && lower && upper) {
  const originalTimes = timeField.values;
  const originalValues = numField.values;
  
  for (let idx = 0; idx < values.length; idx++) {
    const timestamp = times[idx];
    const origIdx = findOriginalIndex(timestamp, originalTimes);
    
    if (origIdx >= 0) {
      const originalValue = originalValues[origIdx];
      if (originalValue < lower[idx]) {
        onAnomalyDetected({ direction: 'lower', idx, time: timestamp, field: numField });
      } else if (originalValue > upper[idx]) {
        onAnomalyDetected({ direction: 'upper', idx, time: timestamp, field: numField });
      }
    }
  }
}

Alternative 4: Recreate Original Dataset with Same Processing

Another approach is to carry along the original data through the same transformations applied to the model outputs:

// During the processing of values, lower, and upper arrays, 
// apply the exact same operations to a copy of the original y values

// Start with the original values
let originalValuesForComparison = Array.from(y);

// If we're slicing based on timeRange
if (timeRange !== undefined) {
  const { from, to } = timeRange;
  const fromIdx = times.findIndex((t) => t >= from.valueOf());
  let toIdx = times.findIndex((t) => t >= to.valueOf());
  if (toIdx === -1) {
    toIdx = times.length;
  }
  
  // Slice original values the same way we slice model outputs
  originalValuesForComparison = originalValuesForComparison.slice(fromIdx, toIdx);
  
  // For out-of-sample predictions, pad with nulls
  if (outOfSampleSteps > 0) {
    originalValuesForComparison = originalValuesForComparison.concat(new Array(outOfSampleSteps).fill(null));
  }
}

// Then in anomaly detection
if (onAnomalyDetected !== undefined && lower && upper) {
  for (let idx = 0; idx < values.length; idx++) {
    if (originalValuesForComparison[idx] !== null) {
      const originalValue = originalValuesForComparison[idx];
      if (originalValue < lower[idx]) {
        onAnomalyDetected({ direction: 'lower', idx, time: times[idx], field: numField });
      } else if (originalValue > upper[idx]) {
        onAnomalyDetected({ direction: 'upper', idx, time: times[idx], field: numField });
      }
    }
  }
}

Comparison of Approaches

1. Tracking Original Indices: Simple to implement, efficient, but requires careful management of the index array through all transformations.

2. Including Original Values in Output: Makes the original data explicitly available in the result, useful for visualization, but requires more memory.

3. Binary Search: Doesn't require maintaining extra arrays during processing, but might be less efficient for large datasets or if timestamp comparisons are tricky.

4. Parallel Transformation: Conceptually clean as it mirrors the exact transformations applied to model outputs, but requires careful synchronization of operations.

Each approach has its merits depending on the specific requirements of your system. The best choice would depend on:

1. The typical size of your datasets
2. Whether you need original values for other purposes
3. The frequency of anomaly detection checks
4. How your time filtering and processing logic might evolve in the future

Which of these approaches seems most suitable for your use case?

User

The following items were attached by the user. They are up-to-date and don't need to be re-read. ```tsx scenes-ml/packages/scenes-ml/src/components/SceneBaseliner.tsx:398-400 // Create an output time field with the right number of entries. let totalSteps = inSampleRange / freq + 1; let times = createTimes(totalSteps, freq, timeField.values.at(0)); ```

I wonder if we could do something like creating an 'aligned' y variable at the bit where we're creating the output times, right after doing in-sample predictions? we could forward-fill perhaps

Agent

You've highlighted a great opportunity for a solution. Looking at the code around line 398-400 where the output times are created, we could indeed create an aligned version of the original y values at that point. Let me elaborate on how we could implement this approach:

Solution Using Aligned y Values at Time Creation

At the point where the output times are created (right after in-sample predictions), we could create an aligned version of the original values that matches the new time array:

// After creating the times array (around line 400)
let totalSteps = inSampleRange / freq + 1;
let times = createTimes(totalSteps, freq, timeField.values.at(0));

// Create an aligned version of the original y values that matches our times array
let alignedY = new Array(times.length);

// For each timestamp in our new times array, find the corresponding original value
for (let i = 0; i < times.length; i++) {
  const timestamp = times[i];
  
  // Find the closest original timestamp
  const closestIdx = findClosestTimestampIndex(timestamp, timeField.values);
  
  if (closestIdx !== -1) {
    // If we found a reasonably close timestamp, use its value
    alignedY[i] = y[closestIdx];
  } else {
    // Otherwise, use forward-fill from the previous value
    alignedY[i] = i > 0 ? alignedY[i-1] : null;
  }
}

// Helper function to find the closest timestamp within a small tolerance
function findClosestTimestampIndex(target: number, timestamps: number[]): number {
  // Define a small tolerance for timestamp comparisons (e.g., half the frequency)
  const tolerance = freq / 2;
  
  // Find the index of the closest timestamp within tolerance
  for (let i = 0; i < timestamps.length; i++) {
    if (Math.abs(timestamps[i] - target) <= tolerance) {
      return i;
    }
  }
  
  return -1; // No matching timestamp found within tolerance
}

Then later in the code, when we apply time range filters and extend with out-of-sample predictions, we would modify the alignedY array in parallel with the other arrays:

// When we filter based on timeRange
if (timeRange !== undefined) {
  const { from, to } = timeRange;
  const fromIdx = times.findIndex((t) => t >= from.valueOf());
  let toIdx = times.findIndex((t) => t >= to.valueOf());
  if (toIdx === -1) {
    toIdx = times.length;
  }
  
  // Slice all arrays including our alignedY
  times = times.slice(fromIdx, toIdx);
  values = values.slice(fromIdx, toIdx);
  lower = lower ? lower.slice(fromIdx, toIdx) : undefined;
  upper = upper ? upper.slice(fromIdx, toIdx) : undefined;
  alignedY = alignedY.slice(fromIdx, toIdx);
  
  // When adding out-of-sample predictions
  if (outOfSampleSteps > 0) {
    // Extend alignedY with nulls for out-of-sample predictions
    alignedY = alignedY.concat(new Array(outOfSampleSteps).fill(null));
    // (other arrays are extended with out-of-sample predictions as in original code)
  }
}

Finally, in the anomaly detection block, we would use our aligned y values for comparison:

// In the anomaly detection block
if (onAnomalyDetected !== undefined && lower && upper) {
  for (let idx = 0; idx < values.length; idx++) {
    // Only check for anomalies where we have actual data (not null)
    if (alignedY[idx] !== null) {
      if (alignedY[idx] < lower[idx]) {
        onAnomalyDetected({ direction: 'lower', idx, time: times[idx], field: numField });
      } else if (alignedY[idx] > upper[idx]) {
        onAnomalyDetected({ direction: 'upper', idx, time: times[idx], field: numField });
      }
    }
  }
}

Forward-Fill Implementation

If we specifically want to use forward-fill as you suggested, we could simplify the initial alignment process:

// After creating the times array (around line 400)
let totalSteps = inSampleRange / freq + 1;
let times = createTimes(totalSteps, freq, timeField.values.at(0));

// Create an aligned version of the original y values that matches our times array
let alignedY = new Array(times.length).fill(null);

// Initialize with the first value we know
let lastKnownValue = y[0];
let lastKnownTime = timeField.values[0];
let yIndex = 1; // Start from the second point in original data

// Forward-fill and update from original data when timestamps align
for (let i = 0; i < times.length; i++) {
  const currentTime = times[i];
  
  // Check if we need to advance in the original data
  while (yIndex < y.length && timeField.values[yIndex] <= currentTime) {
    lastKnownValue = y[yIndex];
    lastKnownTime = timeField.values[yIndex];
    yIndex++;
  }
  
  // If this timestamp is exactly on an original data point, use that value
  if (Math.abs(currentTime - lastKnownTime) < 0.001) {
    alignedY[i] = lastKnownValue;
  } else {
    // Otherwise use the last known value (forward-fill)
    alignedY[i] = lastKnownValue;
  }
}

Benefits of This Approach

1. Simplicity: By aligning the original values with the new time array early in the process, we maintain synchronization throughout all subsequent transformations.

2. Forward-Fill: Your suggestion to use forward-fill ensures we have values for comparison even if timestamps don't exactly match.

3. Minimal Changes: This approach requires minimal changes to the existing code structure, making it easier to implement and less prone to introducing new bugs.

4. Performance: This approach avoids repeated searches or comparisons during anomaly detection, as the alignment is done once up front.

This approach seems very clean and aligns well with the existing code structure. Would this solution work for your use case?

---

We might be able to punt this in the common case though - in the vast majority of cases I think the timestamps will be aligned between inputs and outputs. In that case, your assumption is semi-correct, but I think it depends on the provided timeRange; we filter the predictions to only contain those in that time range, so we'd probably need to filter the input values in the exact same way, then we could index them both equally and safely.

Sorry about the complicated answer! I can take a look next week if it looks fiddly. Alternatively, if you have something that works locally, feel free to push it and I'll give it a review 👍

[tdelacour]

ok yeah I had a feeling there might be some nuance here. FWIW, when I was playing with this locally, just dumbly assuming aligned indexes led to more anomalies than I expected just looking at my charts visually. So it's quite possible things were not well aligned.

I'll see if I get a bit of time & motivation to take a closer look at a PR for this - thanks for the suggested approaches. No promises though - might need to leave it to you for next week

[sd2k]

@tdelacour I've been struggling to prioritise this but had Claude take a look and it seems sensible (see #70). There's a canary release out if you'd like to give it a go:

yarn add @grafana/scenes-ml@0.5.1--canary.70.15561052406.0

I'll wait until I've found time to do more thorough testing before merging.