-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update RCF to v3.8 and Enable Auto AD with 'Alert Once' Option #979
Changes from 2 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -51,6 +51,7 @@ | |
|
||
import com.amazon.randomcutforest.RandomCutForest; | ||
import com.amazon.randomcutforest.config.Precision; | ||
import com.amazon.randomcutforest.config.TransformMethod; | ||
import com.amazon.randomcutforest.parkservices.AnomalyDescriptor; | ||
import com.amazon.randomcutforest.parkservices.ThresholdedRandomCutForest; | ||
|
||
|
@@ -532,6 +533,10 @@ private void trainModelForStep( | |
.boundingBoxCacheFraction(TimeSeriesSettings.REAL_TIME_BOUNDING_BOX_CACHE_RATIO) | ||
.shingleSize(detector.getShingleSize()) | ||
.anomalyRate(1 - thresholdMinPvalue) | ||
.transformMethod(TransformMethod.NORMALIZE) | ||
.alertOnce(true) | ||
.autoAdjust(true) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. AlertOnce and AutoAdjust can interfere. It's ok to have both true -- will have fewer anomalies. If any of these are changed to false then we should consider changing the other (in particular, if autoAdjust is false then do we want alertOnce -- unclear, may be may be not) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. are you saying there are dependencies between these two options? How to figure out whether to set one false if the other one is false? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Well, "alertOnce" is a directive :) That is, if there is a "run" or continuous anomalies then we start alert once and stop (till some internal checks indicates that at least one normal point was seen). In the autoAdjust case, we would learn and try to suppress anomalies of both the observed value and expected value are in some noise range. So if there is a level shift followed quickly by another spike/dip then autoAdjust has some chance of catching it. But all of these are based on unsupervised estimations -- so it's hard to guarantee behavior. In general starting from (false,false) autoAdjust = true should reduce anomalies; and alertOnce = true would reduce even further. But there may be applications where users prefer a lot of continuous anomalies (for example, to detect a level shift!). |
||
.internalShinglingEnabled(false) | ||
.build(); | ||
Arrays.stream(dataPoints).forEach(s -> trcf.process(s, 0)); | ||
|
||
|
@@ -622,6 +627,10 @@ public List<ThresholdingResult> getPreviewResults(double[][] dataPoints, int shi | |
.boundingBoxCacheFraction(AnomalyDetectorSettings.BATCH_BOUNDING_BOX_CACHE_RATIO) | ||
.shingleSize(shingleSize) | ||
.anomalyRate(1 - this.thresholdMinPvalue) | ||
.transformMethod(TransformMethod.NORMALIZE) | ||
.alertOnce(true) | ||
.autoAdjust(true) | ||
.internalShinglingEnabled(false) | ||
.build(); | ||
return Arrays.stream(dataPoints).map(point -> { | ||
AnomalyDescriptor descriptor = trcf.process(point, 0); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
:)