Preliminaries

Protocol types

In general, evaluation criteria can be roughly categorized into point-based protocols, range-based protocols and event-based protocols.

• Point-based protocols like traditional F1 treat each individual data point as a separate sample, disregarding the holistic characteristics of the anomaly segment.
• Range-based protocols incorporate segment-level features, for instance, the detection latency, into the evaluation.
• Event-based protocols treat each anomaly segment as an individual event. Each event contributes to a true positive or false negative only once or limited times.

Point-Adjustment strategy (PA)

Under this strategy, all timestamps within an anomalous segment are assigned the highest anomaly score present within that segment, thus the whole anomaly segment is considered to be detected if at least one anomaly score surpasses the threshold. Then the F1 score is obtained in a point-based manner. This is widely used in current methods, but is flawed when combined with point-based protocols.

Reduced-length PA (event-based protocols with mode log)

Partly Consider the length if anomaly segments against event-wise PA. Details are illustrated as follows:

Delay

As depicted in the illustration, assuming the latency limit (k) is set to 3, an anomaly is considered effectively detected only if it is identified within three sampling points after its occurrence. We designate this strategy as k-delay adjustment. This measure enables a more precise assessment of whether the model can meet the requirement of the scenario where there is a high demand for real-time responsiveness. It is equally essential to acknowledge that this approach is applicable only to datasets whose anomalies are labeled without positional bias. We conduct experiments on the selected datasets with non-biased labels.

All Built-in Protocols

Evaluations.Protocols

EventDetect

EventDetect

EventDetect()

Bases: EvalInterface

Using the UCR detection protocol to evaluate the models. As there is only one anomaly segment in one time series, if and only if the highest score is in the anomaly segment, this time series is considered to be detected.

calc

calc(scores, labels, margins)

Returns:

Name	Type	Description
MetricInterface	type[MetricInterface]	An instance of Precision representing if the anomaly is detected.

EventF1PA

EventF1PA

EventF1PA(mode='log', base=3)

Bases: EvalInterface

Parameters:

Name	Type	Description	Default
mode	str	Defines the scale at which the anomaly segment is processed. One of: - 'squeeze': View an anomaly event lasting t timestamps as one timepoint. - 'log': View an anomaly event lasting t timestamps as log(t) timepoint. - 'sqrt': View an anomaly event lasting t timestamps as sqrt(t) timepoint. - 'raw': View an anomaly event lasting t timestamps as t timepoint. If using 'log', you can specify the param "base" to return the logarithm of x to the given base, calculated as log(x) / log(base).	'log'
base	int	Default is 3.	3

calc

calc(scores, labels, margins)

Returns:

Type	Description
type[MetricInterface]	A F1class (Evaluations.Metrics.F1class), including: best_f1: the value of best f1 score; precision: corresponding precision value; recall: corresponding recall value; threshold: the value of threshold when getting best f1.

EventKthF1PA

EventKthF1PA

EventKthF1PA(k, mode='log', base=3)

Bases: EvalInterface

Parameters:

Name	Type	Description	Default
k	int	Defines the delay limit.	required
mode	str	Defines the scale at which the anomaly segment is processed. One of: - 'squeeze': View an anomaly event lasting t timestamps as one timepoint. - 'log': View an anomaly event lasting t timestamps as log(t) timepoint. - 'sqrt': View an anomaly event lasting t timestamps as sqrt(t) timepoint. - 'raw': View an anomaly event lasting t timestamps as t timepoint. If using 'log', you can specify the param "base" to return the logarithm of x to the given base, calculated as log(x) / log(base).	'log'
base	int	Default is 3.	3

calc

calc(scores, labels, margins)

Returns:

Type	Description
type[MetricInterface]	A F1class (Evaluations.Metrics.F1class), including: best_f1: the value of best f1 score; precision: corresponding precision value; recall: corresponding recall value; threshold: the value of threshold when getting best f1.

EventKthPrcPA

EventKthPrcPA

EventKthPrcPA(k, mode='log', base=3)

Bases: EvalInterface

Parameters:

Name	Type	Description	Default
k	int	Defines the delay limit.	required
mode	str	Defines the scale at which the anomaly segment is processed. One of: - 'squeeze': View an anomaly event lasting t timestamps as one timepoint. - 'log': View an anomaly event lasting t timestamps as log(t) timepoint. - 'sqrt': View an anomaly event lasting t timestamps as sqrt(t) timepoint. - 'raw': View an anomaly event lasting t timestamps as t timepoint. If using 'log', you can specify the param "base" to return the logarithm of x to the given base, calculated as log(x) / log(base).	'log'
base	int	Default is 3.	3

calc

calc(scores, labels, margins)

Returns:

Type	Description
type[MetricInterface]	An Auprc instance (Evaluations.Metrics.Auprc), including: auprc: auprc value.

EventKthRocPA

EventKthRocPA

EventKthRocPA(k, mode='log', base=3)

Bases: EvalInterface

Parameters:

Name	Type	Description	Default
k	int	Defines the delay limit.	required
mode	str	Defines the scale at which the anomaly segment is processed. One of: - 'squeeze': View an anomaly event lasting t timestamps as one timepoint. - 'log': View an anomaly event lasting t timestamps as log(t) timepoint. - 'sqrt': View an anomaly event lasting t timestamps as sqrt(t) timepoint. - 'raw': View an anomaly event lasting t timestamps as t timepoint. If using 'log', you can specify the param "base" to return the logarithm of x to the given base, calculated as log(x) / log(base).	'log'
base	int	Default is 3.	3

calc

calc(scores, labels, margins)

Returns:

Type	Description
type[MetricInterface]	An Auroc instance (Evaluations.Metrics.Auroc), including: auroc: auroc value.

EventPrcPA

EventPrcPA

EventPrcPA(mode='log', base=3, figname=None)

Bases: EvalInterface

Parameters:

Name	Type	Description	Default
mode	str	Defines the scale at which the anomaly segment is processed. One of: - 'squeeze': View an anomaly event lasting t timestamps as one timepoint. - 'log': View an anomaly event lasting t timestamps as log(t) timepoint. - 'sqrt': View an anomaly event lasting t timestamps as sqrt(t) timepoint. - 'raw': View an anomaly event lasting t timestamps as t timepoint. If using 'log', you can specify the param "base" to return the logarithm of x to the given base, calculated as log(x) / log(base).	'log'
base	int	Default is 3.	3

calc

calc(scores, labels, margins)

Returns:

Type	Description
type[MetricInterface]	An Auprc instance (Evaluations.Metrics.Auprc), including: auprc: auprc value.

EventRocPA

EventRocPA

EventRocPA(mode='log', base=3, figname=None)

Bases: EvalInterface

Parameters:

Name	Type	Description	Default
mode	str	Defines the scale at which the anomaly segment is processed. One of: - 'squeeze': View an anomaly event lasting t timestamps as one timepoint. - 'log': View an anomaly event lasting t timestamps as log(t) timepoint. - 'sqrt': View an anomaly event lasting t timestamps as sqrt(t) timepoint. - 'raw': View an anomaly event lasting t timestamps as t timepoint. If using 'log', you can specify the param "base" to return the logarithm of x to the given base, calculated as log(x) / log(base).	'log'
base	int	Default is 3.	3

calc

calc(scores, labels, margins)

Returns:

Type	Description
type[MetricInterface]	An Auroc instance (Evaluations.Metrics.Auroc), including: auroc: auroc value.

PointAuprcPA

PointAuprcPA

PointAuprcPA()

Bases: EvalInterface

Using Point-based point-adjustment Auprc to evaluate the models.

calc

calc(scores, labels, margins)

Returns:

Type	Description
type[MetricInterface]	An Auprc instance (Evaluations.Metrics.Auprc), including: auprc: auprc value.

PointAurocPA

PointAurocPA

PointAurocPA(figname=None)

Bases: EvalInterface

Using Point-based point-adjustment Auroc to evaluate the models.

calc

calc(scores, labels, margins)

Returns:

Type	Description
type[MetricInterface]	An Auroc instance (Evaluations.Metrics.Auroc), including: auroc: auroc value.

PointF1

PointF1

PointF1()

Bases: EvalInterface

Using Traditional F1 score to evaluate the models.

calc

calc(scores, labels, margins)

Returns:

Type	Description
type[MetricInterface]	A F1class (Evaluations.Metrics.F1class), including: best_f1: the value of best f1 score; precision: corresponding precision value; recall: corresponding recall value;

PointF1PA

PointF1PA

PointF1PA()

Bases: EvalInterface

Using Point-based point-adjustment F1 score to evaluate the models.

calc

calc(scores, labels, margins)

Returns:

Type	Description
type[MetricInterface]	A F1class (Evaluations.Metrics.F1class), including: best_f1: the value of best f1 score; precision: corresponding precision value; recall: corresponding recall value; threshold: the value of threshold when getting best f1.

PointKthF1PA

PointKthF1PA

PointKthF1PA(k)

Bases: EvalInterface

Using Point-based point-adjustment F1 score to evaluate the models under k-delay strategy.

calc

calc(scores, labels, margins)

Returns:

Type	Description
type[MetricInterface]	A F1class (Evaluations.Metrics.F1class), including: best_f1: the value of best f1 score; precision: corresponding precision value; recall: corresponding recall value; threshold: the value of threshold when getting best f1.

PointPrc

PointPrc

PointPrc()

Bases: EvalInterface

Using traditional Auprc to evaluate the models.

calc

calc(scores, labels, margins)

Returns:

Type	Description
type[MetricInterface]	An Auprc instance (Evaluations.Metrics.Auprc), including: auprc: auprc value.

PointRoc

PointRoc

PointRoc()

Bases: EvalInterface

Using traditional Auroc to evaluate the models.

calc

calc(scores, labels, margins)

Returns:

Type	Description
type[MetricInterface]	An Auroc instance (Evaluations.Metrics.Auroc), including: auroc: auroc value.

VUS

VUS_PR

VUS_PR()

Bases: EvalInterface

calc

calc(scores, labels, margins)

Returns:

Type	Description
MetricInterface	An Auroc instance (Evaluations.Metrics.Auroc), including: auroc: auroc value.

VUS_ROC

VUS_ROC()

Bases: EvalInterface

calc

calc(scores, labels, margins)

Returns:

Type	Description
MetricInterface	An Auroc instance (Evaluations.Metrics.Auroc), including: auroc: auroc value.

metricor

metricor(a=1, probability=True, bias='flat')

existence_reward

existence_reward(labels, preds)

labels: list of ordered pair preds predicted data

labels_conv

labels_conv(preds)

return indices of predicted anomaly

labels_conv_binary

labels_conv_binary(preds)

return predicted label

metric_new

metric_new(label, score, plot_ROC=False, alpha=0.2)

input

Real labels and anomaly score in prediction

output

AUC, Precision, Recall, F-score, Range-precision, Range-recall, Range-Fscore, Precison@k,

k is chosen to be # of outliers in real labels

range_convers_new

range_convers_new(label)

input: arrays of binary values output: list of ordered pair [[a0,b0], [a1,b1]... ] of the inputs