4.6.4.7 Scoring an Anomaly Detection Model

After generating predictions, you can score the model to assess its reliability. For a list of scoring metrics you can use with anomaly detection models, see Anomaly Detection Metrics. For this use case, you use the test dataset for validation. In a real-world use case, you should use a separate validation dataset that has the target column and ground truth values for the scoring validation. You should also use a larger number of records for training and validation to get a valid score.

To generate a score, the target_column_name column must only contain the anomaly scores as an integer: 1 for an anomaly, or 0 for normal.

Before You Begin

Complete the following tasks:

• Prepare Data for an Anomaly Detection Model

• Train an Anomaly Detection Model

• Generate Predictions for an Anomaly Detection Model

Requirements for Scoring Models

If you run ML_SCORE with the log_anomaly_detection task, at least one column must act as the primary key to establish the temporal order of logs.

Scoring a Semi-Supervised Anomaly Detection Model

1. If not already done, load the model. You can use the session variable for the model that is valid for the duration of the connection. Alternatively, you can use the model handle previously set. For the option to set the user name, you can set it to NULL.

   The following example uses the session variable.

   mysql> CALL sys.ML_MODEL_LOAD(@model, NULL);

   The following example uses the model handle.

   mysql> CALL sys.ML_MODEL_LOAD('anomaly_detection_semi_supervised_use_case', NULL);

2. Score the model with the ML_SCORE routine and use the accuracy metric.

   mysql> CALL sys.ML_SCORE(table_name, target_column_name, model_handle, metric, score, [options]);

   Replace table_name, target_column_name, model_handle, metric, score with your own values.

   The following example runs ML_SCORE on the testing dataset previously created.

   mysql> CALL sys.ML_SCORE('anomaly_data.credit_card_test', 'target', 'anomaly_detection_semi_supervised_use_case', 
                             'accuracy', @anomaly_score, NULL);

   Where:

   • anomaly_data.credit_card_test is the fully qualified name of the validation dataset.

   • target is the target column name with ground truth values.

   • 'anomaly_detection_semi_supervised_use_case' is the model handle for the trained model.

   • accuracy is the selected scoring metric.

   • @anomaly_score is the session variable name for the score value.

   • NULL means that no other options are defined for the routine.

3. Retrieve the score by querying the @score session variable.

   mysql> SELECT @anomaly_score;
   +--------------------+
   | @anomaly_score     |
   +--------------------+
   | 0.6499999761581421 |
   +--------------------+
   1 row in set (0.0481 sec)

4. If done working with the model, unload it with the ML_MODEL_UNLOAD routine.

   mysql> CALL sys.ML_MODEL_UNLOAD('anomaly_detection_semi_supervised_use_case');

   To avoid consuming too much memory, it is good practice to unload a model when you are finished using it.

Scoring an Unsupervised Anomaly Detection Model for Log Data

Even though you score an unsupervised model, you must provide a labeled dataset for generating a score.

1. If not already done, load the model. You can use the session variable for the model that is valid for the duration of the connection. Alternatively, you can use the model handle previously set. For the option to set the user name, you can set it to NULL.

   The following example uses the session variable.

   mysql> CALL sys.ML_MODEL_LOAD(@model, NULL);

   The following example uses the model handle.

   mysql> CALL sys.ML_MODEL_LOAD('anomaly_detection_log_use_case', NULL);

2. Score the model with the ML_SCORE routine and use the accuracy metric.

   mysql> CALL sys.ML_SCORE(table_name, target_column_name, model_handle, metric, score, [options]);

   Replace table_name, target_column_name, model_handle, metric, score with your own values.

   The following example runs ML_SCORE on the testing dataset previously created.

   mysql> CALL sys.ML_SCORE('anomaly_log_data.testing_data', 'target', 'anomaly_detection_log_use_case', 
                             'f1', @anomaly_log_score, NULL);

   Where:

   • anomaly_log_data.testing_data is the fully qualified name of the validation dataset.

   • target is the target column name with ground truth values.

   • 'anomaly_detection_log_use_case' is the model handle for the trained model.

   • f1 is the selected scoring metric.

   • @anomaly_log_score is the session variable name for the score value.

   • NULL means that no other options are defined for the routine.

3. Retrieve the score by querying the @score session variable.

   mysql> SELECT @anomaly_log_score;
   +--------------------+
   | @anomaly_log_score |
   +--------------------+
   | 0.8571428656578064 |
   +--------------------+
   1 row in set (0.0452 sec)

4. If done working with the model, unload it with the ML_MODEL_UNLOAD routine.

   mysql> CALL sys.ML_MODEL_UNLOAD('anomaly_detection_log_use_case');

   To avoid consuming too much memory, it is good practice to unload a model when you are finished using it.

What's Next

• Review other Machine Learning Use Cases.