4.5.7 Scoring a Model

ML_SCORE scores a model by generating predictions using the feature columns in a labeled dataset as input and comparing the predictions to ground truth values in the target column of the labeled dataset.

You cannot score a model with a topic modeling task type.

Before You Begin

• Review the following:

  • Prepare Data

  • Train a Model

  • Load a Model

  • Generate Predictions

  • Generate Model Explanations

  • Generate Prediction Explanations

ML_SCORE Overview

The dataset used with ML_SCORE should have the same feature columns as the dataset used to train the model, but the data sample should be different from the data used to train the model. For example, you might reserve 20 to 30 percent of a labeled dataset for scoring.

ML_SCORE returns a computed metric indicating the quality of the model. A value of None is reported if a score for the specified or default metric cannot be computed. If an invalid metric is specified, the following error message is reported: Invalid data for the metric. Score could not be computed.

Models with a low score can be expected to perform poorly, producing predictions and explanations that cannot be relied upon. A low score typically indicates that the provided feature columns are not a good predictor of the target values. In this case, consider adding more rows or more informative features to the training dataset.

You can also run ML_SCORE on the training dataset and a labeled test dataset and compare results to ensure that the test dataset is representative of the training dataset. A high score on a training dataset and low score on a test dataset indicates that the test data set is not representative of the training dataset. In this case, consider adding rows to the training dataset that better represent the test dataset.

AutoML supports a variety of scoring metrics to help you understand how your model performs across a series of benchmarks. The metric you select to score the model must be compatible with the task type and the target data. See Optimization and Scoring Metrics.

Preparing to Score a Model

Before running ML_SCORE, you must train, and then load the trained model you want to use for scoring.

1. The following example trains a dataset with the classification machine learning task.

   mysql> CALL sys.ML_TRAIN('census_data.census_train', 'revenue', JSON_OBJECT('task', 'classification'), @census_model);

2. The following example loads the trained model.

   mysql> CALL sys.ML_MODEL_LOAD(@census_model, NULL);

For more information about training and loading models, see Train a Model and Load a Model.

After training and loading the model, prepare a table of labeled data to score that has a different set of data from the trained model. This is considered the validation dataset. For parameter and option descriptions, see ML_SCORE.

Scoring a Model

To score a model, run the ML_SCORE routine.

mysql> CALL sys.ML_SCORE(table_name, target_column_name, model_handle, metric, score, [options]);

The following example uses the accuracy metric to compute model quality:

mysql> CALL sys.ML_SCORE('census_data.census_validate', 'revenue', @census_model, 'accuracy', @score, NULL);

Where:

• census_data.census_validate is the fully qualified name of the validation dataset table (schema_name.table_name).

• revenue is the name of the target column containing ground truth values.

• @census_model is the session variable that contains the model handle.

• accuracy is the scoring metric. For other supported scoring metrics, see Optimization and Scoring Metrics.

• @score is the user-defined session variable that stores the computed score. The ML_SCORE routine populates the variable. User variables are written as @var_name. The examples in this guide use @score as the variable name. Any valid name for a user-defined variable is permitted, for example @my_score.

• NULL sets no options for the routine. To view available options, see ML_SCORE.

To retrieve the computed score, query the @score session variable.

mysql> SELECT @score;
+--------------------+
| @score             |
+--------------------+
| 0.8888888955116272 |
+--------------------+
1 row in set (0.0409 sec)

Review the score value and determine if the trained model is reliable enough for generating predictions and explanations.

What's Next

• Review ML_SCORE for parameter descriptions and options.

• Review Machine Learning Use Cases to create machine learning models with sample datasets.