ML_SCORE
scores a model by
generating predictions using the feature columns in a labeled
dataset as input and comparing the predictions to ground truth
values in the target column of the labeled dataset.
You cannot score a model with a topic modeling task type.
The dataset used with ML_SCORE
should have the same feature columns as the dataset used to
train the model, but the data sample should be different from
the data used to train the model. For example, you might
reserve 20 to 30 percent of a labeled dataset for scoring.
ML_SCORE
returns a computed
metric indicating the quality of the model. A value of
None
is reported if a score for the
specified or default metric cannot be computed. If an invalid
metric is specified, the following error message is reported:
Invalid data for the metric. Score could not be
computed.
Models with a low score can be expected to perform poorly, producing predictions and explanations that cannot be relied upon. A low score typically indicates that the provided feature columns are not a good predictor of the target values. In this case, consider adding more rows or more informative features to the training dataset.
You can also run ML_SCORE
on
the training dataset and a labeled test dataset and compare
results to ensure that the test dataset is representative of
the training dataset. A high score on a training dataset and
low score on a test dataset indicates that the test data set
is not representative of the training dataset. In this case,
consider adding rows to the training dataset that better
represent the test dataset.
MySQL HeatWave AutoML supports a variety of scoring metrics to help you
understand how your model performs across a series of
benchmarks.
The metric you select to score the model must be compatible
with the task
type and the target data. See
Optimization and Scoring
Metrics.
Before running ML_SCORE
, you
must train, and then load the trained model you want to use
for scoring.
-
The following example trains a dataset with the classification machine learning task.
mysql> CALL sys.ML_TRAIN('census_data.census_train', 'revenue', JSON_OBJECT('task', 'classification'), @census_model);
-
The following example loads the trained model.
mysql> CALL sys.ML_MODEL_LOAD(@census_model, NULL);
For more information about training and loading models, see Train a Model and Load a Model.
After training and loading the model, prepare a table of labeled data to score that has a different set of data from the trained model. This is considered the validation dataset. For parameter and option descriptions, see ML_SCORE.
To score a model, run the
ML_SCORE
routine.
mysql> CALL sys.ML_SCORE(table_name, target_column_name, model_handle, metric, score, [options]);
The following example uses the accuracy
metric to compute model quality:
mysql> CALL sys.ML_SCORE('census_data.census_validate', 'revenue', @census_model, 'accuracy', @score, NULL);
Where:
census_data.census_validate
is the fully qualified name of the validation dataset table (schema_name.table_name
).revenue
is the name of the target column containing ground truth values.@census_model
is the session variable that contains the model handle.accuracy
is the scoring metric. For other supported scoring metrics, see Optimization and Scoring Metrics.@score
is the user-defined session variable that stores the computed score. TheML_SCORE
routine populates the variable. User variables are written as@
. The examples in this guide usevar_name
@score
as the variable name. Any valid name for a user-defined variable is permitted, for example@my_score
.NULL
sets no options for the routine. To view available options, see ML_SCORE.
To retrieve the computed score, query the
@score
session variable.
mysql> SELECT @score;
+--------------------+
| @score |
+--------------------+
| 0.8888888955116272 |
+--------------------+
1 row in set (0.0409 sec)
Review the score value and determine if the trained model is reliable enough for generating predictions and explanations.
Review ML_SCORE for parameter descriptions and options.
Review Machine Learning Use Cases to create machine learning models with sample datasets.