4.6.5.10 Scoring a Recommendation Model

After generating predicted ratings/rankings and recommendations, you can score the model to assess its reliability. For a list of scoring metrics you can use with recommendation models, see Recommendation Model Metrics. For this use case, you use the test dataset for validation. In a real-world use case, you should use a separate validation dataset that has the target column and ground truth values for the scoring validation. You should also use a larger number of records for training and validation to get a valid score.

Before You Begin

Review and complete the following tasks:

• Prepare Data for a Recommendation Model

• Train a Recommendation Model

• Generate Predictions for a Recommendation Model

• Generate Predictions for Ratings and Rankings

• Generate Item Recommendations for Users

• Generate User Recommendations for Items

• Generate Recommendations for Similar Items

• Generate Recommendations for Similar Users

Options for Scoring Recommendation Models

The options for ML_SCORE include the following:

• threshold: The optional threshold that defines positive feedback, and a relevant sample. Only use with ranking metrics. It can be used for either explicit or implicit feedback.

• topk: The optional top number of recommendations to provide. The default is 3. Set a positive integer between 1 and the number of rows in the table.

  A recommendation task and ranking metrics can use both threshold and topk.

• remove_seen: If the input table overlaps with the training table, and remove_seen is true, then the model will not repeat existing interactions. The default is true. Set remove_seen to false to repeat existing interactions from the training table.

• item_metadata: Defines the table that has item descriptions. It is a JSON object that has the table_name option as a key, which specifies the table that has item descriptions. One column must be the same as the item_id in the input table.

• user_metadata: Defines the table that has user descriptions. It is a JSON object that has the table_name option as a key, which specifies the table that has user descriptions. One column must be the same as the user_id in the input table.

  • table_name: To be used with the item_metadata and user_metadata options. It specifies the table name that has item or user descriptions. It must be a string in a fully qualified format (schema_name.table_name) that specifies the table name.

Scoring the Model

1. If not already done, load the model. You can use the session variable for the model that is valid for the duration of the connection. Alternatively, you can use the model handle previously set. For the option to set the user name, you can set it to NULL.

   The following example uses the session variable.

   mysql> CALL sys.ML_MODEL_LOAD(@model, NULL);

   The following example uses the model handle.

   mysql> CALL sys.ML_MODEL_LOAD('recommendation_use_case', NULL);

2. Score the model with the ML_SCORE routine and use the precision_at_k metric.

   mysql> CALL sys.ML_SCORE(table_name, target_column_name, model_handle, metric, score, [options]);

   Replace table_name, target_column_name, model_handle, metric, score with your own values.

   The following example runs ML_SCORE on the testing dataset previously created.

   mysql> CALL sys.ML_SCORE('recommendation_data.testing_dataset', 'rating', @model, 'precision_at_k', @recommendation_score, NULL);

   Where:

   • recommendation_data.testing_dataset is the fully qualified name of the validation dataset.

   • rating is the target column name with ground truth values.

   • @model is the session variable for the model handle.

   • precision_at_k is the selected scoring metric.

   • @recommendation_score is the session variable name for the score value.

   • NULL means that no other options are defined for the routine.

3. Retrieve the score by querying the @score session variable.

   mysql> SELECT @recommendation_score;
   +-----------------------+
   | @recommendation_score |
   +-----------------------+
   | 0.23333333432674408  |
   +-----------------------+

4. If done working with the model, unload it with the ML_MODEL_UNLOAD routine.

   mysql> CALL sys.ML_MODEL_UNLOAD('recommendation_use_case');

   To avoid consuming too much memory, it is good practice to unload a model when you are finished using it.

What's Next

• Review other Machine Learning Use Cases.