WL#10237: X Protocol expectations for supported protobuf fields

Affects: Server-8.0   —   Status: Complete   —   Priority: Medium

Motivation
==========

Protobuf allows forward compatible message: If a field-id isn't known,
it is ignored. A newer server that returns data to the client it doesn't
know will just work. The client ignores the fields and handles what it
understands.

If the behavior doesn't fit the use-case a high-level mechanism is need
to announce/negotiate with the other side.

General
=======

This functionality is going to help with detecting compatibility problems
between client application and MySQL Server, when server receives X Protocol
message containing field that it doesn't know.


Details
=======

The compatibility problem originates from 'protobufs' assumption that
each node exchanging messages should be able to handle messages containing
unknown data (different versions). In case of server application skipping
unknown data can change the meaning which client "had" when he was sending
the message. The assumption enforces users of 'protobuf' to design/implement
their own mechanism for compatibility verification.

'protobuf' 2 stores unknown data inside "Message/MessageLite" class, which could
be used to implement a runtime validation of received message.
To implement proper check on using "unknown data", X Plugin would need a
protobuf plugin to generate additional checks which would go deep inside
all submessages and on all its fields. In protobuf 3, unknown fields are
removed from the generated API/code.

Instead of doing validation when message is being received, this worklog is
going to introduce new "condition key" to X Protocol expectation mechanism.
The client is going to send an "Expect.Open" message containing message/fields
tag chain and server is going to validate if field specified this way is present
inside the definition of servers X Protocol message. This is done to ensure
pipelining: message processing should be stopped when any message does not meet
the expectation.
Functional requirements
=======================

General
-------

1. User MUST be able to use message with fields unsupported by the server.
2. User MUST be able to perform a check whether a specific field is supported
   by X Plugin (server)
3. User MUST be able to use check if multiple fields are supported by
   X Plugin (server)
4. Querying if the server supports message fields MUST not introduce an
   extra round-trip when used with pipelining.
5. A specific error message should be generated when a check fails.

Expect.Open/Close
-----------------

6. User MUST be able to fail whole expectation block (starting from Expect.Open)
   when its condition contains at last one unsupported field.


Non functional requirements
===========================

1. There should be no degradation of X Plugin performance
2. Code coverage shouldn't be decreased (in percent)
Interface Protocol
==================

X Protocol messages are defined inside 'proto' files. Any change to a message
defined inside 'proto' file can break compatibility between different versions
of X Plugin and X Connector. This worklog is going to introduce a mechanism
that allows to check that concrete field (messages field) is supported by
concrete version of X Plugin.
The mechanism is going to be implemented using X Protocols expectation block:

    Client -> Server: Mysqlx.Expect.Open([+field_exists=12.1.1])
    Client <- Server: Mysqlx.Ok
    ...X Connector requests, supported by X Plugin...
    Client -> Server: Mysqlx.Expect.Close()
    Client <- Server: Mysqlx.Ok

"Mysqlx.Expect.Open" uses conditions key to enable/disable expectations.
To check if field is handled by X Plugin, following constant/key must be added:

* FIELD_EXISTS = 2

The conditions value for the key must be non empty value, defined as chain of
protobuf field-tags separated by "dots". The first element inside the chain is
going to be X Protocol message id. For example to check if:

* "StmtExecute" is supported, value should be "12" (because ID of the
  StmtExecute message is 12)
* "StmtExecute.compact_metadata" is supported, value should be "12.4"
* "StmtExecute.args.obj" is supported, value should be "12.2.3"

The dependencies between messages can be represented as a graph. Cycles inside
the 'graph' are going to be overcome by allowing user to close the cycle but
it must be the last field-tag inside the chain.
In other words user is allowed to add relations that close the cycle but if the
user would like to check a field after the cycle closure, he must check it at
first occurrence of the type.

For example:

* "StmtExecute.args.obj.fld.value.obj" - "12.2.3.1.2.3" is not allowed because
  it checks the type "Any" two times. To check if "Obj" is supported by "Any"
  it is sufficient to execute check it as:
  "StmtExecute.args.obj" - "12.2.3"

Expectation open request which contains a check for a field that is not handled
by X Plugin is going to be "installed" on expectations stack, still and error
is send back. All clients requests which are done inside this expectation block
are not executed and in return the "expectation block error" is sent for each
response:

    Client -> Server: Mysqlx.Expect.Open([field=unknown])
    Client <- Server: Mysqlx.Error

    loop all client requests inside expectation block
      Client -> Server: Any message
      Client <- Server: Mysqlx.Error
    end

    Client -> Server: Mysqlx.Expect.Close()
    Client <- Server: Mysqlx.Error


Handling of unknown fields
--------------------------

This 'worklog' doesn't introduce any additional handling of unknown fields or
values. X Plugin is going to process/execute such message thus its still
possible to break the compatibility when client doesn't check if field is
supported.

Compatibility
-------------

Specification of a feature (that introduces a field inside an existing
'protobuf' message) must say that before feature usage it is necessary to check
with expectation condition "field_exists" if "tag-id-chain-to-new-field" is
supported. The check must be implemented inside connector code and executed
before using the feature.


Error and Warnings
==================

The expectation message flow (behavior and its error codes) is already
implemented by X Plugin , still "field-existance" expectation check
must use new error code:

|Property    |Value                                         |
|------------|----------------------------------------------|
|Name        |ER_X_EXPECT_FIELD_EXISTS_FAILED               |
|Error code  |5159                                          |
|Error text  |Expectation failed: field_exists              |

and old code that we used with condition key "no-error" must be renamed:

|Property    |Value                                         |
|------------|----------------------------------------------|
|Name        |ER_X_EXPECT_FAILED                            |
|Name        |ER_X_EXPECT_NO_ERROR_FAILED                   |
|Error code  |5168                                          |
|Error text  |Expectation failed: no_error                  |
Field-tag-chains
================

Currently X Protocol messages are compiled in two passes: full and lite.
The 'field chain' validation isn't doable on lite version of X Protocol
messages. To be able to validate it with both library types, X Plugin needs to
define its own "protoc plugin". The new plugin is going to generate the
"field-tags-chains". To do fully autonomic chain generation (by protoc plugin),
X Protocol top level messages needs to be marked with "message-ids" using
protobuf custom options:


```  git diff
--- a/rapid/plugin/x/protocol/mysqlx.proto
+++ b/rapid/plugin/x/protocol/mysqlx.proto
@@ -24,8 +24,9 @@ syntax = "proto2";
 
 package Mysqlx;
 option java_package = "com.mysql.cj.mysqlx.protobuf";
 
+import "google/protobuf/descriptor.proto"; // comment_out_if LITE_RUNTIME
 
 // style-guide:
 //
 // see https://developers.google.com/protocol-buffers/docs/style
@@ -102,13 +103,20 @@ message ServerMessages {
     SQL_STMT_EXECUTE_OK = 17;
     RESULTSET_FETCH_DONE_MORE_OUT_PARAMS = 18;
   };
 }
-
+// ifndef PROTOBUF_LITE
+extend google.protobuf.MessageOptions {
+  optional ClientMessages.Type client_message_id = 100001;
+  optional ServerMessages.Type server_message_id = 100002;
+}
+// endif
 
 // generic Ok message
 message Ok {
   optional string msg = 1;
+
+  option (server_message_id) = OK; // comment_out_if LITE_RUNTIME
 }
 
 
 // generic Error message
@@ -133,5 +141,7 @@ message Error {
   enum Severity {
     ERROR = 0;
     FATAL = 1;
   };
+
+  option (server_message_id) = ERROR; // comment_out_if LITE_RUNTIME
 }

```

When "protoc plugin" gets a message which has 'client_message_id' option set,
it should generate "field-tag-chains" for all its fields, recursively going
through submessages. Recursion is stopped when a cycle is detected.

"protoc plugins" that use custom options need to heave already compiled
protocol files. Thus the chain generation needs to be done in separate pass,
where the compilation process is going to change from:

1. compilation of full-version-proto files
2. compilation of lite-version-proto files

to:

1. compilation of full-version-proto files (C++ source generation)
2. compilation of protoc-plugin
3. compilation of full-version-proto files with protoc-plugin (C++ header with
   tag-chains)
4. compilation of lite-version-proto files (C++ source generation)

The "C++" header with tag-chains is generated using full version of 'proto'
files, but still both full and lite X Plugins files can use it.

CMake generates lite version of proto files from full by uncommenting lines like:

``` proto
// ifdef PROTOBUF_LITE: option optimize_for = LITE_RUNTIME;
```

The mechanism of uncommenting and commenting out needs to be improved to be able
to remove custom options from lite version of proto files. It must handle:

* commenting out single line

``` proto
import "google/protobuf/descriptor.proto"; // comment_out_if LITE_RUNTIME
```

* commenting out a block

``` proto
// ifndef PROTOBUF_LITE
extend google.protobuf.MessageOptions {
  optional ClientMessages.Type client_message_id = 100001;
  optional ServerMessages.Type server_message_id = 100002;
}
// endif
```


Expectation blocks
==================

Condition keys are defined as integer field in "Mysqlx.Expect.Open.Condition",
without any description of supported values. New enumeration was introduces to
generate constants which can be used as values for "condition key" to remove
magic numbers:

--- a/rapid/plugin/x/protocol/mysqlx_expect.proto
+++ b/rapid/plugin/x/protocol/mysqlx_expect.proto
@@ -17,8 +17,10 @@
  * 02110-1301  USA
  */
 syntax = "proto2";
 
+import "mysqlx.proto"; // comment_out_if PROTOBUF_LITE
+
 // ifdef PROTOBUF_LITE: option optimize_for = LITE_RUNTIME;
 
 // Expect operations
 package Mysqlx.Expect;
@@ -74,8 +76,14 @@ option java_package = "com.mysql.cj.mysqlx.protobuf";
 // :returns: :protobuf:msg:`Mysqlx::Ok` on success,
:protobuf:msg:`Mysqlx::Error` on error
 //
 message Open {
   message Condition {
+    enum Key {
+      // Change error propagation behaviour
+      EXPECT_NO_ERROR = 1;
+      // Check if X Protocol field exists
+      EXPECT_FIELD_EXIST = 2;
+    };


Testing
=======

Expectation block must be covered in MTR tests and with unit tests.