WL#4162: Add start/end of trx/stm to binlog

Executive Summary
=================
This worklog provides information at each event logged to binary log
about transactions boundaries by setting markers in the logged event.

These markers use three single bits of the Log_event Common-Header
flags.

By using single bits of the Log_event Common-Header flags this feature
will need no additional space in binary log to store these markers.

With such markers, by reading a single Log_event header from an event
stream, it will be possible to state if the event:

a) is from a server with the feature enabled;
b) has no transaction boundary information;
c) is a self contained event;
d) is starting a transaction;
e) is inside a transaction;
f) is ending a transaction;
g) is in the middle of a transaction;
h) should not be considered in the transaction boundary parsing.

Before this worklog it was needed to parse a set of events in an event
stream to get information about transactions boundaries. The old
method will still be used when dealing with events without boundary
markers.

After this worklog, this feature will be always enabled. There will
be no option to disable tagging of transaction boundary markers in
MySQL server. 

References
==========
Bug#17943188
BUG#26395, WL#3735

Functional requirements
=======================

F1) Markers should be written to the binary log.

F2) Markers should be visible as comments in mysqlbinlog output.

F3) BINLOG statements inner events are left untouched in respect
    to transaction boundary markers.


Non-Functional requirements
===========================

NF1) The feature must be supported regardless the use of a specific
     binary log format.

NF2) The feature must be supported regardless the use of GTIDs.

Concepts and Definitions
========================

C1) Transaction Stream/Transaction Event Group/Commit Group: a sequence
    of events related to a single transaction.

D1) TBM: Transaction Boundaries Markers. The bits in the binary log
    event common-header in with a transaction boundary type will be
    written.

D2) Transaction Boundary Type: the type of transaction boundary of a
    single event in a server with the transaction boundaries feature
    enabled.

    Of the eight possible types (three bits), this worklog is using
    six, letting the remaining two types available for future use.

    The transaction boundary types defined in this worklog are:

  D2.0) Not Defined:
        The event was logged by a server without the feature.

  D2.1) Ignore Boundary:
        The event can appear in the middle of a transaction and should
        be ignored by the transaction boundary parser. This is the case
        of fake Rotate events sent by the dump_thread when a slave
        connects asking data.
        Examples: Format_description, Rotate.

  D2.2) Self Contained:
        Any event that is self contained (doesn't depend on previous or
        following events).
        Examples:
        D2.1a) Any Query event with an auto-commit statement (DDL for
               example) if GTIDs are disabled and without Intvar, Rand
               or User_var related events;
        D2.1b) Start, Stop, Rotate, Previous_gtids, etc.

  D2.3) Start Transaction:
        Any event starting a Transaction Stream.

  D2.4) Inside Transaction:
        Any event that is part of a Transaction Stream between the
        event that starts and the event that ends the stream.

  D2.5) End Transaction:
        Any event ending a Transaction Stream.



High-Level Specification
========================

Boundary Type
-------------

Represents a transaction boundary type.

Every Log_event object have a boundary type.

The boundary type for an event will be determined by the event type,
but it can be changed before writing to binary log depending on:

 a) Use of GTIDs;

 b) Dependency on Intvar, Rand or User_var events;

A slave I/O thread reading events from the master can use the event's
boundary type to verify if the event is the middle of a Transaction
Stream. This will be important to WL#7349 and WL#7355.

Transaction Boundary Parser
---------------------------

Responsible for keeping track of transactions boundaries in an event
stream and also for assuring that the event stream is valid.  

A server with this feature enabled will expect that a valid stream of
events follows the rules defined below:

BOUNDARY_TYPE_NOT_DEFINED
(
 (BOUNDARY_TYPE_SELF_CONTAINED|BOUNDARY_TYPE_IGNORE_BOUNDARY)*
 BOUNDARY_TYPE_START_TRANSACTION
 (BOUNDARY_TYPE_INSIDE_TRANSACTION|BOUNDARY_TYPE_IGNORE_BOUNDARY)*
 BOUNDARY_TYPE_END_TRANSACTION
)*

Where ()* means 0 or more repetitions of the types. 

The Transaction Boundary Parser have implemented the rules to support
the verification of a event stream either by a slave I/O thread
reading events from the master or by the mysqlbinlog client program
reading events from a binlog file.

            Transaction Boundary Parser Transition Table 
 ====================================================================
 |From boundary type ->|   0    |   2    |   3    |   4    |   5    |
 |                     |  Not   |Self Con| Start  | Inside |  End   |
 |To boundary type     |Defined | tained | Trans  | Trans  | Trans  |
 |=====================|========|========|========|========|========|
 |0) Not Defined       |      This is the parser starting type      |*1
 |---------------------|--------|--------|--------|--------|--------|
 |1) Ignore Boundary   |   The parser is never set to this type     |*2
 |---------------------|--------|--------|--------|--------|--------|
 |2) Self Contained    |   Ok   |   Ok   |  Err   |  Err   |   Ok   |
 |---------------------|--------|--------|--------|--------|--------|
 |3) Start Transaction |   Ok   |   Ok   |  Err   |  Err   |   Ok   |
 |---------------------|--------|--------|--------|--------|--------|
 |4) Inside Transaction|  Err   |  Err   |   Ok   |   Ok   |  Err   |
 |---------------------|--------|--------|--------|--------|--------|
 |5) End Transaction   |  Err   |  Err   |   Ok   |   Ok   |  Err   |
 ====================================================================
*1 - When feeding the transaction boundary parser with an event with
     BOUNDARY_TYPE_NOT_DEFINED in its markers, the transaction
     boundary parser will define (but will not mark) the event boundary
     type based on same assumptions the master do for marking.
     So, we can have a server without this feature replicating to a
     slave with the feature and, instead of only reading the markers
     (the easy way), the slave will "guess" the boundary type of the
     events to take benefit of the feature.
*2 - When feeding the transaction boundary parser with an event with
     BOUNDARY_TYPE_IGNORE_BOUNDARY in its markers, the transaction
     boundary parser will keep its current type.

A transaction boundary parser must be "reset()" every time the server
move from a current event stream to another one (example: when the
slave server connects to a new master). This reset procedure will move
the boundary type of the parser to BOUNDARY_TYPE_NOT_DEFINED.

When reading events from a master server (or from a binlog file), the
Master_info boundary parser (or the mysqlbinlog client program's one)
will be fed with the buffers of the event (this contains the event
header and, off course, the transaction boundary markers).

If the stream is fed with an event that result in an incorrect
transition in the state machine (examples: Self -> End Transaction or 
End Transaction -> Inside Transaction), a warning/error message I2/I3
described in the Interface Specification will be logged.

What this message means to the user: if a slave connects to a master
and asks events pointing an binlog position in the middle of a
transaction, the error will be thrown (probably stating that it was
"Unable to change boundary parser from Not Defined to Inside
Transaction)". 

How can the user fix things so that he does not get the message again:
In the example above, the user can check where the transaction is
starting (using mysqlbinlog client program) and try to position the
slave connection correctly using CHANGE MASTER TO. 

Interface Specification
=======================

I1) mysqlbinlog client program dump will show the boundary type for
    the events.

I2) A new warning message can be logged by the slave server if the I/O
    thread fails to feed an event into the transaction boundary
    parser. This could happen if the event boundary type breaks the
    rules of the boundary parser. It will also make an assert to fail
    in debug builds.

I3) A new warning message can be logged by mysqlbinlog client program
    if a break in the transaction boundary parser rules happens when
    an event from an event stream is fed.

Low-Level Design Specification
==============================

Transaction boundary types for log event types
==============================================

Log_event type                   |State | Set by
---------------------------------+------+------------------------------
Log_event                        |SELF  | Log_event constructor
  Query_log_event                |SELF  | Log_event constructor      *1
           BEGIN                 |START | Query_log_event constructor
           COMMIT/ROLLBACK       |END   | Query_log_event constructor
           not immediate logging |INSIDE| Query_log_event constructor
           BEGIN                 |INSIDE| binlog write_event         *2
           immediate logging     |END   | binlog write_event         *2
    Execute_load_query_log_event |      | Query_log_event constructor*3
  Load_log_event                 |SELF  | Log_event constructor      *1
    Create_file_log_event        |SELF  | Log_event constructor      *1
  Start_log_event_v3             |SELF  | Log_event constructor      *1
    Format_description_log_event |IGNORE| Log_event constructor
  Intvar_log_event               |INSIDE| Intvar constructor
  Intvar_log_event               |START | binlog write_event         *4
  Rand_log_event                 |INSIDE| Rand_log_event constructor
  Rand_log_event                 |START | binlog write_event         *4
  User_var_log_event             |INSIDE| User_var constructor
  User_var_log_event             |START | binlog write_event         *4
  Xid_log_event                  |END   | Xid_log_event constructor
  Stop_log_event                 |SELF  | Log_event constructor      *1
  Rotate_log_event               |SELF  | Log_event constructor      *1
  Rotate_log_event               |IGNORE| Log_event constructor      *5
  Append_block_log_event         |INSIDE| Append_block... constructor
    Begin_load_query_log_event   |INSIDE| Append_block... constructor
  Delete_file_log_event          |SELF  | Log_event constructor      *1
  Execute_load_log_event         |SELF  | Log_event constructor      *1
  Unknown_log_event              |SELF  | Log_event constructor      *1
  Table_map_log_event            |INSIDE| Table_map constructor
  Rows_log_event                 |INSIDE| Rows_log_event constructor
    Write_rows_log_event         |INSIDE| Rows_log_event constructor
    Update_rows_log_event        |INSIDE| Rows_log_event constructor
    Delete_rows_log_event        |INSIDE| Rows_log_event constructor
  Incident_log_event             |SELF  | Log_event constructor      *1
  Ignorable_log_event            |IGNORE| Ignorable_log... constructor
    Rows_query_log_event         |INSIDE| Rows_query_log... constructor
  Heartbeat_log_event            |SELF  | Log_event constructor      *1
  Gtid_log_event                 |START | Gtid_log_event constructor
  Previous_gtid_log_event        |SELF  | Log_event constructor      *1
=======================================================================
*1 - No special code was added to these events.
     The boundary type is "inherited" from Log_event. 
*2 - See binlog.cc modification M1.
*3 - The state will be set by Query_log_event constructor.
*4 - See binlog.cc modification M2.
*5 - See rpl_binlog_sender.cc modification M1.

Note: not sure yet about boundary type for Incident_log_event.

Modifications in source files
=============================

rpl_trx_boundary_parser.h/rpl_trx_boundary_parser.cc (new files)
----------------------------------------------------------------

M1) Defined the boundary markers macros for dealing with event flags
    bitwise operations, the possible boundary types and their names.

M2) Defined the Transaction_boundary_parser class.


log_event.h
-----------

M1) Added an include to rpl_trx_boundary_parser.h

M2) Log_event class definition:

    M2a) Added a public boundary type named boundary_type to the
         Log_event class definition;

    M2b) Created a new protected method called
         update_boundary_type_from_flags() to update the boundary type
         of the event with the values of the Log_event flags.

M3) Query_log_event class definition:

    M3a) Created a new method is_begin_stmt();

    M3b) Created a new method is_commit_stmt();

    M3c) Created a new method is_rollback_stmt();

    M3d) Changed starts_group() method to use is_begin_stmt();

    M3e) Changed ends_group() method to use is_commit_stmt() and
         is_rollback_stmt().

M4) Intvar_log_event constructor:

    Added code to mark the event as BOUNDARY_TYPE_INSIDE_TRANSACTION.

M5) Rand_log_event constructor:

    Added code to mark the event as BOUNDARY_TYPE_INSIDE_TRANSACTION.

M6) Xid_log_event constructor:

    Added code to mark the event as BOUNDARY_TYPE_END_TRANSACTION.

M7) User_var_log_event constructor:

    Added code to mark the event as BOUNDARY_TYPE_INSIDE_TRANSACTION.

M8) Rows_query_log_event constructor:

    Added code to mark the event as BOUNDARY_TYPE_INSIDE_TRANSACTION.


log_event.cc
------------

M1) Log_event constructors:

    Added code to mark the event as BOUNDARY_TYPE_SELF_CONTAINED.

M2) Log_event::write_header:

    Added code to apply the boundary type into the flags of the event
    before writing it.

M3) Log_event::read_log_event:

    Added code to set event's boundary type according to event's flags.

    Obs.: The *_event constructors that have code to set their boundary
          types are those not used to reconstruct a previously logged
          event. When the event is being reconstructed (like using
          read_log_event), the boundary type is set by this
          modification.

    Changed a debug message to show also event's boundary type.

M4) Log_event::print_header:

    Prints the boundary type of the event.

M5) Query_log_event constructor:

    Before the verification of 'ignore_cmd_internals', we adjust the
    event's boundary type to BOUNDARY_TYPE_INSIDE_TRANSACTION if it is
    a BEGIN statement or to BOUNDARY_TYPE_END_TRANSACTION if it is a
    COMMIT or ROLLBACK statement.

    Later, if the Query event is not using immediate logging, we set
    the event boundary type to BOUNDARY_TYPE_INSIDE_TRANSACTION.

M6) Append_block_log_event constructor:

    Added code to mark the event as BOUNDARY_TYPE_INSIDE_TRANSACTION.

M7) Rows_log_event constructor:

    Added code to mark the event as BOUNDARY_TYPE_INSIDE_TRANSACTION.

M8) Table_map_log_event constructor:

    Added code to mark the event as BOUNDARY_TYPE_INSIDE_TRANSACTION.

M9) Gtid_log_event constructor:

    Added code to mark the event as BOUNDARY_TYPE_START_TRANSACTION.


binlog.cc
---------

M1) binlog_cache_data::write_event:

    Every Gtid_log_event is marked as BOUNDARY_TYPE_START_TRANSACTION
    by Gtid_log_event's constructor.

    If a Gtid_log_event have to be logged before the event to be
    written, it is needed to adjust the boundary type of the event to
    be written as follows:

    M1a) If ev->boundary_type is BOUNDARY_TYPE_START_TRANSACTION, we
         have to mark it as BOUNDARY_TYPE_INSIDE_TRANSACTION;

    M1b) If ev->boundary_type is BOUNDARY_TYPE_SELF_CONTAINED, we have
         to mark it as BOUNDARY_TYPE_END_TRANSACTION.

M2) MYSQL_BIN_LOG::write_event:

    Here it is possible to insert Intvar_log_event, Rand_log_event or
    User_var_log_event if needed before writing the event asked to be
    written.

    As all Intvar_log_event, Rand_log_event and User_var_log_event are
    marked as BOUNDARY_TYPE_INSIDE_TRANSACTION by their constructors,
    if the event to be written is marked as
    BOUNDARY_TYPE_SELF_CONTAINED (it is a DDL, for example), we change
    the first inserted event markers to BOUNDARY_TYPE_START_TRANSACTION
    and the event to be written is marked as
    BOUNDARY_TYPE_END_TRANSACTION.


rpl_binlog_sender.cc
--------------------

M1) Binlog_sender::fake_rotate_event:

    This method will send a rotate log_event without instantiating an
    Rotate_log_event object.

    The flags of the fake rotate event to be sent is marked as
    BOUNDARY_TYPE_IGNORE_BOUNDARY.

M2) Binlog_sender::send_heartbeat_event:

    This method will send a heartbeat log_event without instantiating
    an Heartbeat_log_event object.

    The flags of the heartbeat event to be sent is marked as
    BOUNDARY_TYPE_SELF_CONTAINED.


rpl_mi.h
--------

M1) Added an include to rpl_trx_boundary_parser.h

M2) Master_info class definition:

    Added an Transaction_boundary_parser object named event_parser to
    the Master_info class definition.


rpl_slave.cc
------------

M1) Added an include to rpl_trx_boundary_parser.h

M2) handle_slave_io:

    Added event boundary type to a debug "info" message stating:
    "IO thread received event..."

    Added a call to mi->tbm_state_machine.reset() right before
    entering the loop reading the events.

M3) queue_event:

    Added code to feed the Master_info event_parser with the
    the buffer of the event to being queued.

    If the mi->event_parser returns that it was an invalid
    transition, throw an warning message stating: "Unable to change
    boundary parser from %s to %s." and the boundary type names of the
    previous boundary type in the state machine and the last boundary
    type which the state machine was fed.

    After throwing the warning message, as the event_parser should
    now be in the error state, we will reset() the event parser, moving
    it to the "Not Defined" type.

    For the debug builds, we will add an assert to verify if the
    even_event parser has changed to a type other than error.

    For non debug builds, the I/O thread will throw a warning message
    on each event queued until it reaches a new transaction boundary
    parser consistent state (a self contained event or a start
    transaction).

mysqlbinlog.cc
--------------

M1) Added an include to rpl_trx_boundary_parser.h

M2) Added an Transaction_boundary_parser object named event_parser.

M3) dump_local_log_entries:

    M3a) Added a call to event_parser.reset() to initialize the
         verification of the event stream to be read.

    M3b) Added code to feed the event_parser with the buffer of the
         event read.

         If the event_parser returns that it was an invalid
         transition, do two things:

         M3b1) show a warning message stating: "Unable to change
               boundary parser from %s to %s." and the boundary type
               names of the previous boundary type and the boundary
               type of the event read;

         M3b2) Issue an event_parser.reset() to remove the
               transaction boundary parser from the error state.

M4) Added an include to rpl_trx_boundary_parser.cc at the end of the
    file.