WL#3324: Online Backup: Test breakpoints

Implement facilities to control the flow of the backup and restore operations 
during testing. This should allow tests to synchronize events in the backup and 
restore operations with the execution of other events.

For example, it should be possible to have the backup wait after taking a lock 
during a consistent snapshot so that insert statements can be run at the same 
time. This will allow proper testing of the consistent read.

Backup_sleep variable
=====================

This variable can be used to slow down backup/restore process so that parallel
queries can be executed. It indicates the number of seconds the backup kernel
will wait between each interaction with backup/restore driver.

Online Backup Breakpoints
=========================
Macros for creating breakpoints during testing.

What are breakpoints?
---------------------
Breakpoints are devices used to pause the execution of the backup system at a 
certain point in the code. There is a timeout that you can specify when you set 
the breakpoint which will enable execution to continue after the period in 
seconds expires.

The best use of these breakpoints is for pausing execution at critical points 
in the backup code to allow proper testing of certain features. For example, 
suppose you wanted to ensure the Consistent Snapshot driver was working 
properly. To do so, you would need to ensure no new INSERT statements are 
executed while the data is being backed up. If you use a breakpoint, you can 
set the breakpoint to pause the backup kernel at the point where it has set the 
consistent read and is reading rows. You can then insert some rows and release 
the breakpoint. The result should contain all of the rows in the table except 
those that were inserted once the consistent read was set.

How to use breakpoints.
-----------------------
To make a breakpoint available, you must add a macro call to the code. Simply 
insert the macro call as follows. The breakpoint_name is a text string that 
must be unique among the breakpoints. It is used in the macro as a means of 
tagging the code for pausing and resuming execution. Once the code is compiled, 
you can use a client connectio to set and release the breakpoint.

  BACKUP_BREAKPOINT("");
  
Breakpoints use the user-defined locking functions get_lock() to set the 
breakpoint and release_lock() to release it.

Setting breakpoints.
--------------------
To set an existing breakpoint, issue the following command where timeout is the 
number of seconds execution will wait for the release before continuing 
execution.

  SELECT get_lock("",);

Releasing breakpoints.
----------------------
To release an existing breakpoint, issue the following command. This releases 
execution allow the system to continue.

  SELECT release_lock("");

Example - Testing the Consistent Snapshot Driver
------------------------------------------------
To test the consistent snapshot driver, we can make use of the backup_cs_locked 
breakpoint to pause execution after the consistent read is initiated and before 
all of the rows from the table have been read. Consider an InnoDB table with 
the following structure as our test table.

  CREATE TABLE t1 (a INT) ENGINE=INNODB;

To perform this test using breakpoints, we need two client connections. One 
will be used to execute the backup command and another to set and release the 
breakpoint. In the first client, we set the breakpoint with the SELECT get_lock
("backup_cs_locked", 100); command. In the second client, we start the 
execution of the backup. We can return to the first client and issue several 
@INSERT statements then issue the SELECT release_lock("backup_cs_locked"); 
command to release the breakpoint.

We can then return to the second client, select all of the rows from the table 
to verify the rows were inserted. We can verify that the consistent snapshot 
worked by restoring the database (which is a destructive restore) and then 
select all of the rows. This will show that the new rows inserted while the 
backup was running were not inserted into the table. The following shows the 
output of the commands as described.

  First Client
  ============
  mysql> SELECT * FROM t1;
  +---+
  | a |
  +---+
  | 1 |
  | 2 |
  | 3 |
  +---+
  3 rows in set (0.00 sec)

  mysql> SELECT get_lock("backup_cs_locked", 100);
  +-----------------------------------+
  | get_lock("backup_cs_locked", 100) |
  +-----------------------------------+
  |                                 1 |
  +-----------------------------------+
  1 row in set (0.00 sec) @endcode

  Second Client
  =============
  mysql> BACKUP DATABASE test TO 'test.bak'; @endcode

  Note: The backup will pause while the breakpoint is set (the lock is held).

  First Client
  ============
  mysql> INSERT INTO t1 VALUES (101), (102), (103);
  Query OK, 3 rows affected (0.02 sec)
  Records: 3  Duplicates: 0  Warnings: 0

  mysql> SELECT * FROM t1;
  +-----+
  | a   |
  +-----+
  |   1 |
  |   2 |
  |   3 |
  | 101 |
  | 102 |
  | 103 |
  +-----+
  6 rows in set (0.00 sec)
  
  mysql> SELECT release_lock("backup_cs_locked");
  +----------------------------------+
  | release_lock("backup_cs_locked") |
  +----------------------------------+
  |                                1 |
  +----------------------------------+
  1 row in set (0.01 sec) @endcode

  Second Client
  =============
  +------------------------------+
  | Backup Summary               |
  +------------------------------+
  |  header     =       14 bytes |
  |  meta-data  =      120 bytes |
  |  data       =       30 bytes |
  |               -------------- |
  |  total             164 bytes |
  +------------------------------+
  5 rows in set (33.45 sec)

  mysql> SELECT * FROM t1;
  +-----+
  | a   |
  +-----+
  |   1 |
  |   2 |
  |   3 |
  | 101 |
  | 102 |
  | 103 |
  +-----+
  6 rows in set (0.00 sec)
  
  mysql> RESTORE FROM 'test.bak';
  +------------------------------+
  | Restore Summary              |
  +------------------------------+
  |  header     =       14 bytes |
  |  meta-data  =      120 bytes |
  |  data       =       30 bytes |
  |               -------------- |
  |  total             164 bytes |
  +------------------------------+
  5 rows in set (0.08 sec)
  
  mysql> SELECT * FROM t1;
  +---+
  | a |
  +---+
  | 1 |
  | 2 |
  | 3 |
  +---+
  3 rows in set (0.00 sec)@endcode

Note: The backup will complete once breakpoint is released (the lock is 
released).

Breakpoints
-----------
The following are the available breakpoints included in the code.

* backup_command  Occurs at the start of the backup operation.
* data_init  Occurs at the start of the INITIALIZE PHASE.
* data_prepare  Occurs at the start of the PREPARE PHASE.
* data_lock  Occurs at the start of the SYNC PHASE.
* data_unlock  Occurs after the unlock calls.
* data_finish  Occurs at the start of the FINISH PHASE.
* backup_meta  Occurs before the call to write_meta_data().
* backup_data  Occurs before the call to write_table_data().
* backup_done  Occurs after the call to write_table_data() returns.
* backup_cs_locked  Consistent Snapshot - after the consistent read has been 
initiated but before rows are read.
* backup_cs_open_tables  Consistent Snapshot - before the call to open and lock 
tables.
* backup_cs_reading  Consistent Snapshot - occurs during read.

Developer Notes
---------------
*Breakpoints can be used in debug builds only. You must compile the code using 
the DEBUG_EXTRA preprocessor directive. 
* When adding breakpoints, you must add a list item for each breakpoint to the 
documentation for breakpoints. See the code for the macro definition in @ref 
debug.h for details.

Synchronization points are implemented using debug_sync_point() calls.