The Debug Sync Facility is available as of MySQL 5.1.41, 5.5.0, and 6.0.6. Please note that - in spite of the "debug" in its name - this facility is completely independent from the DBUG facility (except that it uses DBUG to trace its operation, if DBUG is also configured in the server). With a properly configured server (see Section 25.7.2, “Debug Sync Activation/Deactivation”), this facility allows placement of synchronization points in the server code by using the DEBUG_SYNC macro:
open_tables(...) DEBUG_SYNC(thd, "after_open_tables"); lock_tables(...)
When activated, a synchronization point can
Emit a signal and/or
Wait for a signal
Nomenclature:
- signal
A value of a global variable that persists until overwritten by a new signal. The global variable can also be seen as a "signal post" or "flag mast". Then the signal is what is attached to the "signal post" or "flag mast".
- emit a signal
Assign the value (the signal) to the global variable ("set a flag") and broadcast a global condition to wake those waiting for a signal.
- wait for a signal
Loop over waiting for the global condition until the global value matches the wait-for signal.
By default, all synchronization points are inactive. They do nothing (except burn a couple of CPU cycles for checking if they are active).
A synchronization point becomes active when an action is requested for it. To do so, assign a value to the DEBUG_SYNC system variable:
SET DEBUG_SYNC= 'after_open_tables SIGNAL opened WAIT_FOR flushed';
This activates the synchronization point named 'after_open_tables'. The activation requests the synchronization point to emit the signal 'opened' and wait for another thread to emit the signal 'flushed' when the thread's execution runs through the synchronization point.
For every synchronization point there can be one action per thread only. Every thread can request multiple actions, but only one per synchronization point. In other words, a thread can activate multiple synchronization points.
Here is an example how to activate and use the synchronization points:
--connection conn1 SET DEBUG_SYNC= 'after_open_tables SIGNAL opened WAIT_FOR flushed'; send INSERT INTO t1 VALUES(1); --connection conn2 SET DEBUG_SYNC= 'now WAIT_FOR opened'; SET DEBUG_SYNC= 'after_abort_locks SIGNAL flushed'; FLUSH TABLE t1;
When conn1 runs through the INSERT statement, it hits the synchronization point 'after_open_tables'. It notices that it is active and executes its action. It emits the signal 'opened' and waits for another thread to emit the signal 'flushed'.
conn2 waits immediately at the special synchronization point 'now' for another thread to emit the 'opened' signal.
A signal remains in effect until it is overwritten. If conn1 signals 'opened' before conn2 reaches 'now', conn2 will still find the 'opened' signal. It does not wait in this case.
When conn2 reaches 'after_abort_locks', it signals 'flushed', which lets conn1 awake.
Normally the activation of a synchronization point is cleared when it has been executed. Sometimes it is necessary to keep the synchronization point active for another execution. You can add an execute count to the action:
SET DEBUG_SYNC= 'name SIGNAL sig EXECUTE 3';
This sets the synchronization point's activation counter to 3. Each execution decrements the counter. After the third execution the synchronization point becomes inactive.
One of the primary goals of this facility is to eliminate sleeps from the test suite. In most cases it should be possible to rewrite test cases so that they do not need to sleep. (Note that Debug Sync can synchronize only multiple threads within a single process. It cannot synchronize multiple processes.) However, to support test development, and as a last resort, synchronization point waiting times out. There is a default timeout, but it can be overridden:
SET DEBUG_SYNC= 'name WAIT_FOR sig TIMEOUT 10 EXECUTE 2';
TIMEOUT 0 is special: If the signal is not present, the wait times out immediately.
If a wait timeout occurs (even on TIMEOUT 0), a warning is generated so that it shows up in the test result.
You can throw an error message and kill the query when a synchronization point is hit a certain number of times:
SET DEBUG_SYNC= 'name HIT_LIMIT 3';
Or combine it with signal and/or wait:
SET DEBUG_SYNC= 'name SIGNAL sig EXECUTE 2 HIT_LIMIT 3';
Here the first two hits emit the signal, the third hit returns the error message and kills the query.
For cases where you are not sure that an action is taken and thus cleared in any case, you can forcibly clear (deactivate) a synchronization point:
SET DEBUG_SYNC= 'name CLEAR';
If you want to clear all actions and clear the global signal, use:
SET DEBUG_SYNC= 'RESET';
This is the only way to reset the global signal to an empty string.
For testing of the facility itself you can execute a synchronization point just as if it had been hit:
SET DEBUG_SYNC= 'name TEST';
The string to "assign" to the DEBUG_SYNC variable can contain:
{RESET |
<sync point name> TEST |
<sync point name> CLEAR |
<sync point name> {{SIGNAL <signal name> |
WAIT_FOR <signal name> [TIMEOUT <seconds>]}
[EXECUTE <count>] &| HIT_LIMIT <count>}
Here '&|' means 'and/or'. This means that one of the sections separated by '&|' must be present or both of them.
The Debug Sync facility is an optional part of the MySQL server. To cause Debug Sync to be compiled into the server, use the --enable-debug-sync option:
./configure --enable-debug-sync
Debug Sync is also compiled in if you configure with the --with-debug option (which implies --enable-debug-sync), unless you also use the --disable-debug-sync option.
The Debug Sync Facility, when compiled in, is disabled by default. To enable it, start mysqld with the --debug-sync-timeout[=N] option, where N is a timeout value greater than 0. N becomes the default timeout for the WAIT_FOR action of individual synchronization points. If N is 0, Debug Sync stays disabled. If the option is given without a value, the timeout is set to 300 seconds.
The DEBUG_SYNC system variable is the user interface to the Debug Sync facility. If Debug Sync is not compiled in, this variable is not available. If compiled in, the global DEBUG_SYNC value is read only and indicates whether the facility is enabled. By default, Debug Sync is disabled and the value of DEBUG_SYNC is "OFF". If the server is started with --debug-sync-timeout=N, where N is a timeout value greater than 0, Debug Sync is enabled and the value of DEBUG_SYNC is "ON - current signal" followed by the signal name. Also, N becomes the default timeout for individual synchronization points.
The session value can be read by any user and will have the same value as the global variable. The session value can be set by users that have the SUPER privilege to control synchronization points.
Setting the DEBUG_SYNC system variable requires the 'SUPER' privilege. You cannot read back the string that you assigned to the variable, unless you assign the value that the variable does already have. But that would give a parse error. A syntactically correct string is parsed into a debug synchronization point action and stored apart from the variable value.
The Debug Sync facility is enabled by default in the test suite, but can be disabled with:
mysql-test-run.pl ... --debug-sync-timeout=0 ...
Likewise, the default wait timeout can be set:
mysql-test-run.pl ... --debug-sync-timeout=10 ...
For test cases that require the Debug Sync facility, include the following line in the test case file:
--source include/have_debug_sync.inc
Pseudo code for a synchronization point:
#define DEBUG_SYNC(thd, sync_point_name) if (unlikely(opt_debug_sync_timeout)) debug_sync(thd, STRING_WITH_LEN(sync_point_name))
The synchronization point performs a binary search in a sorted array of actions for this thread.
The SET DEBUG_SYNC statement adds a requested action to the array or overwrites an existing action for the same synchronization point. When it adds a new action, the array is sorted again.
There are quite a few places in MySQL, where we use a synchronization pattern like this:
pthread_mutex_lock(&mutex);
thd->enter_cond(&condition_variable, &mutex, new_message);
#if defined(ENABLE_DEBUG_SYNC)
if (!thd->killed && !end_of_wait_condition)
DEBUG_SYNC(thd, "sync_point_name");
#endif
while (!thd->killed && !end_of_wait_condition)
pthread_cond_wait(&condition_variable, &mutex);
thd->exit_cond(old_message);
Here some explanations:
thd->enter_cond() is used to register the condition variable and the mutex in thd->mysys_var. This is done to allow the thread to be interrupted (killed) from its sleep. Another thread can find the condition variable to signal and mutex to use for synchronization in this thread's THD::mysys_var.
thd->enter_cond() requires the mutex to be acquired in advance.
thd->exit_cond() unregisters the condition variable and mutex and releases the mutex.
If you want to have a Debug Sync point with the wait, please place it behind enter_cond(). Only then you can safely decide, if the wait will be taken. Also you will have THD::proc_info correct when the sync point emits a signal. DEBUG_SYNC sets its own proc_info, but restores the previous one before releasing its internal mutex. As soon as another thread sees the signal, it does also see the proc_info from before entering the sync point. In this case it will be "new_message", which is associated with the wait that is to be synchronized.
In the example above, the wait condition is repeated before the sync point. This is done to skip the sync point, if no wait takes place. The sync point is before the loop (not inside the loop) to have it hit once only. It is possible that the condition variable is signaled multiple times without the wait condition to be true.
A bit off-topic: At some places, the loop is taken around the whole synchronization pattern:
while (!thd->killed && !end_of_wait_condition)
{
pthread_mutex_lock(&mutex);
thd->enter_cond(&condition_variable, &mutex, new_message);
if (!thd->killed [&& !end_of_wait_condition])
{
[DEBUG_SYNC(thd, "sync_point_name");]
pthread_cond_wait(&condition_variable, &mutex);
}
thd->exit_cond(old_message);
}
Note that it is important to repeat the test for thd->killed after enter_cond(). Otherwise the killing thread may kill this thread after it tested thd->killed in the loop condition and before it registered the condition variable and mutex in enter_cond(). In this case, the killing thread does not know that this thread is going to wait on a condition variable. It would just set THD::killed. But if we would not test it again, we would go asleep though we are killed. If the killing thread would kill us when we are after the second test, but still before sleeping, we hold the mutex, which is registered in mysys_var. The killing thread would try to acquire the mutex before signaling the condition variable. Since the mutex is only released implicitly in pthread_cond_wait(), the signaling happens at the right place. We have a safe synchronization.
When running the MySQL test suite with a "debug" server (the DBUG facility is configured in) and the --debug command line option, the Debug Sync Facility writes trace messages to the DBUG trace. The following shell commands proved very useful in extracting relevant information:
egrep 'query:|debug_sync_exec:' mysql-test/var/log/mysqld.1.trace
It shows all executed SQL statements and all actions executed by synchronization points.
Sometimes it is also useful to see, which synchronization points have been run through (hit) with or without executing actions. Then add "|debug_sync_point:" to the egrep pattern.
For complete syntax tests, functional tests, and examples see the test case debug_sync.test.
See also worklog entry WL#4259 - Debug Sync Facility
Reference manual 5.1
2.3.2 Typical configure Options (--enable-debug-sync)
5.1.2 Command Options (--debug-sync-timeout)
5.1.4 System Variables (debug_sync)
Test framework manual
4.14 Thread Synchronization in Test Cases (have_debug_sync.inc)
5.3 mysql-test-run.pl (--debug-sync-timeout)
