WL#3653: Loadable Engine on Windows

RATIONALE
To make it possible to dynamically load engines (and other plugins)
on Windows.  (Currently the code is built in the wrong way.)

SUMMARY
Currently it is not possible to build a dynamically loadable engine as a 
separate component (.DLL) on Windows. This task is to track the effort of making
it work on Windows.

Note added by Trudy Pelzer, 2006-12-22:
Remarks from WL#3364, cancelled as a duplicate task:
"As part of WL#3201, storage engines have been made "plugable", except on 
Windows. This Worklog is to make plugable storage engine work on Windows."

REFERENCES
WL#3859 Plug-in Service API

Storage engines have code dependencies within the MySQL server that need to be 
resolved at load time.  Though it is not normally or commonly used, Windows 
executables can export symbols that are used in loadable modules such as dlls.

The method is surprisingly similar to what is done on Linux.  Functions and 
data are marked as exported out of the server and then imported using a import 
library into the storage engine at compile time.  Then, when the engine is 
loaded into the server at runtime, the entry points are mapped up using the 
normal loading process.

===The main problem===
The difference between Windows and Unix  is that Microsoft linker needs 
the list of exports, while Unix linkers typically export every global symbol 
from the binary.  
Storage engines traditionally use mysql functions and data in somewhat random 
manner, and thus we'll take  a practical approach and export all global symbols 
from server objects and selected libraries. We will implement a script to 
produce list of all global symbols from static libraries and objects.


===Importing data===
Microsoft linker uses special decorations when exporting data (i.e variables) 
from dynamic libraries. Users must use compler extension __declspec(dllimport) 
to access the data. For this purpose, a MYSQL_PLUGIN_IMPORT macro will be 
created and all data currently used by plugins will be marked as 
MYSQL_PLUGIN_IMPORTed.

===Future===
This is a stop gap solution until WL#3859 (Server API) is implemented.
Quirks( pre-link step, scripting and MYSQL_PLUGIN_IMPORT) can be removed after 
this WL is finished.

===Known limitations===
- Dynamic storage engine plugin support will not be available within embedded 
server library. DLLs we produce this way have dependency on mysqld.exe, and 
they will not load if current process is not mysqld.exe.
- /GL compiler option (that gives best possible optimization) will not work.
Objects produced with /GL are not in the COFF format and cannot be parsed by 
dumpbin.


==Support for plugins in build tools : win\configure.js and CMake==
Command line switches for win\configure.js will be made compatible to Unix with 
respect to plugins (refer to 
http://forge.mysql.com/wiki/MySQL_Internals_Support_for_Plug-Ins for details). 

Specifically --with-plugin-PLUGIN, --without-plugin-PLUGIN and --with-plugins=
[plugin-list or group] will be supported.

Also, win\configure.js will continue to support existing 
WITH__STORAGE_ENGINE syntax. This option has the effect as --with-
plugin-PLUGIN and means that PLUGIN should be statically compiled into the 
server, if it supports static build. If plugin does not support static build 
(currently example storage engine only), WITH__STORAGE_ENGINE and --
with-plugin-PLUGIN will have no effect.

Inclusion/exclusion logic is the same as on Unix, that is :
For any plugin that is not explicitly or implicitly (as a member of a selected 
group) selected or disabled, it is selected to be built dynamically if it 
supports dynamic build, and is disabled if it does not support dynamic build. 
If no plugin options are given, default group is selected. 

CMake will continue to support WITH__STORAGE_ENGINE and now will also 
understand WITHOUT__STORAGE_ENGINE. CMake will not support grouping 
internally, groups must be resolved within configure.js.

==How plugin-specific configure parameters are passed to Cmake==

win\configure.js will include logic to translate --with-plugins/--without-
plugins to CMake's WITH__STORAGE_ENGINE/WITHOUT__STORAGE_ENGINE

This will also handle grouping, so if someone passes --with-plugins=max, it 
with be expanded to a big list of WITH__STORAGE_ENGINE 

==Parsing plugin definition files \storage\*\plug.in==

Cmake will parse plugin definition file in storage engine directory plug.in to 
find out whether plugin is mandatory, can be built statically or dynamically.

If will decide, based n WITH__STORAGE_ENGINE parameter and plugin 
defintion whether to build and if yes, how to build.

== Cmake macro MYSQL_STORAGE_ENGINE==

CMakeLists.txt for every storage engine will use special macro 
MYSQL_STORAGE_ENGINE(), defined in 
\storage\mysql_storage_engine.cmake. 

This macro basically only needs _SOURCES to be defined,and does the 
rest (making static library or DLL, setting include path) automagically.

=== Exporting functions and data from mysqld ===

1.  Create a pre-link step that will run a script file after compilation of the 
server files but before the link step.
2.  This script is a jscript file that runs the dumpbin utility on all server 
object files and some of the libs that get linked in statically. The script 
needs to tell data from function and export data as "symbol_name DATA"  
3.  Step 2 above produces a module definition file that is used as input to the 
link phase for mysqld.
4.  The linker will now produce an import library for the server.  This import 
library is then used as input for all pluggable storage engines.

=== How to determine which variables needs to be MYSQL_PLUGIN_EXPORTed ==

1. Configure server with all plugins as dynamic - win\configure.js without 
parameters will do that. 
2. Try to make. 
3. If plugin DLL complains about unresolved symbol, find variable declaration, 
add MYSQL_PLUGIN_IMPORT to it. Goto step 2.

===Quirks and workarounds==
Ideally, all plug.in. parsing would be inside CMakeLists.txt. We do not do it 
now, and parse groups from within configure.js. This is a workaround for CMake 
regex limitations (own syntax, absence of non-greedy matches), that makes 
parsing anything non-trivial quite a complicated task.