WL#3653: Loadable Engine on Windows
Affects: Server-5.1 — Status: Complete — Priority: Low
RATIONALE To make it possible to dynamically load engines (and other plugins) on Windows. (Currently the code is built in the wrong way.) SUMMARY Currently it is not possible to build a dynamically loadable engine as a separate component (.DLL) on Windows. This task is to track the effort of making it work on Windows. Note added by Trudy Pelzer, 2006-12-22: Remarks from WL#3364, cancelled as a duplicate task: "As part of WL#3201, storage engines have been made "plugable", except on Windows. This Worklog is to make plugable storage engine work on Windows." REFERENCES WL#3859 Plug-in Service API
WL#2936: Falcon & MySQL plugin interface: server variables
WL#3201: Configure support for server plugins
WL#3201: Configure support for server plugins
Storage engines have code dependencies within the MySQL server that need to be resolved at load time. Though it is not normally or commonly used, Windows executables can export symbols that are used in loadable modules such as dlls. The method is surprisingly similar to what is done on Linux. Functions and data are marked as exported out of the server and then imported using a import library into the storage engine at compile time. Then, when the engine is loaded into the server at runtime, the entry points are mapped up using the normal loading process. ===The main problem=== The difference between Windows and Unix is that Microsoft linker needs the list of exports, while Unix linkers typically export every global symbol from the binary. Storage engines traditionally use mysql functions and data in somewhat random manner, and thus we'll take a practical approach and export all global symbols from server objects and selected libraries. We will implement a script to produce list of all global symbols from static libraries and objects. ===Importing data=== Microsoft linker uses special decorations when exporting data (i.e variables) from dynamic libraries. Users must use compler extension __declspec(dllimport) to access the data. For this purpose, a MYSQL_PLUGIN_IMPORT macro will be created and all data currently used by plugins will be marked as MYSQL_PLUGIN_IMPORTed. ===Future=== This is a stop gap solution until WL#3859 (Server API) is implemented. Quirks( pre-link step, scripting and MYSQL_PLUGIN_IMPORT) can be removed after this WL is finished. ===Known limitations=== - Dynamic storage engine plugin support will not be available within embedded server library. DLLs we produce this way have dependency on mysqld.exe, and they will not load if current process is not mysqld.exe. - /GL compiler option (that gives best possible optimization) will not work. Objects produced with /GL are not in the COFF format and cannot be parsed by dumpbin. ==Support for plugins in build tools : win\configure.js and CMake== Command line switches for win\configure.js will be made compatible to Unix with respect to plugins (refer to http://forge.mysql.com/wiki/MySQL_Internals_Support_for_Plug-Ins for details). Specifically --with-plugin-PLUGIN, --without-plugin-PLUGIN and --with-plugins= [plugin-list or group] will be supported. Also, win\configure.js will continue to support existing WITH_<PLUGIN>_STORAGE_ENGINE syntax. This option has the effect as --with- plugin-PLUGIN and means that PLUGIN should be statically compiled into the server, if it supports static build. If plugin does not support static build (currently example storage engine only), WITH_<PLUGIN>_STORAGE_ENGINE and -- with-plugin-PLUGIN will have no effect. Inclusion/exclusion logic is the same as on Unix, that is : For any plugin that is not explicitly or implicitly (as a member of a selected group) selected or disabled, it is selected to be built dynamically if it supports dynamic build, and is disabled if it does not support dynamic build. If no plugin options are given, default group is selected. CMake will continue to support WITH_<PLUGIN>_STORAGE_ENGINE and now will also understand WITHOUT_<PLUGIN>_STORAGE_ENGINE. CMake will not support grouping internally, groups must be resolved within configure.js.
==How plugin-specific configure parameters are passed to Cmake== win\configure.js will include logic to translate --with-plugins/--without- plugins to CMake's WITH_<PLUGIN>_STORAGE_ENGINE/WITHOUT_<PLUGIN>_STORAGE_ENGINE This will also handle grouping, so if someone passes --with-plugins=max, it with be expanded to a big list of WITH_<PLUGIN>_STORAGE_ENGINE ==Parsing plugin definition files <source-root>\storage\*\plug.in== Cmake will parse plugin definition file in storage engine directory plug.in to find out whether plugin is mandatory, can be built statically or dynamically. If will decide, based n WITH_<PLUGIN>_STORAGE_ENGINE parameter and plugin defintion whether to build and if yes, how to build. == Cmake macro MYSQL_STORAGE_ENGINE== CMakeLists.txt for every storage engine will use special macro MYSQL_STORAGE_ENGINE(<PLUGIN>), defined in <source-root>\storage\mysql_storage_engine.cmake. This macro basically only needs <PLUGIN>_SOURCES to be defined,and does the rest (making static library or DLL, setting include path) automagically. === Exporting functions and data from mysqld === 1. Create a pre-link step that will run a script file after compilation of the server files but before the link step. 2. This script is a jscript file that runs the dumpbin utility on all server object files and some of the libs that get linked in statically. The script needs to tell data from function and export data as "symbol_name DATA" 3. Step 2 above produces a module definition file that is used as input to the link phase for mysqld. 4. The linker will now produce an import library for the server. This import library is then used as input for all pluggable storage engines. === How to determine which variables needs to be MYSQL_PLUGIN_EXPORTed == 1. Configure server with all plugins as dynamic - win\configure.js without parameters will do that. 2. Try to make. 3. If plugin DLL complains about unresolved symbol, find variable declaration, add MYSQL_PLUGIN_IMPORT to it. Goto step 2. ===Quirks and workarounds== Ideally, all plug.in. parsing would be inside CMakeLists.txt. We do not do it now, and parse groups from within configure.js. This is a workaround for CMake regex limitations (own syntax, absence of non-greedy matches), that makes parsing anything non-trivial quite a complicated task.
Copyright (c) 2000, 2018, Oracle Corporation and/or its affiliates. All rights reserved.