The purpose of this task is to support Unicode for command line clients, specifically on Windows, and to do it right, i.e using Unicode (UTF-16LE) APIs provided by Windows for reading and writing on the console. The idea of this worklog is that the only thing a user shall do to use Unicode on the command line is to change console settings to use a TrueType font (Lucida, Consolas) instead of using default raster fonts. The MSI installer can help here (changing a shortcut to mysql.exe to use a TrueType font), but installer changes are outside this WL's scope.
Affected MySQL tools ==================== This task is about command line client mysql.exe only. In the future we can think of extending the other tools as well: - mysqladmin - mysqlshow - mysqlcheck - mysqlimport - maybe others... Unicode API vs standard C API ============================= There was an idea to introduce a new command line parameter --unicode, which will switch between the standard C stdin/stdout communication routines (fprintf/fputc/fgets) to the Windows console API (ReadConsoleW/WriteConsoleW): mysql --unicode test mysql.exe will keep using fprintf/fputc/fgets-family functions by default. The --unicode parameter would only be valid if we determine that the input stream (stdin) is a console (i.e. not a file or a pipe). Otherwise, an error will be printed. The --unicode parameter would be available on Windows only. No Linux, no Mac, etc. The idea of the --unicode parameter was canceled. mysql.exe will always use Unicode API to read lines in interactive mode, and always use Unicode API when printing if stdout/stderr are Windows console. Connection character set ======================== As MySQL server does not support UTF16-LE as a connection character set, we'll convert back and forth between the connection character set and the Windows wide string character set (UTF16-LE). We could change the default connection character set for mysql.exe to be UTF8 (so all of Unicode or at least the BMP is covered "out of the box"), but: - We want the new code to get stabilized first - Change from OEM character set to UTF8 will potentially lead to "illegal mix of collations" errors in some cases. We want to avoid that. So, the connection character set will keep working according to the usual character set detection procedure (as made by WL#1349) in this order: - Command line parameter: mysql --default-character-set=xxx - my.ini: default-character-set=xxx - automatic detection according to machine localization, as reported by GetLocaleInfo(). To start mysql.exe in the full-featured Unicode mode, users will run: mysql.exe --default-character-set=utf8 test which will "use Unicode API to communicate with console" and "use UTF8 to communicate with MySQL server". Although, the command line may look too long and inconvenient, that should not be a problem - the plan is to create a shortcut later anyway (in a separate installer WL). Another approach to avoid long command line is to put "default-character-set=utf8" into my.ini. Convert command line parameters ================================= We'll also correctly translate mysql.exe command line parameters (as they can have Unicode characters) to the connection character set. This is required, for example, for -e parameter, for SQL queries that contain Unicode characters: mysql -uroot -e "SELECT <something with Unicode>" We cannot use argc or argv from main() routine (since C runtime already does some translation from Unicode to the current ANSI codepage). Instead we're going to use GetCommandLineW() and CommandLinetoArgvW() APIs and translate them appropriately to client encoding. Implementation ============== Vladislav Vaintroub previously committed a preliminary patch implementing using of Unicode console API in mysql.exe (See the References sections). The final patch is planned to be based mostly on the Vlad's original patch, with some details changed: - Connection character set will stay detected according to WL#1349, rather than utf8. - utf16le conversion functions will be removed, as our string library now supports utf16le References ========== - Vlad's patch: http://lists.mysql.com/commits/105379 - Michael Kaplan's BLOG: http://blogs.msdn.com/b/michkap/archive/2010/04/07/9989346.aspx - Creating shotcuts: http://msdn.microsoft.com/en-us/library/xsy6k3ys discuss thread "UTF-16LE in Windows" (Roel Van de Paar, Alexander Barkov) [ mysql intranet ]/secure/mailarchive/mail.php?folder=6&mail=14934 [ mysql intranet ]/secure/mailarchive/mail.php?folder=6&mail=14936