[+/-]
To authenticate a user against the server the client-server protocol employs one of serveral authentication methods:
methods provided by auth plugins as defined in WL1054
Since 5.5.7 server, authentication method to be used to
authenticate connections to a particular MySQL account is
indicated in the mysql.users table. For
earlier servers it is always
Secure
Password Authentication or
Old
Password Authentication depending on the
CLIENT_SECURE_CONNECTION
flag.
Client and server negotiate what types of authentication they support as part of the Connection Phase:
Each auth method is made up of
a name
its input and output data
The exchanged in- and output data may either be sent as part of
the
Initial
Handshake Packet and
Handshake
Response Packet or as part of the
Authentication
Method Switch Request Packet and following
packets. The structure is usually the same.
While the overall exchange of data is free-form there are limitations in the initial handshake of the amount of data that can be exchanged:
the auth_plugin_data field in
Initial
Handshake Packet can only carry 255 bytes
max (see
CLIENT_SECURE_CONNECTION)
the auth_response_data field in
Handshake
Response Packet can only carry 255 bytes
max too if
CLIENT_PLUGIN_AUTH_LENENC_CLIENT_DATA
is not yet
the client-side plugin may not receive its initial data in the initial handshake
auth_method_name is
mysql_old_password
client-side requires "8-byte random challenge" from server
client-side sends a 8-byte response based on the algorithm described below
If the server announces
Secure
Password Authentication in the
Initial
Handshake Packet the client may use the
first 8 byte of its 20-byte auth_plugin_data as input.

The hashing algorithm used for this auth method is broken as shown at http://sqlhack.com/ and CVE-2000-0981
client-side expects a 20-byte random challenge
client-side returns a 20-byte response based on the algorithm described below
mysql_native_password
New in 4.1.1

This method fixes a 2 short-comings of the
Old
Password Authentication:
using a tested, crypto-graphic hashing function which isn't broken
knowning the content of the hash in the
mysql.user table isn't
enough to authenticate against the MySQL Server.
The password is calculated by:
SHA1( password ) XOR SHA1( "20-bytes random data from server" <concat> SHA1( SHA1( password ) ) )
client-side requires no input from the server
client-side sends password as
string.NUL
mysql_clear_password
New in 5.6.x

The
Windows
Native Authentication method is more complex
than the other methods and extends the auth protocol as it has
to send more data forth and back than the old handshake allowed.
Basically it wraps the output of the Negotiate SSP in the Auth Phase protocol which either means NTLM or SPNEGO are used as underlying protocol.
Due to the implementation details the
Windows
Native Authentication method doesn't use the
fast path of the
Connection
Phase, but is only triggered on request as
part of the
Authentication
Method Switch Request Packet.

Due to implementation details (again) the first packet sent from the client to the server is expected to be either
254 bytes long max or
send the first 254 bytes first, appended by 1 byte with a magic value plus a 2nd packet with rest of the data
All following win-auth-packets don't get split.
The client will either send a SPNEGO or NTLM packet as next packet.
To implement the protocol several implementations exist:
Windows provides InitializeSecurityContextW and AcceptSecurityContext
a open source implemenation of NTML, SPNEGO and Kerberos5 are provided by Heimdal
Java6 added SPNEGO support to JGSS which also provides the NTLM and Kerberos5 support
removed in Windows Vista and 2008: http://msdn.microsoft.com/en-us/library/aa480152.aspx#appcomp_topic16
documented in http://msdn.microsoft.com/en-us/library/cc207842.aspx

The SHA256 auth method
XORs the clients password with the scramble buffers content
RSA encrypts that with the server's public key using RSA_PKCS1_OAEP_PADDING as padding
If the client doesn't have the public key it can ask for it by:
sending 0x01 as
auth response in the
Protocol::HandshakeResponse41
server will send a
Protocol::AuthMoreData
with the keyfile
client sends the encrypted password
