WL#4675: Unique Index Operation TC cleanup
Status: Assigned — Priority: Medium
The current implementation of unique key operations in the Transaction Coordinator (TC) component of NDB Cluster can be improved. The current implementation implements unique key operations by semi-externally invoking two separate primary key operations on the index table and the base table. This should be modified so that unique key operations are handled alongside primary key operations in the TCKEYREQ handler. By aligning unique key and primary key operation processing, this work should alleviate problems with : - Transaction (APIConnectRecord) state handling - Savepoint alignment between linked table operations - Error handling scenarios Other aspects of unique key indices should not be affected : - NDBAPI interface to unique indexes - Unique index maintenance through triggers - Unique index (re)build.
General implementation goals ---------------------------- G1) Bring implementation of primary key access (TCKEYREQ) and unique index access (TCINDXREQ) closer together, ideally into the same handler function. a) Remove *need* for separate TCINDXREQ, INDXKEYINFO, INDXATTRINFO TCINDXCONF etc. signals (Support for them must be maintained for backwards compatibility) b) Avoid duplication of code for - transaction state handling (e.g. CS_* states) - packed send of TCINDXCONF signals - commit ack marker handling c) Try to avoid complexities around 2 PK requests using a common savepoint G2) Maintain current optimised primary key (TCKEYREQ) flow G3) Simplify TC code where possible Limitations and Restrictions ---------------------------- L1) Unique index maintenance should be mostly unaffected, including : - Unique index definition/alter - Unique index (re)build - Unique index maintenance (insert/delete from triggers etc.) L2) Current unique index implementation limitations (e.g. limitations on locking modes etc.) will be maintained. L3) TC block and maybe NDBAPI will be affected by this WL. No other areas should be affected.
Current implementation notes ---------------------------- NDBAPI NdbIndexOperation is very similar to NdbOperation, with just different signal numbers for the request and response signals. TC BLOCK TCINDXREQ handling : The TCINDXREQ handler stores the received TCINDXREQ in a special pool and then uses the received KeyInfo (From INDXKEYINFO signals) to build a TCKEYREQ on the Index table, which reads the special $PK column from the index table. This 'index table read' TCKEYREQ contains most of the request flags from the original TCINDXREQ, as well as some special indicators to inform TC that it's the first part of a unique index lookup. Additionally the TC block is set as the client, so that the resulting TRANSID_AI information will be sent back from TUP to TC. TCKEYREQ handling : This first TCKEYREQ is directly executed, and results in an LQHKEYREQ being sent to the involved LQH block, which may be on the same, or another node. << LQH, ACC, TUP >> for index table read When the LQHKEYCONF returns, a TCKEYCONF is 'sent' to the TC block(self). TCINDXREQ handling : When the TC block has received all of the TRANSID_AI from TUP for the index table read, as well as the TCKEYCONF, it builds a second TCKEYREQ, on the base table, with the received primary key and fragment identity as key, and the AttrInfo (from IndxAttrInfo signals) as AttrInfo. This TCKEYREQ contains a special indicator to indicate that it is the second part of a unique index lookup operation. The client reference is set to be the original NDBAPI id. The TCKEYREQ is executed directly. TCKEYREQ handling : This second TCKEYREQ results in an LQHKEYREQ being sent to the involved LQH block, which may be on the same, or another node. << LQH, ACC, TUP >> for base table operation When the LQHKEYCONF returns, a TCKEYCONF is 'sent' to the TC block(self). TCINDXREQ handling : When the TCKEYCONF is received, a TCINDXCONF is sent to the NDBAPI client and some operation-related resources can be freed. Observations and discussion --------------------------- Avoiding duplication The original design attempts to keep unique index and primary key access separated by having unique index access be a client of primary key access. Unfortunately, this abstraction 'leaks' as the primary key access handling has to be aware that an index operation is being processed in certain cases. Also, connection (transaction) state checking functionality is duplicated despite having the same requirements for both operation types. The implementation should be refactored to minimise the code where the operation types differ to avoid this duplication. The main technique for doing this is to have both PK and UK access handled via the current TCKEYREQ handler. Provide PK access to index table The Optimize table functionality added to mysql-5.1-telco-6.3 supports optimization of unique indexes. This functionality requires the ability to be able to perform a table scan on a unique index, and perform scan-takeover update operations on the tuples returned to allow a unique index table to be optimized (i.e. to send TCKEYREQ requests). This suggests that there is at least one requirement to provide access to the index table as-is. e.g. TCKEYREQ on an index table should operate on it as a standalone table, TCINDXREQ should use it as an access method to it's base table. Suggestion from Jonas O : Have a unified signal handler, but if the GSN is TCINDXREQ, perform indirect access, otherwise perform direct (PK) access. Interaction between operations in a batch The behaviour of the system when multiple operations in a batch access the same tuple(s) via the same or different access methods is not well defined. For example, a PK delete in the same batch as a unique index read may result in success for both operations, deadlock, or a failing read operation. Locking behaviour Note that unique indexes do not currently support unlocked access. Within NDBAPI (storage/ndb/src/ndbapi/NdbIndexOperation.cpp), simple, dirty/last-committed reads are mapped to shared reads. (Note, does NdbRecord also perform this limited mapping? No.) This restrictive mapping should be done in the kernel rather than relying on the API. Interactions with index maintenance Index maintenance is done via triggers. Immediate triggers are fired in TUP when changes are made by an operation, and the fact that triggers were fired is returned in the LQHKEYCONF. This tells TC that the operation must wait until the fired triggers have completed before continuing with processing the operation. To suspend an operation in this way, saveTriggeringOpState() is called from execLQHKEYCONF(). This method simply saves the received LQHKEYCONF in a buffer. It is assumed that all triggered operations will complete with an LQHKEYCONF or LQHKEYREF signal being processed. These handlers will check whether the triggering operation has completed all of its triggers, and if so, will restart it (which is done by (re)sending it's LQHKEYCONF with the numberOfFiredTriggers set to 0). Immediate triggers are currently used for unique index maintenance and table reorganisation. This behaviour should be unaffected by this WL. For testing, note that primary key operations on unique-indexed tables can result in triggers being fired, and unique key operations on unique-indexed tables can result in triggers being fired. It is not (currently) expected that the first operation on the index table (read only) can fire triggers. Long signals and backwards compatibility WL4258 added long TCINDXREQ signals to mysql-5.1-telco-6.4 while maintaining support for short TCINDXREQ signals from API nodes at previous releases. This WL should maintain support for short and long variants of TCINDXREQ. (Is TCINDXREQ sent internally at all?) Note that TUP does not currently send long TRANSID_AI to kernel blocks, and so does not send long TRANSID_AI to TC as a result of the index table read. There are no plans to change this as part of this WL. Simplifying interactions with Savepoints etc. Unique index operations require that the initial index table access and the second base table occur within the same savepoint. The current savepoint identifier is incremented in TC when a TCKEYREQ with the exec flag set is encountered. One way to simplify having the first and second operations have the same savepoint is to create them both at the start. The first operation takes the KeyInfo from the TCINDXREQ, and gets generated ATTRINFO. The second takes the ATTRINFO from the TCINDXREQ, and waits for TRANSID_AI to get it's KEYINFO. This means that by the time the savepoint is incremented, all operations in the previous savepoint have been defined, but not all will have received all of their KEYINFO/ATTRINFO yet. This aligns with 'normal' primary key access. Proposed Implementation Modifications ------------------------------------- 1) execTCKEYREQ() modified to check received GSN, and for GSN==TCINDXREQ: - Perform unique index access specific checks - Create index table and base table access operations from received request, with each marked as such. The index table access operation also contains a reference to the base table access operation. The index table access operation should : - Read with a read lock - Have the client 'return address' set to the TC block's id. - Place any received KeyInfo (either inline or attached in a long section) into the index table access operation(1). - Place any received AttrInfo (either inline or attached in a long section) into the base table access operation(2). - Generate the AttrInfo to read the index table's $PK column. - Continue processing the index table access operation as normal (if all KeyInfo received, send LQHKEYREQ etc..) 2) TCINDXREQ, INDXKEYINFO and INDXATTRINFO modified to call execTCKEYREQ(), execKEYINFO() and execATTRINFO() handlers. 3) execKEYINFO() handler modified to detect case where operation is (1) and add KeyInfo to it (no work here?). 4) execATTRINFO() handler modified to add received ATTRINFO for operation(1) to operation (2). 5) execTRANSID_AI() handler modified to add received TRANSID_AI as KeyInfo to operation(2) - Specifically, AttributeHeader(s) must be removed if present, and the fragmentid must be set. 6) execATTRINFO() execLQHKEYCONF() and execTRANSID_AI() handlers modified so that when all required ATTRINFO, TRANSID_AI and LQHKEYCONF signals have been received for operation (2), it is executed as normal with : - At least read lock for read operations - client set to original NDBAPI client reference 7) execLQHKEYREF() modified to perform cleanup of both operations if required, and return TCKEYREF() to client as expected. 8) Remove old definitions, code paths etc. New implementation notes ------------------------ Having both operations defined upfront helps with Savepoint issues and also provides buffering for state information (received AttrInfo, fragId of base table key etc.) that is currently placed in index operation specific buffers. The second operation can be considered as a slow-to-start PK op which is fed keyinfo from the first, but otherwise looks like a standard operation. The normal post-execution operation cleanup mechanisms which release the CacheRecord and buffered key and attrinfo can continue to work in the normal way. Implementation approach ----------------------- The intended approach has two steps : 1) modify the current TCKEYREQ handler to also handle TCINDXREQ requests in the 'new' way. 2) remove existing code, types, structures involved in handling TCINDXREQ in the 'old' way. It is hoped that this makes it easier to compare old and new implementations as development progresses, and avoids a big-bang approach which may result in extra debugging. Testing approach ----------------
Copyright (c) 2000, 2018, Oracle Corporation and/or its affiliates. All rights reserved.