WL#9263: InnoDB: Enable partial access of LOB using multiple zlib streams
Affects: Server-8.0
—
Status: Complete
Current Behaviour: ------------------ When a table is created with compressed row format by specifying ROW_FORMAT=COMPRESSED, all the index pages and BLOB pages are compressed by making use of the zlib library. Currently compressed BLOBs are stored as a single stream. So to decompress and access any portion of the BLOB, we have to decompress from the beginning and proceed till the end. This makes partial fetch impossible. For example, if we want to fetch the last 20 bytes of a LOB, then we need to decompress the LOB from the beginning till the end and then only we can access those 20 bytes. To solve this issue what we need is that the LOB must be compressed into smaller zlib streams. This way we can avoid decompressing the full LOB when only a small portion is needed to be read or modified. One single LOB is stored into many smaller zlib streams instead of one big zlib stream. The smaller zlib streams are placed one after the other. So the 2nd zlib stream starts where the 1st zlib stream ended, the 3rd zlib stream starts where the 2nd zlib stream ends and so on. So we can read the full LOB by sequentially processing the zlib streams. The Algorithm: -------------- The input is an uncompressed BLOB data. And the output is compressed BLOB data in a sequence of zlib streams. The input is split into 128KB chunks. And each of those 128KB of uncompressed data is compressed into a single zlib stream. Then each of these zlib streams are placed sequentially. Insert Operation: ----------------- All inserts will follow the new approach whereby each LOB data page will be stored as a sequence of smaller zlib streams. Fetch Operation: ---------------- The fetch operation will be capable of handling old and new formats. This is necessary for backwards compatibility. Destroy Operation: ------------------ This will be used by the purge. Purge will also handle both old and new formats. Scope of this worklog: ---------------------- The scope of this worklog is to change the way the LOB is stored within InnoDB. It deals with only the full LOB. It does not deal with partial fetch, or modify operations.
Functional Requirements: ------------------------ This worklog doesn't add new functionalities. There is no externally visible new functionality. Everything should work as before. Non-Functional Requirements: ---------------------------- * There should be no memory leak or other memory issues. * There should be no performance regression.
The following functions can be provided to implement the feature. Insert Operation: ================= Insertion of compressed LOB is performed by the zInserter class. This class provides the zInserter::write() member function to write an array of compressed LOB data. This worklog will introduce a new member function zInserter::insert() that will write an array of compressed LOB data using the new approach. Fetch Operation: ================ To fetch an LOB written in the old format we will use zReader::fetch(), the current member function to read LOB. This worklog will introduce a new member function to read LOB in new format - zReader::read(). Convert Operation: ================== This worklog will introduce zInserter::convert_old_to_new() member function that will be used to convert LOB written in old format to LOB written in new format. Destroy Operation: ================== This should be largely unaffected by this worklog because it doesn't read/interpret the zlib stream.
Copyright (c) 2000, 2024, Oracle Corporation and/or its affiliates. All rights reserved.