MySQL 8.0.30
Source Code Documentation
aligned_alloc.h
Go to the documentation of this file.
1/*****************************************************************************
2
3Copyright (c) 2021, 2022, Oracle and/or its affiliates.
4
5This program is free software; you can redistribute it and/or modify it under
6the terms of the GNU General Public License, version 2.0, as published by the
7Free Software Foundation.
8
9This program is also distributed with certain software (including but not
10limited to OpenSSL) that is licensed under separate terms, as designated in a
11particular file or component or in included license documentation. The authors
12of MySQL hereby grant you an additional permission to link the program and
13your derivative works with the separately licensed software that they have
14included with MySQL.
15
16This program is distributed in the hope that it will be useful, but WITHOUT
17ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
18FOR A PARTICULAR PURPOSE. See the GNU General Public License, version 2.0,
19for more details.
20
21You should have received a copy of the GNU General Public License along with
22this program; if not, write to the Free Software Foundation, Inc.,
2351 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
24
25*****************************************************************************/
26
27/** @file include/detail/ut/aligned_alloc.h
28 Implementation bits and pieces for aligned allocations. */
29
30#ifndef detail_ut_aligned_alloc_h
31#define detail_ut_aligned_alloc_h
32
33#include <cassert>
34#include <cstddef>
35#include <cstdint>
36#include <cstdlib>
37#include <cstring>
38#include <limits>
39#include <memory>
40#include <utility>
41
42#include "my_compiler.h"
46
47namespace ut {
48namespace detail {
49
51 /** Block of memory returned by this functor will have an additional
52 (metadata) segment (at no additional cost of higher memory consumption)
53 which is guaranteed to be this big, and which can be used to store
54 whatever arbitrary data. See Aligned_alloc and Aligned_alloc_arr for
55 exemplary usages of it.
56 */
57 static constexpr uint32_t metadata_size = alignof(max_align_t);
58
59 /** Alias that we will be using to denote ptr to DATA segment. */
60 using data_segment_ptr = void *;
61
62 /** Dynamically allocates storage of given size and at the address aligned to
63 the requested alignment.
64
65 It is guaranteed that storage allocated by this functor is always
66 (size + alignment) big _and_ that there is always
67 alignof(std::max_align_t) spare bytes within that segment which can be
68 freely used. This means that pointer which is returned by this function
69 can always be safely reversed by alignof(std::max_align_t) bytes and hence
70 make this sub-segment accessible to any subsequent implementation.
71
72 Reversing the pointer for a value which is bigger than
73 alignof(std::max_align_t) bytes is in certain cases possible and can be
74 checked by inspecting the returned offset value. Some more specialized
75 implementations can take advantage of that fact too.
76
77 This property is very important because it can be taken advantage of by
78 other implementations so they are able to store whatever metadata they
79 would like into this segment of alignof(std::max_align_t) bytes, or FWIW
80 (pointer - offset) bytes in more specialized cases.
81
82 For example, let's say that size=10, alignment=32, and underlying
83 allocation function used by this function implementation returns a pointer
84 at address 128. This address must be a multiple of
85 alignof(std::max_align_t) which in this example is 16.
86
87 ------------------------------
88 | VARLEN | META | DATA |
89 ------------------------------
90 128 144 160 170
91
92 DATA is an actual data which has been requested with given size (10) and
93 alignment (32).
94 META is the alignof(std::max_align_t) segment that can always be freely
95 used by other implementations.
96 VARLEN is the leftover variable-length segment that specialized
97 implementations can further make use of by deducing its size from returned
98 offset.
99
100 See Aligned_alloc and Aligned_alloc_arr implementations to see an example
101 of how META segment can be used.
102
103 @param[in] size Size of storage (in bytes) requested to be allocated.
104 @param[in] alignment Alignment requirement for storage to be allocated.
105 Must be power of two and larger than alignof(std::max_align_t).
106 @return {pointer, offset} where pointer is a pointer to dynamically
107 allocated storage aligned to requested alignment, and offset is distance
108 in bytes from pointer which has been originally returned by underlying
109 dynamic allocation function (e.g. std::malloc). Otherwise {nullptr, 0} if
110 dynamic storage allocation failed.
111 */
112 template <bool Zero_initialized>
113 static inline std::pair<data_segment_ptr, std::size_t> alloc(
114 std::size_t size, std::size_t alignment) noexcept {
115 // This API is only about the extended alignments. Non-extended are
116 // already handled with std::malloc.
117 assert(alignment > alignof(std::max_align_t));
118
119 // Here we can take advantage of the fact that std::malloc is always
120 // guaranteed to return suitably aligned pointer P. In other words, this
121 // means that pointer P will be aligned to the value of
122 // alignof(std::max_align_t). That value on Linux implementations is
123 // typically 8 bytes (x86) or 16 bytes (x86-64) but this is completely
124 // implementation-defined. Windows implementation, for example, defines this
125 // value to be 8 bytes for both x86 _and_ x86-64.
126 //
127 // Question which arises is how many bytes do we need to ask std::malloc in
128 // order to satisfy the requested alignment, A, and requested size, S.
129 //
130 // Short answer is (size + alignment) and you can skip the following section
131 // if you're not interested in how we got to that value.
132 //
133 // Long answer is that we can model the problem mathematically as follows:
134 // A = alignment
135 // S = size
136 // P = std::malloc(N);
137 // P % alignof(std::max_align_t) = 0;
138 // where
139 // N is our unknown and for which we can only say that:
140 // N = S + K
141 // is true where K is also some unknown value.
142 //
143 // (A) In cases when P % A = 0, we know that we are already at suitable
144 // address so we can deduce that:
145 // K = 0
146 // from which it follows that:
147 // N = S + K = S
148 //
149 // (B) In cases when P % A != 0, our K must be sufficiently large so that
150 // <P, P + K]
151 // range contains value, P', so that
152 // P' % A = 0
153 // If K = A, then
154 // <P, P + A]
155 // which can be expanded as
156 // <P, P + 1, P + 2, P + 3, ..., P + A]
157 //
158 // If we know that
159 // P % A != 0
160 // then we also know that
161 // (P + A) % A = P % A + A % A = P % A + 0 = P % A != 0
162 // which implies that our P' value is in
163 // <P, P + A>
164 // range.
165 //
166 // Given that
167 // P % alignof(std::max_align_t) = 0
168 // P' % A = 0
169 // A % alignof(std::max_align_t) = 0
170 // P' % alignof(std::max_align_t) = 0
171 // it follows that our P' value can only be found at
172 // P' = P + L*alignof(std::max_align_t)
173 // increments where L is some constant value in range of
174 // L = [1, x>
175 // Expanding <P, P + A> into what we know so far:
176 // <P, ..., P + L*alignof(std::max_align_t), ..., P + A>
177 // it follows that
178 // P + L*alignof(std::max_align_t) < P + A
179 // which equals to
180 // L*alignof(std::max_align_t) < A
181 //
182 // We now just need to prove that
183 // L*alignof(std::max_align_t) < A
184 // is always true for some L value in which case this would mean that our
185 // <P, P + A> range contains at least one value for which P' % = 0.
186 //
187 // Picking the first available L value from L = [1, x> range,
188 // we will get
189 // 1*alignof(std::max_align_t) < A
190 // Given that
191 // A > 2*alignof(std::max_align_t) - 1
192 // must be true, it follows that
193 // 1*alignof(std::max_align_t) < 2*alignof(std::max_align_t) - 1 < A
194 // and
195 // 1*alignof(std::max_align_t) < 2*alignof(std::max_align_t) - 1
196 // alignof(std::max_align_t) > 1
197 // is always true.
198 //
199 // To conclude:
200 // (1) N = S for P % A = 0
201 // (2) N = S + A for P % A != 0
202 //
203 // Given that P is a runtime value which we cannot know upfront we must opt
204 // for N = S + A.
205 const std::size_t data_len = size + alignment;
206 void *mem = Alloc_fn::alloc<Zero_initialized>(data_len);
207 if (unlikely(!mem)) return {nullptr, 0};
208
209 // To guarantee that storage allocated by this function is as advertised
210 // (exactly (size + alignment) big with at least alignof(std::max_align_t)
211 // spare bytes to be used from within that segment) we must handle the N = S
212 // case (see above) and offset the memory by 1 byte. Given the above proof
213 // we know that the next suitable address aligned to requested alignment
214 // must be at least alignof(std::max_align_t) bytes far away.
215 void *buf =
216 reinterpret_cast<void *>(reinterpret_cast<std::uintptr_t>(mem) + 1);
217 std::size_t buf_size = data_len - 1;
218
219 // TODO(jbakamovic): wrap following std::align expression with ut_a(...)
220 // after we clean up circular-dependencies from our headers. Until then, it
221 // is not possible as including ut0dbg.h header makes the build fail in
222 // mysterious ways.
223 [[maybe_unused]] auto ret = std::align(alignment, size, buf, buf_size);
224 assert(ret != nullptr);
225
226 return {buf, reinterpret_cast<std::uintptr_t>(buf) -
227 reinterpret_cast<std::uintptr_t>(mem)};
228 }
229
230 /** Releases storage allocated through alloc().
231
232 @param[in] ptr data_segment_pointer decreased by offset bytes. Both are
233 obtained through alloc().
234 */
235 static inline void free(void *ptr) noexcept { Alloc_fn::free(ptr); }
236};
237
238/** Memory layout representation of metadata segment guaranteed by
239 the inner workings of Aligned_alloc_impl.
240
241 ----------------------------------------------------
242 | VARLEN | ALIGNED-ALLOC-META | ... DATA ... |
243 ----------------------------------------------------
244 ^ ^
245 | |
246 | |
247 | ptr returned by Aligned_alloc_impl
248 |
249 --------------------------------
250 | META_2 | META_1 |
251 --------------------------------
252 \ \
253 0 \
254 \
255 alignof(max_align_t) - 1
256
257 VARLEN and ALIGNED-ALLOC-META are direct byproduct of Aligned_alloc_impl
258 layout and guarantees.
259
260 VARLEN is the leftover variable-length segment that specialized
261 implementations can further make use of by deducing its size from returned
262 offset. Not used by this implementation.
263
264 ALIGNED-ALLOC-META is the segment which this abstraction is about. It
265 can hold up to sizeof(META_1) + sizeof(META_2) bytes which is, due to
266 Aligned_alloc_impl guarantees, at most alignof(max_align_t) bytes large.
267 Providing larger data than supported is not possible and it is guarded
268 through the means of static_assert. META_1 and META_2 fields can be
269 arbitrarily sized meaning that they can even be of different sizes each.
270
271 DATA is an actual segment which will keep the user data.
272 */
273template <typename Meta_1_type, typename Meta_2_type>
275 /** Convenience types that we will be using to serialize necessary details
276 into the metadata segment.
277 */
278 using meta_1_t = Meta_1_type;
279 using meta_2_t = Meta_2_type;
281
282 /** Metadata size */
283 static constexpr auto allocator_metadata_size =
284 sizeof(meta_1_t) + sizeof(meta_2_t);
285 /** Max metadata size */
287 /** Bail out if we cannot fit the requested data size. */
289 "Aligned_alloc_impl provides a strong guarantee "
290 "of only up to Aligned_alloc_impl::metadata_size bytes.");
291
292 /** Helper function which stores user-provided detail into the META_1 field.
293 */
295 std::size_t meta_1_v) noexcept {
296 assert(meta_1_v <= std::numeric_limits<meta_1_t>::max());
297 *ptr_to_meta_1(data) = meta_1_v;
298 }
299 /** Helper function which stores user-provided detail into the META_2 field.
300 */
302 std::size_t meta_2_v) noexcept {
303 assert(meta_2_v <= std::numeric_limits<meta_2_t>::max());
304 memcpy(ptr_to_meta_2(data), &meta_2_v, sizeof(meta_2_t));
305 }
306 /** Helper function which recovers the information user previously stored in
307 META_1 field.
308 */
309 static inline meta_1_t meta_1(
311 return *ptr_to_meta_1(data);
312 }
313 /** Helper function which recovers the information user previously stored in
314 META_2 field.
315 */
316 static inline meta_2_t meta_2(
318 meta_2_t meta_2_v;
319 memcpy(&meta_2_v, ptr_to_meta_2(data), sizeof(meta_2_t));
320 return meta_2_v;
321 }
322
323 private:
324 /** Helper accessor function to metadata (offset). */
325 static inline meta_1_t *ptr_to_meta_1(
327 return reinterpret_cast<meta_1_t *>(data) - 1;
328 }
329 /** Helper accessor function to metadata (data length). */
332 return reinterpret_cast<meta_2_t *>(ptr_to_meta_1(data)) - 1;
333 }
334};
335
336/** Aligned allocation routines.
337
338 They're implemented in terms of Aligned_alloc_impl (and
339 Aligned_alloc_metadata), and given the guarantees it provides,
340 Aligned_alloc::alloc() is able to encode offset and requested
341 allocation datalen into the metadata section without sacrificing memory or
342 making the implementation or end usage more complex.
343
344 Serializing offset into the metadata is what will enable
345 Aligned_alloc::free() to later on recover original pointer returned by
346 the underlying Aligned_alloc_impl allocation mechanism (std::malloc,
347 std::calloc) and consequently be able to appropriately release it
348 (std::free).
349
350 Serializing requested allocation datalen into the metadata is what will
351 enable higher-kinded functions, implemented on top of Aligned_alloc, to
352 take necessary actions such as cleaning up the resources by invoking
353 appropriate number of destructors of non-trivially-destructible types.
354 Otherwise, this would create a burden on end users by having to remember and
355 carry the array size all around the code. This is equivalent to what we find
356 in other standard implementations. For example, new int x[10] is always
357 released without passing the array size: delete[] x; The same holds with
358 this design.
359
360 Memory layout representation looks like the following:
361
362 ----------------------------------------------------
363 | VARLEN | ALIGNED-ALLOC-META | ... DATA ... |
364 ----------------------------------------------------
365 ^ ^
366 | |
367 | |
368 | ptr returned by Aligned_alloc_impl
369 |
370 -----------------------------------
371 | DATALEN | VARLEN-OFFSET |
372 -----------------------------------
373 \ \
374 0 \
375 \
376 alignof(max_align_t) - 1
377
378 VARLEN and ALIGNED-ALLOC-META are direct byproduct of Aligned_alloc_impl
379 layout and guarantees.
380
381 VARLEN is the leftover variable segment of bytes that specialized
382 implementations can further make use of by deducing its size from returned
383 offset. Not used by this implementation.
384
385 DATALEN field in ALIGNED-ALLOC-META segment encodes the total length of DATA
386 segment, which is the actual allocation size that client code has requested.
387
388 VARLEN-OFFSET in ALIGNED-ALLOC-META segment encodes the offset to VARLEN
389 segment which represents the original pointer obtained by underlying
390 allocation Aligned_alloc_impl mechanism.
391 */
392struct Aligned_alloc : public allocator_traits<false> {
394
395 /** Dynamically allocates storage of given size and at the address aligned
396 to the requested alignment.
397
398 @param[in] size Size of storage (in bytes) requested to be allocated.
399 @param[in] alignment Alignment requirement for storage to be allocated.
400 @return Pointer to the allocated storage. nullptr if dynamic storage
401 allocation failed.
402 */
403 template <bool Zero_initialized>
404 static inline void *alloc(std::size_t size, std::size_t alignment) {
405 auto ret = Aligned_alloc_impl::alloc<Zero_initialized>(size, alignment);
406 if (likely(ret.first)) {
407 // We are here taking advantage of Aligned_alloc_impl(S, A) for which
408 // we know that it will always return pointer P and offset O such that:
409 // 1. (P - O) expression is always well-defined.
410 // 2. And O is never less than alignof(std::max_align_t), that is
411 // Aligned_alloc_impl::metadata_size.
412 //
413 // Practically, this means that we can encode whatever metadata we want
414 // into [P - O, P> segment of memory whose length corresponds to
415 // the value of alignof(std::max_align_t). Commonly, this value is 8 bytes
416 // on 32-bit platforms and 16 bytes on 64-bit platforms but not always
417 // (e.g. Windows) and which is why we have to handle it in more generic
418 // way to be fully portable.
419 //
420 // Here we encode the offset so we can later on recover the
421 // original pointer, P' = (P - O), from within Aligned_alloc::free(P)
422 // context. Similarly, we encode the requested allocation datalen.
423 allocator_metadata::meta_1(ret.first, ret.second);
424 allocator_metadata::meta_2(ret.first, size);
425 }
426 return ret.first;
427 }
428
429 /** Releases storage dynamically allocated through
430 Aligned_alloc::alloc().
431
432 @param[in] data Pointer to storage allocated through
433 Aligned_alloc::alloc()
434 */
435 static inline void free(Aligned_alloc_impl::data_segment_ptr data) noexcept {
436 if (unlikely(!data)) return;
437 // Here we make use of the offset which has been encoded by
438 // Aligned_alloc::alloc() to be able to deduce the original pointer and
439 // simply forward it to std::free.
441 }
442
443 /** Returns the number of bytes requested to be allocated.
444
445 @param[in] data Pointer to storage allocated through
446 Aligned_alloc::alloc()
447 @return Number of bytes.
448 */
451 // Deducing the datalen field is straightforward, provided that
452 // data points to the DATA segment, which it is expected to do so.
453 return allocator_metadata::meta_2(data);
454 }
455
456 private:
457 /** Helper function which deduces the original pointer returned by
458 Aligned_alloc_impl from a pointer which is passed to us by the call-site.
459 */
460 static inline void *deduce(
462 // To recover the original pointer we just need to read the offset
463 // we have serialized into the first (allocator) metadata field.
464 auto offset = allocator_metadata::meta_1(data);
465 return reinterpret_cast<void *>(reinterpret_cast<std::uintptr_t>(data) -
466 offset);
467 }
468};
469
470/** Aligned allocation routines which are instrumented through PFS
471 (performance-schema).
472
473 They're implemented in terms of Aligned_alloc_impl (and
474 Aligned_alloc_metadata), and given the guarantees it provides,
475 Aligned_alloc::alloc() is able to encode offset and requested
476 allocation datalen into the metadata section without sacrificing memory or
477 making the implementation or end usage more complex.
478
479 Serializing offset into the metadata is what will enable
480 Aligned_alloc_pfs::free() to later on recover original pointer returned
481 by the underlying Aligned_alloc_impl allocation mechanism (std::malloc,
482 std::calloc) and consequently be able to appropriately release it
483 (std::free).
484
485 Serializing requested allocation datalen into the metadata is what will
486 enable higher-kinded functions, implemented on top of
487 Aligned_alloc_pfs, to take necessary actions such as cleaning up the
488 resources by invoking appropriate number of destructors of
489 non-trivially-destructible types. Otherwise, this would create a burden on
490 end users by having to remember and carry the array size all around the
491 code. This is equivalent to what we find in other standard implementations.
492 For example, new int x[10] is always released without passing the array
493 size: delete[] x; The same holds with this design.
494
495 PFS-wise this allocation routine will be storing the information that PFS
496 needs to do its own work:
497 * Owning thread
498 * Total length of bytes allocated
499 * Key
500
501 Memory layout representation looks like the following:
502
503 ------------------------------------------------------------------------------
504 | VARLEN1 | ALIGNED-ALLOC-META | PFS-META | VARLEN2 | PFS-META-OFFSET | DATA |
505 ------------------------------------------------------------------------------
506 ^ ^ ^ ^
507 | | | |
508 | | --------------------------- |
509 | | | OWNER | DATALEN' | KEY | |
510 | | --------------------------- |
511 | | |
512 | ptr returned by |
513 | Aligned_alloc_impl |
514 | |
515 | ptr to be returned to call-site
516 | will be pointing here
517 |
518 ------------------------------
519 | DATALEN | VARLEN1-OFFSET |
520 ------------------------------
521 \ \
522 0 \
523 \
524 alignof(max_align_t) - 1
525
526 VARLEN1 and ALIGNED-ALLOC-META are direct byproduct of Aligned_alloc_impl
527 (and Aligned_alloc_metadata) layout and guarantees.
528
529 VARLEN1 is the leftover variable-length segment that specialized
530 implementations can further make use of by deducing its size from returned
531 offset. Not used by this implementation.
532
533 DATALEN field in ALIGNED-ALLOC-META segment encodes the total length of DATA
534 segment, which is the actual allocation size that client code has requested.
535
536 VARLEN1-OFFSET in ALIGNED-ALLOC-META segment encodes the offset to VARLEN1
537 segment which represents the original pointer obtained by underlying
538 allocation Aligned_alloc_impl mechanism.
539
540 PFS-META, VARLEN2 and PFS-META-OFFSET are memory layout representation of
541 PFS_metadata.
542
543 OWNER field encode the owning thread. DATALEN' field encodes total size
544 of memory consumed and not only the size of the DATA segment. KEY field
545 encodes the PFS/PSI key.
546
547 VARLEN2 is the leftover variable-length segment that specialized
548 implementations can further make use of by deducing its size from the
549 following formulae: requested_alignment - sizeof(PFS-META-OFFSET) -
550 sizeof(PFS-META). In code that would be alignment -
551 PFS_metadata::size. Not used by this implementation.
552
553 PFS-META-OFFSET is a field which allows us to recover the pointer to
554 PFS-META segment from a pointer to DATA segment. Having a pointer to
555 PFS-META segment allows us to deduce the VARLEN1-OFFSET field from
556 ALIGNED-ALLOC-META segment which finally gives us a pointer obtained by the
557 underlying allocation Aligned_alloc_impl mechanism.
558 */
559struct Aligned_alloc_pfs : public allocator_traits<true> {
562
563 /** Dynamically allocates storage of given size at the address aligned to the
564 requested alignment.
565
566 @param[in] size Size of storage (in bytes) requested to be allocated.
567 @param[in] alignment Alignment requirement for storage to be allocated.
568 @param[in] key PSI memory key to be used for PFS memory instrumentation.
569 @return Pointer to the allocated storage. nullptr if dynamic storage
570 allocation failed.
571 */
572 template <bool Zero_initialized>
573 static inline void *alloc(std::size_t size, std::size_t alignment,
575 // We must take special care to allocate enough extra space to hold the
576 // PFS metadata (PFS-META + PFS-META-OFFSET) but we also need to take
577 // special care that the pointer which will be returned to the callee by
578 // this function will still be suitably over-aligned as requested. Both of
579 // these requirements can be fulfilled by finding the smallest multiple of
580 // requested alignment that is not smaller than actual PFS metadata size.
581 const auto metadata_len = calc_align(pfs_metadata::size, alignment);
582 const auto total_len = size + metadata_len;
583 auto ret =
584 Aligned_alloc_impl::alloc<Zero_initialized>(total_len, alignment);
585 if (unlikely(!ret.first)) return nullptr;
586
587 // Same as we do with non-PFS variant of Aligned_alloc::alloc(), here we
588 // encode the offset so we can later on recover the original pointer, P' =
589 // (P - O), from within Aligned_alloc_pfs::free(P) context. Similarly, we
590 // encode the requested allocation datalen.
591 allocator_metadata::meta_1(ret.first, ret.second);
592 allocator_metadata::meta_2(ret.first, size);
593
594#ifdef HAVE_PSI_MEMORY_INTERFACE
595 // When computing total number of bytes allocated. we must not only account
596 // for the size that we have requested (total_len) but we also need
597 // to account for extra memory Aligned_alloc_impl may have allocated in
598 // order to be able to accomodate the request. Amount of extra memory
599 // allocated corresponds to the offset value returned by Aligned_alloc_impl.
600 const auto datalen = total_len + ret.second;
601 // The point of this allocator variant is to trace the memory allocations
602 // through PFS (PSI) so do it.
604 key = PSI_MEMORY_CALL(memory_alloc)(key, datalen, &owner);
605 // To be able to do the opposite action of tracing when we are releasing the
606 // memory, we need right about the same data we passed to the tracing
607 // memory_alloc function. Let's encode this it into our allocator so we
608 // don't have to carry and keep this data around.
609 pfs_metadata::pfs_owning_thread(ret.first, owner);
611 pfs_metadata::pfs_key(ret.first, key);
612 pfs_metadata::pfs_metaoffset(ret.first, metadata_len);
613#endif
614
615 return static_cast<uint8_t *>(ret.first) + metadata_len;
616 }
617
618 /** Releases storage dynamically allocated through
619 Aligned_alloc_pfs::alloc().
620
621 @param[in] data Pointer to storage allocated through
622 Aligned_alloc_pfs::alloc()
623 */
624 static inline void free(PFS_metadata::data_segment_ptr data) noexcept {
625 if (unlikely(!data)) return;
626
627#ifdef HAVE_PSI_MEMORY_INTERFACE
628 // Deduce the PFS data we encoded in Aligned_alloc_pfs::alloc()
629 auto key = pfs_metadata::pfs_key(data);
630 auto owner = pfs_metadata::pfs_owning_thread(data);
632 // With the deduced PFS data, now trace the memory release action.
633 PSI_MEMORY_CALL(memory_free)
634 (key, datalen, owner);
635#endif
636
637 // Here we make use of the offset which has been encoded by
638 // Aligned_alloc_pfs::alloc() to be able to deduce the original pointer and
639 // simply forward it to std::free.
641 }
642
643 /** Returns the number of bytes requested to be allocated.
644
645 @param[in] data Pointer to storage allocated through
646 Aligned_alloc_pfs::alloc()
647 @return Number of bytes.
648 */
651 // In order to be able to deduce the datalen field, we have to deduce the
652 // beginning of PFS metadata segment first.
654 }
655
656 private:
657 /** Helper function which deduces the original pointer returned by
658 Aligned_alloc_impl from a pointer which is passed to us by the call-site.
659 */
660 static inline void *deduce(PFS_metadata::data_segment_ptr data) noexcept {
661 // To recover the original pointer we need to read the offset
662 // we have serialized into the first (allocator) metadata field. But to read
663 // that offset we have to jump over the PFS metadata first. We use PFS meta
664 // offset for that.
665 auto pfs_meta = pfs_metadata::deduce_pfs_meta(data);
666 auto offset = allocator_metadata::meta_1(pfs_meta);
667 return reinterpret_cast<void *>(reinterpret_cast<std::uintptr_t>(pfs_meta) -
668 offset);
669 }
670};
671
672/** Simple utility metafunction which selects appropriate allocator variant
673 (implementation) depending on the input parameter(s).
674 */
675template <bool Pfs_memory_instrumentation_on>
677 using type =
678 Aligned_alloc; // When PFS is OFF, pick ordinary, non-PFS, variant
679};
680
681template <>
682struct select_alloc_impl<true> {
683 using type = Aligned_alloc_pfs; // Otherwise, pick PFS variant
684};
685
686/** Just a small helper type which saves us some keystrokes. */
687template <bool Pfs_memory_instrumentation_on>
690
691/** Small wrapper which utilizes SFINAE to dispatch the call to appropriate
692 aligned allocator implementation.
693 */
694template <typename Impl>
696 template <bool Zero_initialized, typename T = Impl>
698 alloc(size_t size, size_t alignment, PSI_memory_key key) {
699 return Impl::template alloc<Zero_initialized>(size, alignment, key);
700 }
701 template <bool Zero_initialized, typename T = Impl>
703 alloc(size_t size, size_t alignment, PSI_memory_key /*key*/) {
704 return Impl::template alloc<Zero_initialized>(size, alignment);
705 }
706 static inline void free(void *ptr) { Impl::free(ptr); }
707 static inline size_t datalen(void *ptr) { return Impl::datalen(ptr); }
708};
709
710} // namespace detail
711} // namespace ut
712
713#endif
Simple allocator traits.
#define PSI_MEMORY_CALL(M)
Definition: psi_memory.h:35
constexpr DWORD buf_size
Definition: create_def.cc:227
unsigned int PSI_memory_key
Instrumented memory key.
Definition: psi_memory_bits.h:48
Small helper functions.
#define free(A)
Definition: lexyy.cc:915
Header for compiler-dependent features.
constexpr bool likely(bool expr)
Definition: my_compiler.h:54
constexpr bool unlikely(bool expr)
Definition: my_compiler.h:55
Definition: buf0block_hint.cc:29
Definition: ut0tuple.h:56
constexpr size_t calc_align(size_t n, size_t m)
Calculates the smallest multiple of m that is not smaller than n when m is a power of two.
Definition: helper.h:44
typename select_alloc_impl< Pfs_memory_instrumentation_on >::type select_alloc_impl_t
Just a small helper type which saves us some keystrokes.
Definition: aligned_alloc.h:689
This file contains a set of libraries providing overloads for regular dynamic allocation routines whi...
Definition: aligned_alloc.h:47
required string key
Definition: replication_asynchronous_connection_failover.proto:59
required string type
Definition: replication_group_member_actions.proto:33
static MEM_ROOT mem
Definition: sql_servers.cc:98
Implementation bits and pieces for PFS metadata handling.
Small wrapper which utilizes SFINAE to dispatch the call to appropriate aligned allocator implementat...
Definition: aligned_alloc.h:695
static void free(void *ptr)
Definition: aligned_alloc.h:706
static size_t datalen(void *ptr)
Definition: aligned_alloc.h:707
static std::enable_if< T::is_pfs_instrumented_v, void * >::type alloc(size_t size, size_t alignment, PSI_memory_key key)
Definition: aligned_alloc.h:698
static std::enable_if<!T::is_pfs_instrumented_v, void * >::type alloc(size_t size, size_t alignment, PSI_memory_key)
Definition: aligned_alloc.h:703
Definition: aligned_alloc.h:50
static constexpr uint32_t metadata_size
Block of memory returned by this functor will have an additional (metadata) segment (at no additional...
Definition: aligned_alloc.h:57
static void free(void *ptr) noexcept
Releases storage allocated through alloc().
Definition: aligned_alloc.h:235
static std::pair< data_segment_ptr, std::size_t > alloc(std::size_t size, std::size_t alignment) noexcept
Dynamically allocates storage of given size and at the address aligned to the requested alignment.
Definition: aligned_alloc.h:113
void * data_segment_ptr
Alias that we will be using to denote ptr to DATA segment.
Definition: aligned_alloc.h:60
Memory layout representation of metadata segment guaranteed by the inner workings of Aligned_alloc_im...
Definition: aligned_alloc.h:274
static constexpr auto allocator_metadata_size
Metadata size.
Definition: aligned_alloc.h:283
static void meta_2(Aligned_alloc_impl::data_segment_ptr data, std::size_t meta_2_v) noexcept
Helper function which stores user-provided detail into the META_2 field.
Definition: aligned_alloc.h:301
static void meta_1(Aligned_alloc_impl::data_segment_ptr data, std::size_t meta_1_v) noexcept
Bail out if we cannot fit the requested data size.
Definition: aligned_alloc.h:294
static meta_1_t meta_1(Aligned_alloc_impl::data_segment_ptr data) noexcept
Helper function which recovers the information user previously stored in META_1 field.
Definition: aligned_alloc.h:309
static unaligned_meta_2_t * ptr_to_meta_2(Aligned_alloc_impl::data_segment_ptr data) noexcept
Helper accessor function to metadata (data length).
Definition: aligned_alloc.h:330
static constexpr auto max_metadata_size
Max metadata size.
Definition: aligned_alloc.h:286
Meta_1_type meta_1_t
Convenience types that we will be using to serialize necessary details into the metadata segment.
Definition: aligned_alloc.h:278
static meta_2_t meta_2(Aligned_alloc_impl::data_segment_ptr data) noexcept
Helper function which recovers the information user previously stored in META_2 field.
Definition: aligned_alloc.h:316
meta_2_t unaligned_meta_2_t
Definition: aligned_alloc.h:280
Meta_2_type meta_2_t
Definition: aligned_alloc.h:279
static meta_1_t * ptr_to_meta_1(Aligned_alloc_impl::data_segment_ptr data) noexcept
Helper accessor function to metadata (offset).
Definition: aligned_alloc.h:325
Aligned allocation routines which are instrumented through PFS (performance-schema).
Definition: aligned_alloc.h:559
static void * alloc(std::size_t size, std::size_t alignment, pfs_metadata::pfs_memory_key_t key)
Dynamically allocates storage of given size at the address aligned to the requested alignment.
Definition: aligned_alloc.h:573
static void free(PFS_metadata::data_segment_ptr data) noexcept
Releases storage dynamically allocated through Aligned_alloc_pfs::alloc().
Definition: aligned_alloc.h:624
static void * deduce(PFS_metadata::data_segment_ptr data) noexcept
Helper function which deduces the original pointer returned by Aligned_alloc_impl from a pointer whic...
Definition: aligned_alloc.h:660
static allocator_metadata::meta_2_t datalen(PFS_metadata::data_segment_ptr data)
Returns the number of bytes requested to be allocated.
Definition: aligned_alloc.h:649
Aligned allocation routines.
Definition: aligned_alloc.h:392
static void * alloc(std::size_t size, std::size_t alignment)
Dynamically allocates storage of given size and at the address aligned to the requested alignment.
Definition: aligned_alloc.h:404
static void * deduce(Aligned_alloc_impl::data_segment_ptr data) noexcept
Helper function which deduces the original pointer returned by Aligned_alloc_impl from a pointer whic...
Definition: aligned_alloc.h:460
static allocator_metadata::meta_2_t datalen(Aligned_alloc_impl::data_segment_ptr data)
Returns the number of bytes requested to be allocated.
Definition: aligned_alloc.h:449
static void free(Aligned_alloc_impl::data_segment_ptr data) noexcept
Releases storage dynamically allocated through Aligned_alloc::alloc().
Definition: aligned_alloc.h:435
static void free(void *ptr)
Definition: allocator_traits.h:104
Memory layout representation of PFS metadata segment that is used by the allocator variants which als...
Definition: pfs.h:85
void * data_segment_ptr
Definition: pfs.h:93
static void pfs_datalen(data_segment_ptr data, size_t datalen) noexcept
Helper function which stores the PFS datalen info into the DATALEN field.
Definition: pfs.h:108
static void pfs_key(data_segment_ptr data, pfs_memory_key_t key) noexcept
Helper function which stores the PFS key info into the KEY field.
Definition: pfs.h:114
PSI_thread * pfs_owning_thread_t
Convenience types that we will be using to serialize necessary details into the Aligned_alloc metadat...
Definition: pfs.h:89
static constexpr auto size
Definition: pfs.h:99
static void pfs_owning_thread(data_segment_ptr data, pfs_owning_thread_t thread) noexcept
Helper function which stores the PFS thread info into the OWNER field.
Definition: pfs.h:102
static void pfs_metaoffset(data_segment_ptr data, std::size_t alignment) noexcept
Helper function which stores the offset to PFS metadata segment into the PFS-META-OFFSET field.
Definition: pfs.h:121
PSI_memory_key pfs_memory_key_t
Definition: pfs.h:91
static void * deduce_pfs_meta(data_segment_ptr data) noexcept
Helper function which deduces the pointer to the beginning of PFS metadata segment given the pointer ...
Definition: pfs.h:155
Simple allocator traits.
Definition: allocator_traits.h:81
Simple utility metafunction which selects appropriate allocator variant (implementation) depending on...
Definition: aligned_alloc.h:676