MySQL 8.1.0
Source Code Documentation
aligned_alloc.h
Go to the documentation of this file.
1/*****************************************************************************
2
3Copyright (c) 2021, 2023, Oracle and/or its affiliates.
4
5This program is free software; you can redistribute it and/or modify it under
6the terms of the GNU General Public License, version 2.0, as published by the
7Free Software Foundation.
8
9This program is also distributed with certain software (including but not
10limited to OpenSSL) that is licensed under separate terms, as designated in a
11particular file or component or in included license documentation. The authors
12of MySQL hereby grant you an additional permission to link the program and
13your derivative works with the separately licensed software that they have
14included with MySQL.
15
16This program is distributed in the hope that it will be useful, but WITHOUT
17ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
18FOR A PARTICULAR PURPOSE. See the GNU General Public License, version 2.0,
19for more details.
20
21You should have received a copy of the GNU General Public License along with
22this program; if not, write to the Free Software Foundation, Inc.,
2351 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
24
25*****************************************************************************/
26
27/** @file include/detail/ut/aligned_alloc.h
28 Implementation bits and pieces for aligned allocations. */
29
30#ifndef detail_ut_aligned_alloc_h
31#define detail_ut_aligned_alloc_h
32
33#include <cassert>
34#include <cstddef>
35#include <cstdint>
36#include <cstdlib>
37#include <cstring>
38#include <limits>
39#include <memory>
40#include <utility>
41
42#include "my_compiler.h"
46
47namespace ut {
48namespace detail {
49
51 /** Block of memory returned by this functor will have an additional
52 (metadata) segment (at no additional cost of higher memory consumption)
53 which is guaranteed to be this big, and which can be used to store
54 whatever arbitrary data. See Aligned_alloc and Aligned_alloc_arr for
55 exemplary usages of it.
56 */
57 static constexpr uint32_t metadata_size = alignof(max_align_t);
58
59 /** Alias that we will be using to denote ptr to DATA segment. */
60 using data_segment_ptr = void *;
61
62 /** Dynamically allocates storage of given size and at the address aligned to
63 the requested alignment.
64
65 It is guaranteed that storage allocated by this functor is always
66 (size + alignment) big _and_ that there is always
67 alignof(std::max_align_t) spare bytes within that segment which can be
68 freely used. This means that pointer which is returned by this function
69 can always be safely reversed by alignof(std::max_align_t) bytes and hence
70 make this sub-segment accessible to any subsequent implementation.
71
72 Reversing the pointer for a value which is bigger than
73 alignof(std::max_align_t) bytes is in certain cases possible and can be
74 checked by inspecting the returned offset value. Some more specialized
75 implementations can take advantage of that fact too.
76
77 This property is very important because it can be taken advantage of by
78 other implementations so they are able to store whatever metadata they
79 would like into this segment of alignof(std::max_align_t) bytes, or FWIW
80 (pointer - offset) bytes in more specialized cases.
81
82 For example, let's say that size=10, alignment=32, and underlying
83 allocation function used by this function implementation returns a pointer
84 at address 128. This address must be a multiple of
85 alignof(std::max_align_t) which in this example is 16.
86
87 ------------------------------
88 | VARLEN | META | DATA |
89 ------------------------------
90 128 144 160 170
91
92 DATA is an actual data which has been requested with given size (10) and
93 alignment (32).
94 META is the alignof(std::max_align_t) segment that can always be freely
95 used by other implementations.
96 VARLEN is the leftover variable-length segment that specialized
97 implementations can further make use of by deducing its size from returned
98 offset.
99
100 See Aligned_alloc and Aligned_alloc_arr implementations to see an example
101 of how META segment can be used.
102
103 @param[in] size Size of storage (in bytes) requested to be allocated.
104 @param[in] alignment Alignment requirement for storage to be allocated.
105 Must be power of two and larger than alignof(std::max_align_t).
106 @return {pointer, offset} where pointer is a pointer to dynamically
107 allocated storage aligned to requested alignment, and offset is distance
108 in bytes from pointer which has been originally returned by underlying
109 dynamic allocation function (e.g. std::malloc). Otherwise {nullptr, 0} if
110 dynamic storage allocation failed.
111 */
112 template <bool Zero_initialized>
113 static inline std::pair<data_segment_ptr, std::size_t> alloc(
114 std::size_t size, std::size_t alignment) noexcept {
115 // This API is only about the extended alignments. Non-extended are
116 // already handled with std::malloc.
117 assert(alignment > alignof(std::max_align_t));
118
119 // Here we can take advantage of the fact that std::malloc is always
120 // guaranteed to return suitably aligned pointer P. In other words, this
121 // means that pointer P will be aligned to the value of
122 // alignof(std::max_align_t). That value on Linux implementations is
123 // typically 8 bytes (x86) or 16 bytes (x86-64) but this is completely
124 // implementation-defined. Windows implementation, for example, defines this
125 // value to be 8 bytes for both x86 _and_ x86-64.
126 //
127 // Question which arises is how many bytes do we need to ask std::malloc in
128 // order to satisfy the requested alignment, A, and requested size, S.
129 //
130 // Short answer is (size + alignment) and you can skip the following section
131 // if you're not interested in how we got to that value.
132 //
133 // Long answer is that we can model the problem mathematically as follows:
134 // A = alignment
135 // S = size
136 // P = std::malloc(N);
137 // P % alignof(std::max_align_t) = 0;
138 // where
139 // N is our unknown and for which we can only say that:
140 // N = S + K
141 // is true where K is also some unknown value.
142 //
143 // (A) In cases when P % A = 0, we know that we are already at suitable
144 // address so we can deduce that:
145 // K = 0
146 // from which it follows that:
147 // N = S + K = S
148 //
149 // (B) In cases when P % A != 0, our K must be sufficiently large so that
150 // <P, P + K]
151 // range contains value, P', so that
152 // P' % A = 0
153 // If K = A, then
154 // <P, P + A]
155 // which can be expanded as
156 // <P, P + 1, P + 2, P + 3, ..., P + A]
157 //
158 // If we know that
159 // P % A != 0
160 // then we also know that
161 // (P + A) % A = P % A + A % A = P % A + 0 = P % A != 0
162 // which implies that our P' value is in
163 // <P, P + A>
164 // range.
165 //
166 // Given that
167 // P % alignof(std::max_align_t) = 0
168 // P' % A = 0
169 // A % alignof(std::max_align_t) = 0
170 // P' % alignof(std::max_align_t) = 0
171 // it follows that our P' value can only be found at
172 // P' = P + L*alignof(std::max_align_t)
173 // increments where L is some constant value in range of
174 // L = [1, x>
175 // Expanding <P, P + A> into what we know so far:
176 // <P, ..., P + L*alignof(std::max_align_t), ..., P + A>
177 // it follows that
178 // P + L*alignof(std::max_align_t) < P + A
179 // which equals to
180 // L*alignof(std::max_align_t) < A
181 //
182 // We now just need to prove that
183 // L*alignof(std::max_align_t) < A
184 // is always true for some L value in which case this would mean that our
185 // <P, P + A> range contains at least one value for which P' % = 0.
186 //
187 // Picking the first available L value from L = [1, x> range,
188 // we will get
189 // 1*alignof(std::max_align_t) < A
190 // Given that
191 // A > 2*alignof(std::max_align_t) - 1
192 // must be true, it follows that
193 // 1*alignof(std::max_align_t) < 2*alignof(std::max_align_t) - 1 < A
194 // and
195 // 1*alignof(std::max_align_t) < 2*alignof(std::max_align_t) - 1
196 // alignof(std::max_align_t) > 1
197 // is always true.
198 //
199 // To conclude:
200 // (1) N = S for P % A = 0
201 // (2) N = S + A for P % A != 0
202 //
203 // Given that P is a runtime value which we cannot know upfront we must opt
204 // for N = S + A.
205 const std::size_t data_len = size + alignment;
206 void *mem = Alloc_fn::alloc<Zero_initialized>(data_len);
207 if (unlikely(!mem)) return {nullptr, 0};
208
209 // To guarantee that storage allocated by this function is as advertised
210 // (exactly (size + alignment) big with at least alignof(std::max_align_t)
211 // spare bytes to be used from within that segment) we must handle the N = S
212 // case (see above) and offset the memory by 1 byte. Given the above proof
213 // we know that the next suitable address aligned to requested alignment
214 // must be at least alignof(std::max_align_t) bytes far away.
215 void *buf =
216 reinterpret_cast<void *>(reinterpret_cast<std::uintptr_t>(mem) + 1);
217 std::size_t buf_size = data_len - 1;
218
219 // TODO(jbakamovic): wrap following std::align expression with ut_a(...)
220 // after we clean up circular-dependencies from our headers. Until then, it
221 // is not possible as including ut0dbg.h header makes the build fail in
222 // mysterious ways.
223 [[maybe_unused]] auto ret = std::align(alignment, size, buf, buf_size);
224 assert(ret != nullptr);
225
226 return {buf, reinterpret_cast<std::uintptr_t>(buf) -
227 reinterpret_cast<std::uintptr_t>(mem)};
228 }
229
230 /** Releases storage allocated through alloc().
231
232 @param[in] ptr data_segment_pointer decreased by offset bytes. Both are
233 obtained through alloc().
234 */
235 static inline void free(void *ptr) noexcept { Alloc_fn::free(ptr); }
236};
237
238/** Memory layout representation of metadata segment guaranteed by
239 the inner workings of Aligned_alloc_impl.
240
241 ----------------------------------------------------
242 | VARLEN | ALIGNED-ALLOC-META | ... DATA ... |
243 ----------------------------------------------------
244 ^ ^
245 | |
246 | |
247 | ptr returned by Aligned_alloc_impl
248 |
249 --------------------------------
250 | META_2 | META_1 |
251 --------------------------------
252 \ \
253 0 \
254 \
255 alignof(max_align_t) - 1
256
257 VARLEN and ALIGNED-ALLOC-META are direct byproduct of Aligned_alloc_impl
258 layout and guarantees.
259
260 VARLEN is the leftover variable-length segment that specialized
261 implementations can further make use of by deducing its size from returned
262 offset. Not used by this implementation.
263
264 ALIGNED-ALLOC-META is the segment which this abstraction is about. It
265 can hold up to sizeof(META_1) + sizeof(META_2) bytes which is, due to
266 Aligned_alloc_impl guarantees, at most alignof(max_align_t) bytes large.
267 Providing larger data than supported is not possible and it is guarded
268 through the means of static_assert. META_1 and META_2 fields can be
269 arbitrarily sized meaning that they can even be of different sizes each.
270
271 DATA is an actual segment which will keep the user data.
272 */
273template <typename Meta_1_type, typename Meta_2_type>
275 /** Convenience types that we will be using to serialize necessary details
276 into the metadata segment.
277 */
278 using meta_1_t = Meta_1_type;
279 using meta_2_t = Meta_2_type;
281
282 /** Metadata size */
283 static constexpr auto allocator_metadata_size =
284 sizeof(meta_1_t) + sizeof(meta_2_t);
285 /** Max metadata size */
287 /** Bail out if we cannot fit the requested data size. */
289 "Aligned_alloc_impl provides a strong guarantee "
290 "of only up to Aligned_alloc_impl::metadata_size bytes.");
291
292 /** Helper function which stores user-provided detail into the META_1 field.
293 */
295 std::size_t meta_1_v) noexcept {
296 assert(meta_1_v <= std::numeric_limits<meta_1_t>::max());
297 *ptr_to_meta_1(data) = meta_1_v;
298 }
299 /** Helper function which stores user-provided detail into the META_2 field.
300 */
302 std::size_t meta_2_v) noexcept {
303 assert(meta_2_v <= std::numeric_limits<meta_2_t>::max());
304 meta_2_t meta_2_v_typed = meta_2_v;
305 memcpy(ptr_to_meta_2(data), &meta_2_v_typed, sizeof(meta_2_t));
306 }
307 /** Helper function which recovers the information user previously stored in
308 META_1 field.
309 */
310 static inline meta_1_t meta_1(
312 return *ptr_to_meta_1(data);
313 }
314 /** Helper function which recovers the information user previously stored in
315 META_2 field.
316 */
317 static inline meta_2_t meta_2(
319 meta_2_t meta_2_v;
320 memcpy(&meta_2_v, ptr_to_meta_2(data), sizeof(meta_2_t));
321 return meta_2_v;
322 }
323
324 private:
325 /** Helper accessor function to metadata (offset). */
326 static inline meta_1_t *ptr_to_meta_1(
328 return reinterpret_cast<meta_1_t *>(data) - 1;
329 }
330 /** Helper accessor function to metadata (data length). */
333 return reinterpret_cast<meta_2_t *>(ptr_to_meta_1(data)) - 1;
334 }
335};
336
337/** Aligned allocation routines.
338
339 They're implemented in terms of Aligned_alloc_impl (and
340 Aligned_alloc_metadata), and given the guarantees it provides,
341 Aligned_alloc::alloc() is able to encode offset and requested
342 allocation datalen into the metadata section without sacrificing memory or
343 making the implementation or end usage more complex.
344
345 Serializing offset into the metadata is what will enable
346 Aligned_alloc::free() to later on recover original pointer returned by
347 the underlying Aligned_alloc_impl allocation mechanism (std::malloc,
348 std::calloc) and consequently be able to appropriately release it
349 (std::free).
350
351 Serializing requested allocation datalen into the metadata is what will
352 enable higher-kinded functions, implemented on top of Aligned_alloc, to
353 take necessary actions such as cleaning up the resources by invoking
354 appropriate number of destructors of non-trivially-destructible types.
355 Otherwise, this would create a burden on end users by having to remember and
356 carry the array size all around the code. This is equivalent to what we find
357 in other standard implementations. For example, new int x[10] is always
358 released without passing the array size: delete[] x; The same holds with
359 this design.
360
361 Memory layout representation looks like the following:
362
363 ----------------------------------------------------
364 | VARLEN | ALIGNED-ALLOC-META | ... DATA ... |
365 ----------------------------------------------------
366 ^ ^
367 | |
368 | |
369 | ptr returned by Aligned_alloc_impl
370 |
371 -----------------------------------
372 | DATALEN | VARLEN-OFFSET |
373 -----------------------------------
374 \ \
375 0 \
376 \
377 alignof(max_align_t) - 1
378
379 VARLEN and ALIGNED-ALLOC-META are direct byproduct of Aligned_alloc_impl
380 layout and guarantees.
381
382 VARLEN is the leftover variable segment of bytes that specialized
383 implementations can further make use of by deducing its size from returned
384 offset. Not used by this implementation.
385
386 DATALEN field in ALIGNED-ALLOC-META segment encodes the total length of DATA
387 segment, which is the actual allocation size that client code has requested.
388
389 VARLEN-OFFSET in ALIGNED-ALLOC-META segment encodes the offset to VARLEN
390 segment which represents the original pointer obtained by underlying
391 allocation Aligned_alloc_impl mechanism.
392 */
393struct Aligned_alloc : public allocator_traits<false> {
395
396 /** Dynamically allocates storage of given size and at the address aligned
397 to the requested alignment.
398
399 @param[in] size Size of storage (in bytes) requested to be allocated.
400 @param[in] alignment Alignment requirement for storage to be allocated.
401 @return Pointer to the allocated storage. nullptr if dynamic storage
402 allocation failed.
403 */
404 template <bool Zero_initialized>
405 static inline void *alloc(std::size_t size, std::size_t alignment) {
406 auto ret = Aligned_alloc_impl::alloc<Zero_initialized>(size, alignment);
407 if (likely(ret.first)) {
408 // We are here taking advantage of Aligned_alloc_impl(S, A) for which
409 // we know that it will always return pointer P and offset O such that:
410 // 1. (P - O) expression is always well-defined.
411 // 2. And O is never less than alignof(std::max_align_t), that is
412 // Aligned_alloc_impl::metadata_size.
413 //
414 // Practically, this means that we can encode whatever metadata we want
415 // into [P - O, P> segment of memory whose length corresponds to
416 // the value of alignof(std::max_align_t). Commonly, this value is 8 bytes
417 // on 32-bit platforms and 16 bytes on 64-bit platforms but not always
418 // (e.g. Windows) and which is why we have to handle it in more generic
419 // way to be fully portable.
420 //
421 // Here we encode the offset so we can later on recover the
422 // original pointer, P' = (P - O), from within Aligned_alloc::free(P)
423 // context. Similarly, we encode the requested allocation datalen.
424 allocator_metadata::meta_1(ret.first, ret.second);
425 allocator_metadata::meta_2(ret.first, size);
426 }
427 return ret.first;
428 }
429
430 /** Releases storage dynamically allocated through
431 Aligned_alloc::alloc().
432
433 @param[in] data Pointer to storage allocated through
434 Aligned_alloc::alloc()
435 */
436 static inline void free(Aligned_alloc_impl::data_segment_ptr data) noexcept {
437 if (unlikely(!data)) return;
438 // Here we make use of the offset which has been encoded by
439 // Aligned_alloc::alloc() to be able to deduce the original pointer and
440 // simply forward it to std::free.
442 }
443
444 /** Returns the number of bytes requested to be allocated.
445
446 @param[in] data Pointer to storage allocated through
447 Aligned_alloc::alloc()
448 @return Number of bytes.
449 */
452 // Deducing the datalen field is straightforward, provided that
453 // data points to the DATA segment, which it is expected to do so.
454 return allocator_metadata::meta_2(data);
455 }
456
457 private:
458 /** Helper function which deduces the original pointer returned by
459 Aligned_alloc_impl from a pointer which is passed to us by the call-site.
460 */
461 static inline void *deduce(
463 // To recover the original pointer we just need to read the offset
464 // we have serialized into the first (allocator) metadata field.
465 auto offset = allocator_metadata::meta_1(data);
466 return reinterpret_cast<void *>(reinterpret_cast<std::uintptr_t>(data) -
467 offset);
468 }
469};
470
471/** Aligned allocation routines which are instrumented through PFS
472 (performance-schema).
473
474 They're implemented in terms of Aligned_alloc_impl (and
475 Aligned_alloc_metadata), and given the guarantees it provides,
476 Aligned_alloc::alloc() is able to encode offset and requested
477 allocation datalen into the metadata section without sacrificing memory or
478 making the implementation or end usage more complex.
479
480 Serializing offset into the metadata is what will enable
481 Aligned_alloc_pfs::free() to later on recover original pointer returned
482 by the underlying Aligned_alloc_impl allocation mechanism (std::malloc,
483 std::calloc) and consequently be able to appropriately release it
484 (std::free).
485
486 Serializing requested allocation datalen into the metadata is what will
487 enable higher-kinded functions, implemented on top of
488 Aligned_alloc_pfs, to take necessary actions such as cleaning up the
489 resources by invoking appropriate number of destructors of
490 non-trivially-destructible types. Otherwise, this would create a burden on
491 end users by having to remember and carry the array size all around the
492 code. This is equivalent to what we find in other standard implementations.
493 For example, new int x[10] is always released without passing the array
494 size: delete[] x; The same holds with this design.
495
496 PFS-wise this allocation routine will be storing the information that PFS
497 needs to do its own work:
498 - Owning thread
499 - Total length of bytes allocated
500 - Key
501
502 Memory layout representation looks like the following:
503
504 ------------------------------------------------------------------------------
505 | VARLEN1 | ALIGNED-ALLOC-META | PFS-META | VARLEN2 | PFS-META-OFFSET | DATA |
506 ------------------------------------------------------------------------------
507 ^ ^ ^ ^
508 | | | |
509 | | --------------------------- |
510 | | | OWNER | DATALEN' | KEY | |
511 | | --------------------------- |
512 | | |
513 | ptr returned by |
514 | Aligned_alloc_impl |
515 | |
516 | ptr to be returned to call-site
517 | will be pointing here
518 |
519 ------------------------------
520 | DATALEN | VARLEN1-OFFSET |
521 ------------------------------
522 \ \
523 0 \
524 \
525 alignof(max_align_t) - 1
526
527 VARLEN1 and ALIGNED-ALLOC-META are direct byproduct of Aligned_alloc_impl
528 (and Aligned_alloc_metadata) layout and guarantees.
529
530 VARLEN1 is the leftover variable-length segment that specialized
531 implementations can further make use of by deducing its size from returned
532 offset. Not used by this implementation.
533
534 DATALEN field in ALIGNED-ALLOC-META segment encodes the total length of DATA
535 segment, which is the actual allocation size that client code has requested.
536
537 VARLEN1-OFFSET in ALIGNED-ALLOC-META segment encodes the offset to VARLEN1
538 segment which represents the original pointer obtained by underlying
539 allocation Aligned_alloc_impl mechanism.
540
541 PFS-META, VARLEN2 and PFS-META-OFFSET are memory layout representation of
542 PFS_metadata.
543
544 OWNER field encode the owning thread. DATALEN' field encodes total size
545 of memory consumed and not only the size of the DATA segment. KEY field
546 encodes the PFS/PSI key.
547
548 VARLEN2 is the leftover variable-length segment that specialized
549 implementations can further make use of by deducing its size from the
550 following formulae: requested_alignment - sizeof(PFS-META-OFFSET) -
551 sizeof(PFS-META). In code that would be alignment -
552 PFS_metadata::size. Not used by this implementation.
553
554 PFS-META-OFFSET is a field which allows us to recover the pointer to
555 PFS-META segment from a pointer to DATA segment. Having a pointer to
556 PFS-META segment allows us to deduce the VARLEN1-OFFSET field from
557 ALIGNED-ALLOC-META segment which finally gives us a pointer obtained by the
558 underlying allocation Aligned_alloc_impl mechanism.
559 */
560struct Aligned_alloc_pfs : public allocator_traits<true> {
563
564 /** Dynamically allocates storage of given size at the address aligned to the
565 requested alignment.
566
567 @param[in] size Size of storage (in bytes) requested to be allocated.
568 @param[in] alignment Alignment requirement for storage to be allocated.
569 @param[in] key PSI memory key to be used for PFS memory instrumentation.
570 @return Pointer to the allocated storage. nullptr if dynamic storage
571 allocation failed.
572 */
573 template <bool Zero_initialized>
574 static inline void *alloc(std::size_t size, std::size_t alignment,
576 // We must take special care to allocate enough extra space to hold the
577 // PFS metadata (PFS-META + PFS-META-OFFSET) but we also need to take
578 // special care that the pointer which will be returned to the callee by
579 // this function will still be suitably over-aligned as requested. Both of
580 // these requirements can be fulfilled by finding the smallest multiple of
581 // requested alignment that is not smaller than actual PFS metadata size.
582 const auto metadata_len = calc_align(pfs_metadata::size, alignment);
583 const auto total_len = size + metadata_len;
584 auto ret =
585 Aligned_alloc_impl::alloc<Zero_initialized>(total_len, alignment);
586 if (unlikely(!ret.first)) return nullptr;
587
588 // Same as we do with non-PFS variant of Aligned_alloc::alloc(), here we
589 // encode the offset so we can later on recover the original pointer, P' =
590 // (P - O), from within Aligned_alloc_pfs::free(P) context. Similarly, we
591 // encode the requested allocation datalen.
592 allocator_metadata::meta_1(ret.first, ret.second);
593 allocator_metadata::meta_2(ret.first, size);
594
595#ifdef HAVE_PSI_MEMORY_INTERFACE
596 // When computing total number of bytes allocated. we must not only account
597 // for the size that we have requested (total_len) but we also need
598 // to account for extra memory Aligned_alloc_impl may have allocated in
599 // order to be able to accommodate the request. Amount of extra memory
600 // allocated corresponds to the offset value returned by Aligned_alloc_impl.
601 const auto datalen = total_len + ret.second;
602 // The point of this allocator variant is to trace the memory allocations
603 // through PFS (PSI) so do it.
606 // To be able to do the opposite action of tracing when we are releasing the
607 // memory, we need right about the same data we passed to the tracing
608 // memory_alloc function. Let's encode this it into our allocator so we
609 // don't have to carry and keep this data around.
610 pfs_metadata::pfs_owning_thread(ret.first, owner);
612 pfs_metadata::pfs_key(ret.first, key);
613 pfs_metadata::pfs_metaoffset(ret.first, metadata_len);
614#endif
615
616 return static_cast<uint8_t *>(ret.first) + metadata_len;
617 }
618
619 /** Releases storage dynamically allocated through
620 Aligned_alloc_pfs::alloc().
621
622 @param[in] data Pointer to storage allocated through
623 Aligned_alloc_pfs::alloc()
624 */
625 static inline void free(PFS_metadata::data_segment_ptr data) noexcept {
626 if (unlikely(!data)) return;
627
628#ifdef HAVE_PSI_MEMORY_INTERFACE
629 // Deduce the PFS data we encoded in Aligned_alloc_pfs::alloc()
630 auto key = pfs_metadata::pfs_key(data);
631 auto owner = pfs_metadata::pfs_owning_thread(data);
633 // With the deduced PFS data, now trace the memory release action.
635 (key, datalen, owner);
636#endif
637
638 // Here we make use of the offset which has been encoded by
639 // Aligned_alloc_pfs::alloc() to be able to deduce the original pointer and
640 // simply forward it to std::free.
642 }
643
644 /** Returns the number of bytes requested to be allocated.
645
646 @param[in] data Pointer to storage allocated through
647 Aligned_alloc_pfs::alloc()
648 @return Number of bytes.
649 */
652 // In order to be able to deduce the datalen field, we have to deduce the
653 // beginning of PFS metadata segment first.
655 }
656
657 private:
658 /** Helper function which deduces the original pointer returned by
659 Aligned_alloc_impl from a pointer which is passed to us by the call-site.
660 */
661 static inline void *deduce(PFS_metadata::data_segment_ptr data) noexcept {
662 // To recover the original pointer we need to read the offset
663 // we have serialized into the first (allocator) metadata field. But to read
664 // that offset we have to jump over the PFS metadata first. We use PFS meta
665 // offset for that.
666 auto pfs_meta = pfs_metadata::deduce_pfs_meta(data);
667 auto offset = allocator_metadata::meta_1(pfs_meta);
668 return reinterpret_cast<void *>(reinterpret_cast<std::uintptr_t>(pfs_meta) -
669 offset);
670 }
671};
672
673/** Simple utility metafunction which selects appropriate allocator variant
674 (implementation) depending on the input parameter(s).
675 */
676template <bool Pfs_memory_instrumentation_on>
678 using type =
679 Aligned_alloc; // When PFS is OFF, pick ordinary, non-PFS, variant
680};
681
682template <>
683struct select_alloc_impl<true> {
684 using type = Aligned_alloc_pfs; // Otherwise, pick PFS variant
685};
686
687/** Just a small helper type which saves us some keystrokes. */
688template <bool Pfs_memory_instrumentation_on>
691
692/** Small wrapper which utilizes SFINAE to dispatch the call to appropriate
693 aligned allocator implementation.
694 */
695template <typename Impl>
697 template <bool Zero_initialized, typename T = Impl>
699 alloc(size_t size, size_t alignment, PSI_memory_key key) {
700 return Impl::template alloc<Zero_initialized>(size, alignment, key);
701 }
702 template <bool Zero_initialized, typename T = Impl>
704 alloc(size_t size, size_t alignment, PSI_memory_key /*key*/) {
705 return Impl::template alloc<Zero_initialized>(size, alignment);
706 }
707 static inline void free(void *ptr) { Impl::free(ptr); }
708 static inline size_t datalen(void *ptr) { return Impl::datalen(ptr); }
709};
710
711} // namespace detail
712} // namespace ut
713
714#endif
Simple allocator traits.
#define PSI_MEMORY_CALL(M)
Definition: psi_memory.h:35
static char buf[MAX_BUF]
Definition: conf_to_src.cc:72
constexpr DWORD buf_size
Definition: create_def.cc:227
unsigned int PSI_memory_key
Instrumented memory key.
Definition: psi_memory_bits.h:48
Small helper functions.
#define free(A)
Definition: lexyy.cc:915
Header for compiler-dependent features.
constexpr bool likely(bool expr)
Definition: my_compiler.h:56
constexpr bool unlikely(bool expr)
Definition: my_compiler.h:57
Definition: buf0block_hint.cc:29
Definition: ut0tuple.h:56
static PSI_memory_key memory_alloc(PSI_memory_key, size_t, struct PSI_thread **owner)
Definition: psi_memory_v2_empty.cc:34
static void memory_free(PSI_memory_key, size_t, struct PSI_thread *)
Definition: psi_memory_v2_empty.cc:52
constexpr size_t calc_align(size_t n, size_t m)
Calculates the smallest multiple of m that is not smaller than n when m is a power of two.
Definition: helper.h:44
typename select_alloc_impl< Pfs_memory_instrumentation_on >::type select_alloc_impl_t
Just a small helper type which saves us some keystrokes.
Definition: aligned_alloc.h:690
This file contains a set of libraries providing overloads for regular dynamic allocation routines whi...
Definition: aligned_alloc.h:47
required string key
Definition: replication_asynchronous_connection_failover.proto:59
required string type
Definition: replication_group_member_actions.proto:33
static MEM_ROOT mem
Definition: sql_servers.cc:99
Implementation bits and pieces for PFS metadata handling.
Small wrapper which utilizes SFINAE to dispatch the call to appropriate aligned allocator implementat...
Definition: aligned_alloc.h:696
static void free(void *ptr)
Definition: aligned_alloc.h:707
static size_t datalen(void *ptr)
Definition: aligned_alloc.h:708
static std::enable_if< T::is_pfs_instrumented_v, void * >::type alloc(size_t size, size_t alignment, PSI_memory_key key)
Definition: aligned_alloc.h:699
static std::enable_if<!T::is_pfs_instrumented_v, void * >::type alloc(size_t size, size_t alignment, PSI_memory_key)
Definition: aligned_alloc.h:704
Definition: aligned_alloc.h:50
static constexpr uint32_t metadata_size
Block of memory returned by this functor will have an additional (metadata) segment (at no additional...
Definition: aligned_alloc.h:57
static void free(void *ptr) noexcept
Releases storage allocated through alloc().
Definition: aligned_alloc.h:235
static std::pair< data_segment_ptr, std::size_t > alloc(std::size_t size, std::size_t alignment) noexcept
Dynamically allocates storage of given size and at the address aligned to the requested alignment.
Definition: aligned_alloc.h:113
void * data_segment_ptr
Alias that we will be using to denote ptr to DATA segment.
Definition: aligned_alloc.h:60
Memory layout representation of metadata segment guaranteed by the inner workings of Aligned_alloc_im...
Definition: aligned_alloc.h:274
static constexpr auto allocator_metadata_size
Metadata size.
Definition: aligned_alloc.h:283
static void meta_2(Aligned_alloc_impl::data_segment_ptr data, std::size_t meta_2_v) noexcept
Helper function which stores user-provided detail into the META_2 field.
Definition: aligned_alloc.h:301
static void meta_1(Aligned_alloc_impl::data_segment_ptr data, std::size_t meta_1_v) noexcept
Bail out if we cannot fit the requested data size.
Definition: aligned_alloc.h:294
static meta_1_t meta_1(Aligned_alloc_impl::data_segment_ptr data) noexcept
Helper function which recovers the information user previously stored in META_1 field.
Definition: aligned_alloc.h:310
static unaligned_meta_2_t * ptr_to_meta_2(Aligned_alloc_impl::data_segment_ptr data) noexcept
Helper accessor function to metadata (data length).
Definition: aligned_alloc.h:331
static constexpr auto max_metadata_size
Max metadata size.
Definition: aligned_alloc.h:286
Meta_1_type meta_1_t
Convenience types that we will be using to serialize necessary details into the metadata segment.
Definition: aligned_alloc.h:278
static meta_2_t meta_2(Aligned_alloc_impl::data_segment_ptr data) noexcept
Helper function which recovers the information user previously stored in META_2 field.
Definition: aligned_alloc.h:317
meta_2_t unaligned_meta_2_t
Definition: aligned_alloc.h:280
Meta_2_type meta_2_t
Definition: aligned_alloc.h:279
static meta_1_t * ptr_to_meta_1(Aligned_alloc_impl::data_segment_ptr data) noexcept
Helper accessor function to metadata (offset).
Definition: aligned_alloc.h:326
Aligned allocation routines which are instrumented through PFS (performance-schema).
Definition: aligned_alloc.h:560
static void * alloc(std::size_t size, std::size_t alignment, pfs_metadata::pfs_memory_key_t key)
Dynamically allocates storage of given size at the address aligned to the requested alignment.
Definition: aligned_alloc.h:574
static void free(PFS_metadata::data_segment_ptr data) noexcept
Releases storage dynamically allocated through Aligned_alloc_pfs::alloc().
Definition: aligned_alloc.h:625
static void * deduce(PFS_metadata::data_segment_ptr data) noexcept
Helper function which deduces the original pointer returned by Aligned_alloc_impl from a pointer whic...
Definition: aligned_alloc.h:661
static allocator_metadata::meta_2_t datalen(PFS_metadata::data_segment_ptr data)
Returns the number of bytes requested to be allocated.
Definition: aligned_alloc.h:650
Aligned allocation routines.
Definition: aligned_alloc.h:393
static void * alloc(std::size_t size, std::size_t alignment)
Dynamically allocates storage of given size and at the address aligned to the requested alignment.
Definition: aligned_alloc.h:405
static void * deduce(Aligned_alloc_impl::data_segment_ptr data) noexcept
Helper function which deduces the original pointer returned by Aligned_alloc_impl from a pointer whic...
Definition: aligned_alloc.h:461
static allocator_metadata::meta_2_t datalen(Aligned_alloc_impl::data_segment_ptr data)
Returns the number of bytes requested to be allocated.
Definition: aligned_alloc.h:450
static void free(Aligned_alloc_impl::data_segment_ptr data) noexcept
Releases storage dynamically allocated through Aligned_alloc::alloc().
Definition: aligned_alloc.h:436
static void free(void *ptr)
Definition: allocator_traits.h:104
Memory layout representation of PFS metadata segment that is used by the allocator variants which als...
Definition: pfs.h:85
void * data_segment_ptr
Definition: pfs.h:93
static void pfs_datalen(data_segment_ptr data, size_t datalen) noexcept
Helper function which stores the PFS datalen info into the DATALEN field.
Definition: pfs.h:108
static void pfs_key(data_segment_ptr data, pfs_memory_key_t key) noexcept
Helper function which stores the PFS key info into the KEY field.
Definition: pfs.h:114
PSI_thread * pfs_owning_thread_t
Convenience types that we will be using to serialize necessary details into the Aligned_alloc metadat...
Definition: pfs.h:89
static constexpr auto size
Definition: pfs.h:99
static void pfs_owning_thread(data_segment_ptr data, pfs_owning_thread_t thread) noexcept
Helper function which stores the PFS thread info into the OWNER field.
Definition: pfs.h:102
static void pfs_metaoffset(data_segment_ptr data, std::size_t alignment) noexcept
Helper function which stores the offset to PFS metadata segment into the PFS-META-OFFSET field.
Definition: pfs.h:121
PSI_memory_key pfs_memory_key_t
Definition: pfs.h:91
static void * deduce_pfs_meta(data_segment_ptr data) noexcept
Helper function which deduces the pointer to the beginning of PFS metadata segment given the pointer ...
Definition: pfs.h:155
Simple allocator traits.
Definition: allocator_traits.h:81
Simple utility metafunction which selects appropriate allocator variant (implementation) depending on...
Definition: aligned_alloc.h:677