MySQL 8.0.30
Source Code Documentation
large_page_alloc.h
Go to the documentation of this file.
1/*****************************************************************************
2
3Copyright (c) 2021, 2022, Oracle and/or its affiliates.
4
5This program is free software; you can redistribute it and/or modify it under
6the terms of the GNU General Public License, version 2.0, as published by the
7Free Software Foundation.
8
9This program is also distributed with certain software (including but not
10limited to OpenSSL) that is licensed under separate terms, as designated in a
11particular file or component or in included license documentation. The authors
12of MySQL hereby grant you an additional permission to link the program and
13your derivative works with the separately licensed software that they have
14included with MySQL.
15
16This program is distributed in the hope that it will be useful, but WITHOUT
17ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
18FOR A PARTICULAR PURPOSE. See the GNU General Public License, version 2.0,
19for more details.
20
21You should have received a copy of the GNU General Public License along with
22this program; if not, write to the Free Software Foundation, Inc.,
2351 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
24
25*****************************************************************************/
26
27/** @file include/detail/ut/large_page_alloc.h
28 Implementation bits and pieces for large (huge) page allocations. */
29
30#ifndef detail_ut_large_page_alloc_h
31#define detail_ut_large_page_alloc_h
32
33#include <cstddef>
34#include <fstream>
35
36#include "my_compiler.h"
37#ifdef _WIN32
39#elif defined(__APPLE__)
41#elif defined(__sun)
43#else
45#endif
50
51extern const size_t large_page_default_size;
52
53namespace ut {
54namespace detail {
55
56/** Allocation routines which are purposed for allocating memory through the
57 means of what is known as large (huge) pages.
58
59 large_page_aligned_alloc() and large_page_aligned_free() are taking care of
60 OS specific details and Large_page_alloc is a convenience wrapper which only
61 makes the use of large pages more ergonomic so that it serializes the actual
62 size being allocated into the raw memory. This size is then automagically
63 deduced when large page memory is being freed. Otherwise, client code would
64 have been responsible to store and keep that value somewhere until it frees
65 the large page memory segment. Additionally, information on type of page
66 used to back up requested allocation is also serialized into the memory
67 allowing to build higher-kinded abstractions more easily. See
68 ut::malloc_large_page with option to fallback to regular pages through
69 ut::malloc_page.
70
71 Cost associated with this abstraction is the size of a single CPU page. In
72 terms of virtual memory, especially in 64-bit address space, this cost is
73 negligible. In practice this means that for each N huge-page sized
74 allocation request, application code will get to use CPU_PAGE_SIZE bytes
75 less. In other words, for a request that is backed up by three 2MiB
76 huge-pages, application code will get to use 3 * 2MiB - CPU_PAGE_SIZE of
77 total bytes. CPU_PAGE_SIZE is normally 4K but some architectures such as
78 SPARC have it set to 8K. ARM64 can be set to 4K, 8K or 64K.
79
80 Memory layout representation looks like the following:
81
82 -------------------------------------------
83 | PAGE-ALLOC-METADATA | ... DATA ... |
84 -------------------------------------------
85 ^ ^
86 | |
87 | |
88 | ptr (large-page) to be returned to call-site
89 |
90 --------------------------------
91 | DATALEN | PAGE-TYPE | VARLEN |
92 --------------------------------
93 \ \
94 0 \
95 CPU_PAGE_SIZE - 1
96
97 For details on DATALEN, PAGE-TYPE and VARLEN fields see Page_alloc_metadata.
98
99 DATA is an actual page-aligned (!) segment backed by large (huge) page
100 memory that will be returned to the call-site and which the client code will
101 be able to use for the application data.
102 */
103struct Large_page_alloc : public allocator_traits<false> {
105
106 /** Allocates memory through large-page support.
107
108 @param[in] size Size of storage (in bytes) requested to be allocated.
109 @return Pointer to the allocated storage. nullptr if allocation failed.
110 */
111 static inline void *alloc(std::size_t size) {
112 auto total_len = round_to_next_multiple(
114 auto mem = large_page_aligned_alloc(total_len);
115 if (unlikely(!mem)) return nullptr;
118 return static_cast<uint8_t *>(mem) + page_allocation_metadata::len;
119 }
120
121 /** Releases storage allocated through
122 Large_page_alloc::alloc().
123
124 @param[in] data Pointer to storage allocated through
125 Large_page_alloc::alloc()
126 @return True if releasing the memory was successful.
127 */
128 static inline bool free(void *data) noexcept {
129 if (unlikely(!data)) return false;
131 return large_page_aligned_free(deduce(data),
133 }
134
135 /** Returns the number of bytes that have been allocated.
136
137 @param[in] data Pointer to storage allocated through
138 Large_page_alloc::alloc()
139 @return Number of bytes.
140 */
145 }
146
147 /** Returns the type of the page.
148
149 @param[in] data Pointer to storage allocated through
150 Large_page_alloc::alloc()
151 @return Page type.
152 */
153 static inline Page_type page_type(void *data) {
155 }
156
157 /** Retrieves the pointer and size of the allocation provided by the OS. It is
158 a low level information, and is needed only to call low level
159 memory-related OS functions.
160
161 @param[in] data Pointer to storage allocated through
162 Large_page_alloc::alloc()
163 @return Low level allocation info.
164 */
165 static inline allocation_low_level_info low_level_info(void *data) {
167 return {deduce(data), page_allocation_metadata::datalen(data)};
168 }
169
170 private:
171 /** Helper function which deduces the original pointer returned by
172 Large_page_alloc from a pointer which is passed to us by the call-site.
173 */
174 static inline void *deduce(void *data) noexcept {
176 const auto res = reinterpret_cast<void *>(static_cast<uint8_t *>(data) -
178 ut_ad(reinterpret_cast<std::uintptr_t>(res) % large_page_size() == 0);
179 return res;
180 }
181};
182
183/** Allocation routines which are purposed for allocating memory through the
184 means of what is known as large (huge) pages. This is a PFS
185 (performance-schema) variant of Large_page_alloc. Implemented in terms
186 of Page_alloc_metadata_pfs.
187
188 large_page_aligned_alloc() and large_page_aligned_free() are taking care of
189 OS specific details and Large_page_alloc is a convenience wrapper which only
190 makes the use of large pages more ergonomic so that it serializes all the
191 relevant PFS details into the raw memory. Otherwise, client code would have
192 been responsible to store and keep those details somewhere until the memory
193 segment is freed. Additionally, information on type of page used to back up
194 requested allocation is also serialized into the memory allowing to build
195 higher-kinded abstractions more easily. See ut::malloc_large_page with
196 option to fallback to regular pages through ut::malloc_page.
197
198 Cost associated with this abstraction is the size of a single CPU page. In
199 terms of virtual memory, especially in 64-bit address space, this cost is
200 negligible. In practice this means that for each N huge-page sized
201 allocation request, application code will get to use CPU_PAGE_SIZE bytes
202 less. In other words, for a request that is backed up by three 2MiB
203 huge-pages, application code will get to use 3 * 2MiB - CPU_PAGE_SIZE of
204 total bytes. CPU_PAGE_SIZE is normally 4K but some architectures such as
205 SPARC have it set to 8K. ARM64 can be set to 4K, 8K or 64K.
206
207 Memory layout representation looks like the following:
208
209 ----------------------------------------------
210 | PAGE-ALLOC-METADATA-PFS | ... DATA ... |
211 ----------------------------------------------
212 ^ ^
213 | |
214 | |
215 | ptr (large-page) to be
216 | returned to call-site
217 |
218 ---------------------------------------------------
219 | PFS-META | PAGE-TYPE | VARLEN | PFS-META-OFFSET |
220 ---------------------------------------------------
221 ^ ^
222 | |
223 | ---------------------------
224 | | OWNER | DATALEN | KEY |
225 | ---------------------------
226 |
227 ptr returned by
228 large_page_aligned_alloc
229
230 For details on PFS-META, PAGE-TYPE, VARLEN and PFS-META-OFFSET fields
231 see Page_alloc_metadata_pfs.
232
233 DATA is an actual page-aligned (!) segment backed by large (huge) page
234 memory that will be returned to the call-site and which the client code will
235 be able to use for the application data.
236 */
239
240 /** Allocates memory through large-page support.
241
242 @param[in] size Size of storage (in bytes) requested to be allocated.
243 @param[in] key PSI memory key to be used for PFS memory instrumentation.
244 @return Pointer to the allocated storage. nullptr if allocation failed.
245 */
246 static inline void *alloc(
247 std::size_t size,
249 auto total_len = round_to_next_multiple(
251 auto mem = large_page_aligned_alloc(total_len);
252 if (unlikely(!mem)) return nullptr;
253
254#ifdef HAVE_PSI_MEMORY_INTERFACE
255 // The point of this allocator variant is to trace the memory allocations
256 // through PFS (PSI) so do it.
258 key = PSI_MEMORY_CALL(memory_alloc)(key, total_len, &owner);
259 // To be able to do the opposite action of tracing when we are releasing the
260 // memory, we need right about the same data we passed to the tracing
261 // memory_alloc function. Let's encode this it into our allocator so we
262 // don't have to carry and keep this data around.
269#endif
270
271 return static_cast<uint8_t *>(mem) + page_allocation_metadata::len;
272 }
273
274 /** Releases storage allocated through Large_page_alloc_pfs::alloc().
275
276 @param[in] data Pointer to storage allocated through
277 Large_page_alloc_pfs::alloc()
278 @return True if releasing the memory was successful.
279 */
280 static inline bool free(PFS_metadata::data_segment_ptr data) noexcept {
281 if (unlikely(!data)) return false;
283
284 PFS_metadata::pfs_datalen_t total_len = {};
285#ifdef HAVE_PSI_MEMORY_INTERFACE
286 // Deduce the PFS data we encoded in Large_page_alloc_pfs::alloc()
288 auto owner =
291 // With the deduced PFS data, now trace the memory release action.
292 PSI_MEMORY_CALL(memory_free)
293 (key, total_len, owner);
294#endif
295
296 return large_page_aligned_free(deduce(data), total_len);
297 }
298
299 /** Returns the number of bytes that have been allocated.
300
301 @param[in] data Pointer to storage allocated through
302 Large_page_alloc_pfs::alloc()
303 @return Number of bytes.
304 */
305 static inline size_t datalen(PFS_metadata::data_segment_ptr data) {
309 }
310
311 /** Returns the Page_type.
312
313 @param[in] data Pointer to storage allocated through
314 Large_page_alloc_pfs::alloc()
315 @return Page type.
316 */
319 }
320
321 /** Retrieves the pointer and size of the allocation provided by the OS. It is
322 a low level information, and is needed only to call low level
323 memory-related OS functions.
324
325 @param[in] data Pointer to storage allocated through
326 Large_page_alloc_pfs::alloc()
327 @return Low level allocation info.
328 */
329 static inline allocation_low_level_info low_level_info(void *data) {
331 return {deduce(data),
333 }
334
335 private:
336 /** Helper function which deduces the original pointer returned by
337 Large_page_alloc_pfs from a pointer which is passed to us by the
338 call-site.
339 */
340 static inline void *deduce(PFS_metadata::data_segment_ptr data) noexcept {
342 const auto res =
344 ut_ad(reinterpret_cast<std::uintptr_t>(res) % large_page_size() == 0);
345 return res;
346 }
347};
348
349/** Simple utility metafunction which selects appropriate allocator variant
350 (implementation) depending on the input parameter(s).
351 */
352template <bool Pfs_memory_instrumentation_on>
354 using type =
355 Large_page_alloc; // When PFS is OFF, pick ordinary, non-PFS, variant
356};
357
358template <>
360 using type = Large_page_alloc_pfs; // Otherwise, pick PFS variant
361};
362
363/** Just a small helper type which saves us some keystrokes. */
364template <bool Pfs_memory_instrumentation_on>
367
368/** Small wrapper which utilizes SFINAE to dispatch the call to appropriate
369 aligned allocator implementation.
370 */
371template <typename Impl>
373 template <typename T = Impl>
375 alloc(size_t size, PSI_memory_key key) {
376 return Impl::alloc(size, key);
377 }
378 template <typename T = Impl>
380 alloc(size_t size, PSI_memory_key /*key*/) {
381 return Impl::alloc(size);
382 }
383 static inline bool free(void *ptr) { return Impl::free(ptr); }
384 static inline size_t datalen(void *ptr) { return Impl::datalen(ptr); }
385 static inline Page_type page_type(void *ptr) { return Impl::page_type(ptr); }
386 static inline allocation_low_level_info low_level_info(void *ptr) {
387 return Impl::low_level_info(ptr);
388 }
389};
390
391} // namespace detail
392} // namespace ut
393
394#endif
Simple allocator traits.
#define PSI_MEMORY_CALL(M)
Definition: psi_memory.h:35
unsigned int PSI_memory_key
Instrumented memory key.
Definition: psi_memory_bits.h:48
Small helper functions.
Linux-specific implementation bits and pieces for large (huge) page allocations.
OSX-specific implementation bits and pieces for large (huge) page allocations.
Solaris-specific implementation bits and pieces for large (huge) page allocations.
Windows-specific implementation bits and pieces for large (huge) page allocations.
const size_t large_page_default_size
System-default huge (large) page setting.
Definition: ut0new.cc:39
#define free(A)
Definition: lexyy.cc:915
Header for compiler-dependent features.
constexpr bool unlikely(bool expr)
Definition: my_compiler.h:55
Definition: ut0tuple.h:56
void * large_page_aligned_alloc(size_t n_bytes)
Allocates memory backed by large (huge) pages.
Definition: large_page_alloc-linux.h:51
Page_type
Types of pages currently supported by ut:: library functions.
Definition: page_metadata.h:42
typename select_large_page_alloc_impl< Pfs_memory_instrumentation_on >::type select_large_page_alloc_impl_t
Just a small helper type which saves us some keystrokes.
Definition: large_page_alloc.h:366
constexpr size_t round_to_next_multiple(size_t n, size_t m)
Calculates the next multiple of m that is bigger or equal to n.
Definition: helper.h:67
bool large_page_aligned_free(void *ptr, size_t n_bytes)
Releases memory backed by large (huge) pages.
Definition: large_page_alloc-linux.h:74
size_t large_page_size()
Queries the current size of large (huge) pages on running system.
Definition: large_page_alloc-linux.h:94
This file contains a set of libraries providing overloads for regular dynamic allocation routines whi...
Definition: aligned_alloc.h:47
Implementation bits and pieces for metadata for normal and large (huge) page allocations.
required string key
Definition: replication_asynchronous_connection_failover.proto:59
required string type
Definition: replication_group_member_actions.proto:33
static MEM_ROOT mem
Definition: sql_servers.cc:98
Implementation bits and pieces for PFS metadata handling.
Can be used to extract pointer and size of the allocation provided by the OS.
Definition: ut0new.h:138
Small wrapper which utilizes SFINAE to dispatch the call to appropriate aligned allocator implementat...
Definition: large_page_alloc.h:372
static bool free(void *ptr)
Definition: large_page_alloc.h:383
static std::enable_if<!T::is_pfs_instrumented_v, void * >::type alloc(size_t size, PSI_memory_key)
Definition: large_page_alloc.h:380
static Page_type page_type(void *ptr)
Definition: large_page_alloc.h:385
static std::enable_if< T::is_pfs_instrumented_v, void * >::type alloc(size_t size, PSI_memory_key key)
Definition: large_page_alloc.h:375
static size_t datalen(void *ptr)
Definition: large_page_alloc.h:384
static allocation_low_level_info low_level_info(void *ptr)
Definition: large_page_alloc.h:386
Allocation routines which are purposed for allocating memory through the means of what is known as la...
Definition: large_page_alloc.h:237
static allocation_low_level_info low_level_info(void *data)
Retrieves the pointer and size of the allocation provided by the OS.
Definition: large_page_alloc.h:329
static void * deduce(PFS_metadata::data_segment_ptr data) noexcept
Helper function which deduces the original pointer returned by Large_page_alloc_pfs from a pointer wh...
Definition: large_page_alloc.h:340
static void * alloc(std::size_t size, page_allocation_metadata::pfs_metadata::pfs_memory_key_t key)
Allocates memory through large-page support.
Definition: large_page_alloc.h:246
static size_t datalen(PFS_metadata::data_segment_ptr data)
Returns the number of bytes that have been allocated.
Definition: large_page_alloc.h:305
static Page_type page_type(PFS_metadata::data_segment_ptr data)
Returns the Page_type.
Definition: large_page_alloc.h:317
static bool free(PFS_metadata::data_segment_ptr data) noexcept
Releases storage allocated through Large_page_alloc_pfs::alloc().
Definition: large_page_alloc.h:280
Allocation routines which are purposed for allocating memory through the means of what is known as la...
Definition: large_page_alloc.h:103
static bool free(void *data) noexcept
Releases storage allocated through Large_page_alloc::alloc().
Definition: large_page_alloc.h:128
static void * deduce(void *data) noexcept
Helper function which deduces the original pointer returned by Large_page_alloc from a pointer which ...
Definition: large_page_alloc.h:174
static void * alloc(std::size_t size)
Allocates memory through large-page support.
Definition: large_page_alloc.h:111
static Page_type page_type(void *data)
Returns the type of the page.
Definition: large_page_alloc.h:153
static page_allocation_metadata::datalen_t datalen(void *data)
Returns the number of bytes that have been allocated.
Definition: large_page_alloc.h:141
static allocation_low_level_info low_level_info(void *data)
Retrieves the pointer and size of the allocation provided by the OS.
Definition: large_page_alloc.h:165
void * data_segment_ptr
Definition: pfs.h:93
std::size_t pfs_datalen_t
Definition: pfs.h:90
static void pfs_datalen(data_segment_ptr data, size_t datalen) noexcept
Helper function which stores the PFS datalen info into the DATALEN field.
Definition: pfs.h:108
static void pfs_key(data_segment_ptr data, pfs_memory_key_t key) noexcept
Helper function which stores the PFS key info into the KEY field.
Definition: pfs.h:114
PSI_thread * pfs_owning_thread_t
Convenience types that we will be using to serialize necessary details into the Aligned_alloc metadat...
Definition: pfs.h:89
static void pfs_owning_thread(data_segment_ptr data, pfs_owning_thread_t thread) noexcept
Helper function which stores the PFS thread info into the OWNER field.
Definition: pfs.h:102
static void pfs_metaoffset(data_segment_ptr data, std::size_t alignment) noexcept
Helper function which stores the offset to PFS metadata segment into the PFS-META-OFFSET field.
Definition: pfs.h:121
PSI_memory_key pfs_memory_key_t
Definition: pfs.h:91
static void * deduce_pfs_meta(data_segment_ptr data) noexcept
Helper function which deduces the pointer to the beginning of PFS metadata segment given the pointer ...
Definition: pfs.h:155
Helper struct implementing the type which represents the metadata for all types of PFS-aware page-ali...
Definition: page_metadata.h:197
static constexpr auto len
This is how much this metadata segment will be big.
Definition: page_metadata.h:202
static Page_type page_type(void *data)
Sanity check so that we can be sure that the size of our metadata segment is such so that the pointer...
Definition: page_metadata.h:220
Helper struct implementing the type which represents the metadata for all types of page-aligned alloc...
Definition: page_metadata.h:74
static constexpr auto len
This is how much tise metadata segment will be big.
Definition: page_metadata.h:76
static Page_type page_type(void *data)
Accessor to the page_type_t field.
Definition: page_metadata.h:109
static datalen_t datalen(void *data)
Sanity check so that we can be sure that the size of our metadata segment is such so that the next se...
Definition: page_metadata.h:100
size_t datalen_t
These are the types representing our memory fields.
Definition: page_metadata.h:79
Simple allocator traits.
Definition: allocator_traits.h:81
Simple utility metafunction which selects appropriate allocator variant (implementation) depending on...
Definition: large_page_alloc.h:353
#define ut_ad(EXPR)
Debug assertion.
Definition: ut0dbg.h:68