MySQL  8.0.20
Source Code Documentation
ut0new.h File Reference

Instrumented memory allocator. More...

#include <algorithm>
#include <cerrno>
#include <cstddef>
#include <cstdlib>
#include <cstring>
#include <limits>
#include <map>
#include <type_traits>
#include "my_basename.h"
#include "mysql/psi/mysql_memory.h"
#include "mysql/psi/psi_base.h"
#include "mysql/psi/psi_memory.h"
#include "os0proc.h"
#include "os0thread.h"
#include "univ.i"
#include "ut0byte.h"
#include "ut0counter.h"
#include "ut0ut.h"

Go to the source code of this file.

Classes

struct  force_constexpr< Value >
 
struct  ut_new_pfx_t
 A structure that holds the necessary data for performance schema accounting. More...
 
class  ut_allocator< T >
 Allocator class for allocating memory from inside std::* containers. More...
 
struct  ut_allocator< T >::rebind< U >
 
class  aligned_memory< T_Type, T_Align_to >
 Abstract class to manage an object that is aligned to specified number of bytes. More...
 
class  aligned_pointer< T_Type, T_Align_to >
 Manages an object that is aligned to specified number of bytes. More...
 
class  aligned_array_pointer< T_Type, T_Align_to >
 Manages an array of objects. More...
 

Namespaces

 ut
 

Macros

#define OUT_OF_MEMORY_MSG
 Dynamic memory allocation within InnoDB guidelines. More...
 
#define UT_NEW_THIS_FILE_PSI_INDEX   (force_constexpr<ut_new_get_key_by_file(MY_BASENAME)>::value)
 
#define UT_NEW_THIS_FILE_PSI_KEY
 
#define UT_NEW(expr, key)
 Allocate, trace the allocation and construct an object. More...
 
#define UT_NEW_NOKEY(expr)   UT_NEW(expr, PSI_NOT_INSTRUMENTED)
 Allocate, trace the allocation and construct an object. More...
 
#define UT_DELETE(ptr)   ut_delete(ptr)
 Destroy, deallocate and trace the deallocation of an object created by UT_NEW() or UT_NEW_NOKEY(). More...
 
#define UT_NEW_ARRAY(type, n_elements, key)   ut_allocator<type>(key).new_array(n_elements, UT_NEW_THIS_FILE_PSI_KEY)
 Allocate and account 'n_elements' objects of type 'type'. More...
 
#define UT_NEW_ARRAY_NOKEY(type, n_elements)   UT_NEW_ARRAY(type, n_elements, PSI_NOT_INSTRUMENTED)
 Allocate and account 'n_elements' objects of type 'type'. More...
 
#define UT_DELETE_ARRAY(ptr)   ut_delete_array(ptr)
 Destroy, deallocate and trace the deallocation of an array created by UT_NEW_ARRAY() or UT_NEW_ARRAY_NOKEY(). More...
 
#define ut_malloc(n_bytes, key)
 Do not use ut_malloc, ut_zalloc, ut_malloc_nokey, ut_zalloc_nokey, ut_zalloc_nokey_nofatal and ut_realloc when allocating memory for over-aligned types. More...
 
#define ut_zalloc(n_bytes, key)
 
#define ut_malloc_nokey(n_bytes)
 
#define ut_zalloc_nokey(n_bytes)
 
#define ut_zalloc_nokey_nofatal(n_bytes)
 
#define ut_realloc(ptr, n_bytes)
 
#define ut_free(ptr)
 

Typedefs

using ut::ostringstream = std::basic_ostringstream< char, std::char_traits< char >, ut_allocator< char > >
 Specialization of basic_ostringstream which uses ut_allocator. More...
 
template<typename T >
using ut::vector = std::vector< T, ut_allocator< T > >
 Specialization of vector which uses ut_allocator. More...
 

Functions

void ut_new_boot ()
 Setup the internal objects needed for UT_NEW() to operate. More...
 
void ut_new_boot_safe ()
 Setup the internal objects needed for UT_NEW() to operate. More...
 
constexpr bool ut_string_begins_with (const char *a, const char *b, size_t b_len)
 gcc 5 fails to evalutate costexprs at compile time. More...
 
constexpr size_t ut_len_without_extension (const char *file)
 Find the length of the filename without its file extension. More...
 
constexpr int ut_new_get_key_by_base_file (const char *file, size_t len)
 Retrieve a memory key (registered with PFS), given the file name of the caller. More...
 
constexpr int ut_new_get_key_by_file (const char *file)
 Retrieve a memory key (registered with PFS), given the file name of the caller. More...
 
template<typename T >
bool operator== (const ut_allocator< T > &lhs, const ut_allocator< T > &rhs)
 Compare two allocators of the same type. More...
 
template<typename T >
bool operator!= (const ut_allocator< T > &lhs, const ut_allocator< T > &rhs)
 Compare two allocators of the same type. More...
 
template<typename T >
void ut_delete (T *ptr)
 Destroy and account object created by UT_NEW() or UT_NEW_NOKEY(). More...
 
template<typename T >
void ut_delete_array (T *ptr)
 Destroy and account objects created by UT_NEW_ARRAY() or UT_NEW_ARRAY_NOKEY(). More...
 
UNIV_INLINE void * ut_align (const void *ptr, ulint align_no)
 This is a forward declaration, which is because of the circular dependency between ut0new.h and ut0byte.h (going through univ.i and sync0types.h). More...
 

Variables

const size_t alloc_max_retries
 Maximum number of retries to allocate memory. More...
 
PSI_memory_key mem_key_ahi
 Keys for registering allocations with performance schema. More...
 
PSI_memory_key mem_key_archive
 
PSI_memory_key mem_key_buf_buf_pool
 
PSI_memory_key mem_key_buf_stat_per_index_t
 
PSI_memory_key mem_key_clone
 Memory key for clone. More...
 
PSI_memory_key mem_key_dict_stats_bg_recalc_pool_t
 
PSI_memory_key mem_key_dict_stats_index_map_t
 
PSI_memory_key mem_key_dict_stats_n_diff_on_level
 
PSI_memory_key mem_key_redo_log_archive_queue_element
 
PSI_memory_key mem_key_other
 
PSI_memory_key mem_key_partitioning
 
PSI_memory_key mem_key_row_log_buf
 
PSI_memory_key mem_key_row_merge_sort
 
PSI_memory_key mem_key_std
 
PSI_memory_key mem_key_trx_sys_t_rw_trx_ids
 
PSI_memory_key mem_key_undo_spaces
 
PSI_memory_key mem_key_ut_lock_free_hash_t
 
static constexpr const char * auto_event_names []
 List of filenames that allocate memory and are instrumented via PFS. More...
 
static constexpr size_t n_auto = UT_ARR_SIZE(auto_event_names)
 
PSI_memory_key auto_event_keys [n_auto]
 
PSI_memory_info pfs_info_auto [n_auto]
 

Detailed Description

Instrumented memory allocator.

Created May 26, 2014 Vasil Dimov

Macro Definition Documentation

◆ OUT_OF_MEMORY_MSG

#define OUT_OF_MEMORY_MSG
Value:
"Check if you should increase the swap file or ulimits of your" \
" operating system. Note that on most 32-bit computers the process" \
" memory space is limited to 2 GB or 4 GB."

Dynamic memory allocation within InnoDB guidelines.

All dynamic (heap) memory allocations (malloc(3), strdup(3), etc, "new", various std:: containers that allocate memory internally), that are done within InnoDB are instrumented. This means that InnoDB uses a custom set of functions for allocating memory, rather than calling e.g. "new" directly.

Here follows a cheat sheet on what InnoDB functions to use whenever a standard one would have been used.

Creating new objects with "new":

Standard: new expression or new(std::nothrow) expression InnoDB, default instrumentation: UT_NEW_NOKEY(expression) InnoDB, custom instrumentation, preferred: UT_NEW(expression, key)

Destroying objects, created with "new":

Standard: delete ptr InnoDB: UT_DELETE(ptr)

Creating new arrays with "new[]":

Standard: new type[num] or new(std::nothrow) type[num] InnoDB, default instrumentation: UT_NEW_ARRAY_NOKEY(type, num) InnoDB, custom instrumentation, preferred: UT_NEW_ARRAY(type, num, key)

Destroying arrays, created with "new[]":

Standard: delete[] ptr InnoDB: UT_DELETE_ARRAY(ptr)

Declaring a type with a std:: container, e.g. std::vector:

Standard: std::vector<t> InnoDB: std::vector<t, ut_allocator<t> >

Declaring objects of some std:: type:

Standard: std::vector<t> v InnoDB, default instrumentation: std::vector<t, ut_allocator<t> > v InnoDB, custom instrumentation, preferred: std::vector<t, ut_allocator<t> > v(ut_allocator<t>(key))

Raw block allocation (as usual in C++, consider whether using "new" would

not be more appropriate):

Standard: malloc(num) InnoDB, default instrumentation: ut_malloc_nokey(num) InnoDB, custom instrumentation, preferred: ut_malloc(num, key)

Raw block resize:

Standard: realloc(ptr, new_size) InnoDB: ut_realloc(ptr, new_size)

Raw block deallocation:

Standard: free(ptr) InnoDB: ut_free(ptr)

Note: the expression passed to UT_NEW() or UT_NEW_NOKEY() must always end with (), thus: Standard: new int InnoDB: UT_NEW_NOKEY(int())

◆ UT_DELETE

#define UT_DELETE (   ptr)    ut_delete(ptr)

Destroy, deallocate and trace the deallocation of an object created by UT_NEW() or UT_NEW_NOKEY().

We can't instantiate ut_allocator without having the type of the object, thus we redirect this to a template function.

◆ UT_DELETE_ARRAY

#define UT_DELETE_ARRAY (   ptr)    ut_delete_array(ptr)

Destroy, deallocate and trace the deallocation of an array created by UT_NEW_ARRAY() or UT_NEW_ARRAY_NOKEY().

We can't instantiate ut_allocator without having the type of the object, thus we redirect this to a template function.

◆ ut_free

#define ut_free (   ptr)
Value:
.deallocate(reinterpret_cast<byte *>(ptr))
Allocator class for allocating memory from inside std::* containers.
Definition: ut0new.h:571
#define PSI_NOT_INSTRUMENTED
Definition: validate_password_imp.cc:39

◆ ut_malloc

#define ut_malloc (   n_bytes,
  key 
)
Value:
static_cast<void *>(ut_allocator<byte>(key).allocate( \
n_bytes, NULL, UT_NEW_THIS_FILE_PSI_KEY, false, false))
Allocator class for allocating memory from inside std::* containers.
Definition: ut0new.h:571
#define UT_NEW_THIS_FILE_PSI_KEY
Definition: ut0new.h:514
#define NULL
Definition: types.h:55
static const char * key
Definition: suite_stubs.c:14

Do not use ut_malloc, ut_zalloc, ut_malloc_nokey, ut_zalloc_nokey, ut_zalloc_nokey_nofatal and ut_realloc when allocating memory for over-aligned types.

We have to use aligned_pointer instead, analogously to how we have to use aligned_alloc when working with the standard library to handle dynamic allocation for over-aligned types. These macros use ut_allocator to allocate raw memory (no type information is passed). This is why ut_allocator needs to be instantiated with the byte type. This has implications on the max alignment of the objects that are allocated using this API. ut_allocator returns memory aligned to alignof(std::max_align_t), similarly to library allocation functions. This value is 16 bytes on most x64 machines. A static_assert enforces this when using UT_NEW, however, since the ut_allocator template is instantiated with byte here the assert will not be hit if using alignment >= alignof(std::max_align_t). Not meeting the alignment requirements for a type causes undefined behaviour. One should avoid using the macros below when writing new code in general, and try to remove them when refactoring existing code (in favor of using the UT_NEW). The reason behind this lies both in the undefined behaviour problem described above, and in the fact that standard C-like malloc use is discouraged in c++ (see CppCoreGuidelines - R.10: Avoid malloc() and free()). Using ut_malloc has the same problems as the standard library malloc.

◆ ut_malloc_nokey

#define ut_malloc_nokey (   n_bytes)
Value:
static_cast<void *>( \
ut_allocator<byte>(PSI_NOT_INSTRUMENTED) \
.allocate(n_bytes, NULL, UT_NEW_THIS_FILE_PSI_KEY, false, false))
#define PSI_NOT_INSTRUMENTED
Definition: validate_password_imp.cc:39
#define UT_NEW_THIS_FILE_PSI_KEY
Definition: ut0new.h:514
#define NULL
Definition: types.h:55

◆ UT_NEW

#define UT_NEW (   expr,
  key 
)
Value:
/* Placement new will return NULL and not attempt to construct an \
object if the passed in pointer is NULL, e.g. if allocate() has \
failed to allocate memory and has returned NULL. */ \
::new (ut_allocator<decltype(expr)>(key).allocate(1, NULL, key, false, \
false)) expr
Allocator class for allocating memory from inside std::* containers.
Definition: ut0new.h:571
#define NULL
Definition: types.h:55
static const char * key
Definition: suite_stubs.c:14

Allocate, trace the allocation and construct an object.

Use this macro instead of 'new' within InnoDB. For example: instead of Foo* f = new Foo(args); use: Foo* f = UT_NEW(Foo(args), mem_key_some); Upon failure to allocate the memory, this macro may return NULL. It will not throw exceptions. After successful allocation the returned pointer must be passed to UT_DELETE() when no longer needed.

Parameters
[in]exprany expression that could follow "new"
[in]keyperformance schema memory tracing key
Returns
pointer to the created object or NULL

◆ UT_NEW_ARRAY

#define UT_NEW_ARRAY (   type,
  n_elements,
  key 
)    ut_allocator<type>(key).new_array(n_elements, UT_NEW_THIS_FILE_PSI_KEY)

Allocate and account 'n_elements' objects of type 'type'.

Use this macro to allocate memory within InnoDB instead of 'new[]'. The returned pointer must be passed to UT_DELETE_ARRAY().

Parameters
[in]typetype of objects being created
[in]n_elementsnumber of objects to create
[in]keyperformance schema memory tracing key
Returns
pointer to the first allocated object or NULL

◆ UT_NEW_ARRAY_NOKEY

#define UT_NEW_ARRAY_NOKEY (   type,
  n_elements 
)    UT_NEW_ARRAY(type, n_elements, PSI_NOT_INSTRUMENTED)

Allocate and account 'n_elements' objects of type 'type'.

Use this macro to allocate memory within InnoDB instead of 'new[]' and instead of UT_NEW_ARRAY() when it is not feasible to create a dedicated key.

Parameters
[in]typetype of objects being created
[in]n_elementsnumber of objects to create
Returns
pointer to the first allocated object or NULL

◆ UT_NEW_NOKEY

#define UT_NEW_NOKEY (   expr)    UT_NEW(expr, PSI_NOT_INSTRUMENTED)

Allocate, trace the allocation and construct an object.

Use this macro instead of 'new' within InnoDB and instead of UT_NEW() when creating a dedicated memory key is not feasible. For example: instead of Foo* f = new Foo(args); use: Foo* f = UT_NEW_NOKEY(Foo(args)); Upon failure to allocate the memory, this macro may return NULL. It will not throw exceptions. After successful allocation the returned pointer must be passed to UT_DELETE() when no longer needed.

Parameters
[in]exprany expression that could follow "new"
Returns
pointer to the created object or NULL

◆ UT_NEW_THIS_FILE_PSI_INDEX

#define UT_NEW_THIS_FILE_PSI_INDEX   (force_constexpr<ut_new_get_key_by_file(MY_BASENAME)>::value)

◆ UT_NEW_THIS_FILE_PSI_KEY

#define UT_NEW_THIS_FILE_PSI_KEY
Value:
PSI_memory_key auto_event_keys[n_auto]
#define PSI_NOT_INSTRUMENTED
Definition: validate_password_imp.cc:39
#define UT_NEW_THIS_FILE_PSI_INDEX
Definition: ut0new.h:511

◆ ut_realloc

#define ut_realloc (   ptr,
  n_bytes 
)
Value:
.reallocate(ptr, n_bytes, UT_NEW_THIS_FILE_PSI_KEY))
Allocator class for allocating memory from inside std::* containers.
Definition: ut0new.h:571
#define PSI_NOT_INSTRUMENTED
Definition: validate_password_imp.cc:39
#define UT_NEW_THIS_FILE_PSI_KEY
Definition: ut0new.h:514

◆ ut_zalloc

#define ut_zalloc (   n_bytes,
  key 
)
Value:
static_cast<void *>(ut_allocator<byte>(key).allocate( \
n_bytes, NULL, UT_NEW_THIS_FILE_PSI_KEY, true, false))
Allocator class for allocating memory from inside std::* containers.
Definition: ut0new.h:571
#define UT_NEW_THIS_FILE_PSI_KEY
Definition: ut0new.h:514
#define NULL
Definition: types.h:55
static const char * key
Definition: suite_stubs.c:14

◆ ut_zalloc_nokey

#define ut_zalloc_nokey (   n_bytes)
Value:
static_cast<void *>( \
ut_allocator<byte>(PSI_NOT_INSTRUMENTED) \
.allocate(n_bytes, NULL, UT_NEW_THIS_FILE_PSI_KEY, true, false))
#define PSI_NOT_INSTRUMENTED
Definition: validate_password_imp.cc:39
#define UT_NEW_THIS_FILE_PSI_KEY
Definition: ut0new.h:514
#define NULL
Definition: types.h:55

◆ ut_zalloc_nokey_nofatal

#define ut_zalloc_nokey_nofatal (   n_bytes)
Value:
static_cast<void *>( \
ut_allocator<byte>(PSI_NOT_INSTRUMENTED) \
.set_oom_not_fatal() \
.allocate(n_bytes, NULL, UT_NEW_THIS_FILE_PSI_KEY, true, false))
#define PSI_NOT_INSTRUMENTED
Definition: validate_password_imp.cc:39
#define UT_NEW_THIS_FILE_PSI_KEY
Definition: ut0new.h:514
#define NULL
Definition: types.h:55

Function Documentation

◆ operator!=()

template<typename T >
bool operator!= ( const ut_allocator< T > &  lhs,
const ut_allocator< T > &  rhs 
)
inline

Compare two allocators of the same type.

◆ operator==()

template<typename T >
bool operator== ( const ut_allocator< T > &  lhs,
const ut_allocator< T > &  rhs 
)
inline

Compare two allocators of the same type.

As long as the type of A1 and A2 is the same, a memory allocated by A1 could be freed by A2 even if the pfs mem key is different.

◆ ut_align()

UNIV_INLINE void* ut_align ( const void *  ptr,
ulint  align_no 
)

This is a forward declaration, which is because of the circular dependency between ut0new.h and ut0byte.h (going through univ.i and sync0types.h).

I've managed to observe problem when building MEB and this helps then.

◆ ut_delete()

template<typename T >
void ut_delete ( T *  ptr)
inline

Destroy and account object created by UT_NEW() or UT_NEW_NOKEY().

Parameters
[in,out]ptrpointer to the object

◆ ut_delete_array()

template<typename T >
void ut_delete_array ( T *  ptr)
inline

Destroy and account objects created by UT_NEW_ARRAY() or UT_NEW_ARRAY_NOKEY().

Parameters
[in,out]ptrpointer to the first object in the array

◆ ut_len_without_extension()

constexpr size_t ut_len_without_extension ( const char *  file)

Find the length of the filename without its file extension.

Parameters
[in]filefilename, with extension but without directory
Returns
length, in bytes

◆ ut_new_boot()

void ut_new_boot ( )

Setup the internal objects needed for UT_NEW() to operate.

This must be called before the first call to UT_NEW().

◆ ut_new_boot_safe()

void ut_new_boot_safe ( )

Setup the internal objects needed for UT_NEW() to operate.

This must be called before the first call to UT_NEW(). This version of function might be called several times and it will simply skip all calls except the first one, during which the initialization will happen.

◆ ut_new_get_key_by_base_file()

constexpr int ut_new_get_key_by_base_file ( const char *  file,
size_t  len 
)

Retrieve a memory key (registered with PFS), given the file name of the caller.

Parameters
[in]fileportion of the filename - basename, with extension
[in]lenlength of the filename to check for
Returns
index to registered memory key or -1 if not found

◆ ut_new_get_key_by_file()

constexpr int ut_new_get_key_by_file ( const char *  file)

Retrieve a memory key (registered with PFS), given the file name of the caller.

Parameters
[in]fileportion of the filename - basename, with extension
Returns
index to memory key or -1 if not found

◆ ut_string_begins_with()

constexpr bool ut_string_begins_with ( const char *  a,
const char *  b,
size_t  b_len 
)

gcc 5 fails to evalutate costexprs at compile time.

Compute whether a string begins with a given prefix, compile-time.

Parameters
[in]afirst string, taken to be zero-terminated
[in]bsecond string (prefix to search for)
[in]b_lenlength in bytes of second string
Returns
whether b is a prefix of a

Variable Documentation

◆ alloc_max_retries

const size_t alloc_max_retries

Maximum number of retries to allocate memory.

◆ auto_event_keys

PSI_memory_key auto_event_keys[n_auto]

◆ auto_event_names

constexpr const char* auto_event_names[]
static

List of filenames that allocate memory and are instrumented via PFS.

◆ mem_key_ahi

PSI_memory_key mem_key_ahi

Keys for registering allocations with performance schema.

Pointers to these variables are supplied to PFS code via the pfs_info[] array and the PFS code initializes them via PSI_MEMORY_CALL(register_memory)(). mem_key_other and mem_key_std are special in the following way. If the caller has not provided a key and the file name of the caller is unknown, then mem_key_std will be used. This happens only when called from within std::* containers. If the caller has not provided a key and the file name of the caller is known, but is not amongst the predefined names (see ut_new_boot()) then mem_key_other will be used. Generally this should not happen and if it happens then that means that the list of predefined names must be extended. Keep this list alphabetically sorted.

Keep this list alphabetically sorted.

◆ mem_key_archive

PSI_memory_key mem_key_archive

◆ mem_key_buf_buf_pool

PSI_memory_key mem_key_buf_buf_pool

◆ mem_key_buf_stat_per_index_t

PSI_memory_key mem_key_buf_stat_per_index_t

◆ mem_key_clone

PSI_memory_key mem_key_clone

Memory key for clone.

◆ mem_key_dict_stats_bg_recalc_pool_t

PSI_memory_key mem_key_dict_stats_bg_recalc_pool_t

◆ mem_key_dict_stats_index_map_t

PSI_memory_key mem_key_dict_stats_index_map_t

◆ mem_key_dict_stats_n_diff_on_level

PSI_memory_key mem_key_dict_stats_n_diff_on_level

◆ mem_key_other

PSI_memory_key mem_key_other

◆ mem_key_partitioning

PSI_memory_key mem_key_partitioning

◆ mem_key_redo_log_archive_queue_element

PSI_memory_key mem_key_redo_log_archive_queue_element

◆ mem_key_row_log_buf

PSI_memory_key mem_key_row_log_buf

◆ mem_key_row_merge_sort

PSI_memory_key mem_key_row_merge_sort

◆ mem_key_std

PSI_memory_key mem_key_std

◆ mem_key_trx_sys_t_rw_trx_ids

PSI_memory_key mem_key_trx_sys_t_rw_trx_ids

◆ mem_key_undo_spaces

PSI_memory_key mem_key_undo_spaces

◆ mem_key_ut_lock_free_hash_t

PSI_memory_key mem_key_ut_lock_free_hash_t

◆ n_auto

constexpr size_t n_auto = UT_ARR_SIZE(auto_event_names)
static

◆ pfs_info_auto

PSI_memory_info pfs_info_auto[n_auto]