bulk_load_service.h File Reference

This service provides interface for loading data in bulk from CSV files. More...

#include "my_rapidjson_size_t.h"
#include <mysql/components/service.h>
#include <rapidjson/document.h>
#include <rapidjson/error/en.h>
#include <rapidjson/rapidjson.h>
#include <rapidjson/stringbuffer.h>
#include <rapidjson/writer.h>
#include <algorithm>
#include <cctype>
#include <cstdlib>
#include <optional>
#include <sstream>
#include <string>
#include <unordered_set>
#include "m_string.h"
#include "my_thread_local.h"

Go to the source code of this file.

Classes
struct	Bulk_load_file_info
struct	s_mysql_bulk_load_driver
	Bulk load driver service. More...

Typedefs
using	Bulk_loader = void
typedef struct s_mysql_bulk_load_driver	mysql_service_bulk_load_driver_t
	Bulk load driver service. More...

Enumerations
enum class	Bulk_source { LOCAL , OCI , S3 }
	Bulk loader source. More...
enum class	Bulk_compression_algorithm { NONE , ZSTD }
	Bulk data compression algorithm. More...
enum class	Bulk_string {   SCHEMA_NAME , TABLE_NAME , FILE_PREFIX , FILE_SUFFIX ,   COLUMN_TERM , ROW_TERM , APPENDTOLASTPREFIX }
	Bulk loader string attributes. More...
enum class	Bulk_condition { ORDERED_DATA , OPTIONAL_ENCLOSE , DRYRUN }
	Bulk loader boolean attributes. More...
enum class	Bulk_size {   COUNT_FILES , COUNT_ROW_SKIP , COUNT_COLUMNS , CONCURRENCY ,   MEMORY , START_INDEX }
	Bulk loader size attributes. More...
enum class	Bulk_char { ESCAPE_CHAR , ENCLOSE_CHAR }
	Bulk loader single byte attributes. More...

Functions
std::string	trim_left (const std::string &s)
std::ostream &	operator<< (std::ostream &out, const Bulk_load_file_info &obj)
static bool	is_json_object (const std::string &file_name_arg)
	Check whether the specified argument is a valid JSON object. More...
static bool	parse_input_arg (std::string &error, Bulk_load_file_info &info, const rapidjson::Document &doc)
	Validates whether the json argument matches the expected schema for bulk load, if it matches it fills out the Bulk_load_input structure, sets error and returns false otherwise. More...

Detailed Description

This service provides interface for loading data in bulk from CSV files.

Typedef Documentation

Bulk_loader

using Bulk_loader = void

mysql_service_bulk_load_driver_t

typedef struct s_mysql_bulk_load_driver mysql_service_bulk_load_driver_t

Bulk load driver service.

Enumeration Type Documentation

Bulk_char

enum class Bulk_char

strong

Bulk loader single byte attributes.

Enumerator
ESCAPE_CHAR	Escape character.
ENCLOSE_CHAR	Column enclosing character.

Bulk_compression_algorithm

enum class Bulk_compression_algorithm

strong

Bulk data compression algorithm.

Enumerator
NONE
ZSTD

Bulk_condition

enum class Bulk_condition

strong

Bulk loader boolean attributes.

Enumerator
ORDERED_DATA	The algorithm used is different based on whether the data is in sorted primary key order. This option tells whether to expect sorted input.
OPTIONAL_ENCLOSE	If enclosing is optional.
DRYRUN	If true, the current execution is only a dry run. No need to load data into the table.

Bulk_size

enum class Bulk_size

strong

Bulk loader size attributes.

Enumerator
COUNT_FILES	Number of input files.
COUNT_ROW_SKIP	Number of rows to skip.
COUNT_COLUMNS	Number of columns in the table.
CONCURRENCY	Number of concurrent loaders to use,.
MEMORY	Total memory size to use for LOAD in bytes.
START_INDEX	Index of the first file.

Bulk_source

enum class Bulk_source

strong

Bulk loader source.

Enumerator
LOCAL	Local file system.
OCI	OCI object store.
S3	Amazon S3.

Bulk_string

enum class Bulk_string

strong

Bulk loader string attributes.

Enumerator
SCHEMA_NAME	Schema name.
TABLE_NAME
FILE_PREFIX
FILE_SUFFIX
COLUMN_TERM	Column terminator.
ROW_TERM	Row terminator.
APPENDTOLASTPREFIX	String to append to last file prefix.

Function Documentation

is_json_object()

static bool is_json_object ( const std::string &  file_name_arg )

inlinestatic

Check whether the specified argument is a valid JSON object.

Used to check whether the user specified JSON or a regular filename as the LOAD location argument.

Parameters

[in]

file_name_arg

filename argument provided by the user.

Returns

true if the arg is a JSON object.

operator<<()

std::ostream & operator<< ( std::ostream &  out, const Bulk_load_file_info &  obj  )

inline

parse_input_arg()

static bool parse_input_arg ( std::string &  error, Bulk_load_file_info &  info, const rapidjson::Document &  doc  )

inlinestatic

Validates whether the json argument matches the expected schema for bulk load, if it matches it fills out the Bulk_load_input structure, sets error and returns false otherwise.

Parameters

[out]	error	contains the appropriate error message.
[out]	info	parsed structure of file information (containing prefix and optional suffix)
[in]	doc	rapidjson document

Returns

false if configuration contains unknown or unsupported values.

trim_left()

std::string trim_left ( const std::string &  s )

inline