Introduction
The loader class is responsible for managing the life-cycle of plugins in the harness. Each plugin goes through seven steps in the life-cycle, of which steps #2, #3, #5 and #6 are optional:
- Loading
- Initialization
- Starting
- Running
- Stopping
- Deinitialization
- Unloading
Overview of Life-cycle Steps
1. Loading
When loading, the plugin is loaded using the dynamic library support available on the operating system. Symbols are evaluated lazily (for example, the RTLD_LAZY flag is used for dlopen) to allow plugins to be loaded in any order. The symbols that are exported by the plugin are made available to all other plugins loaded (flag RTLD_GLOBAL to dlopen).
As part of the loading procedure, the plugin structure (see Plugin class) is fetched from the module and used for the four optional steps below.
2. Initialization
After all the plugins are successfully loaded, each plugin is given a chance to perform initialization. This step is only executed if the plugin structure defines an init function. Note that it is guaranteed that the init function of a plugin is called after the init function of all plugins it requires have been called. The list of these dependencies is specified via requires field of the Plugin struct.
- Note
- if some plugin
init() function fails, any plugin init() functions schedulled to run after will not run, and harness will proceed straight to deinitialization step, bypassing calling start() and stop() functions.
3. Starting
After all plugins have been successfully initialized, a thread is created for each plugin that has a non-NULL start field in the plugin structure. The threads are started in an arbitrary order, so you have to be careful about not assuming that, for example, other plugins required by the plugin have started their thread. If the plugin does not define a start function, no thread is created. There is a "running" flag associated with each such thread; this flag is set when the thread starts but before the start function is called. If necessary, the plugin can spawn more threads using standard C++11 thread calls, however, these threads should not call harness API functions.
4. Running
After starting all plugins (that needed to be started), the harness will enter the running step. This is the "normal" phase, where the application spends most of its lifetime (application and plugins service requests or do whatever it is they do). Harness will remain in this step until one of two things happen:
- shutdown signal is received by the harness
- one of the plugins exits with error
When one of these two events occurs, harness progresses to the next step.
5. Stopping
In this step, harness "tells" plugins running start() to exit this function by clearing the "running" flag. It also invokes stop() function for all plugins that provided it. It then waits for all running plugin threads to exit.
- Note
- under certain circumstances,
stop() may overlap execution with start(), or even be called before start().
6. Deinitialization
After all threads have stopped, regardless of whether they stopped with an error or not, the plugins are deinitialized in reverse order of initialization by calling the function in the deinit field of the Plugin structure. Regardless of whether the deinit() functions return an error or not, all plugins schedulled for deinitialisation will be deinitialized.
- Note
- for any
init() functions that failed, deinit() functions will not run.
-
plugins may have a
deinit() function despite not having a corresponding init(). In such cases, the missing init() is treated as if it existed and ran successfully.
7. Unloading
After a plugin has deinitialized, it can be unloaded. It is guaranteed that no module is unloaded before it has been deinitialized.
- Note
- This step is currently unimplemented - meaning, it does nothing. The plugins will remain loaded in memory until the process shuts down. This makes no practical difference on application behavior at present, but might be needed if Harness gained ability to reload plugins in the future.
Behavior Diagrams
Previous section described quickly each step of the life-cycle process. In this section, two flow charts are presented which show the operation of all seven steps. First shows a high-level overview, and the second shows all 7 life-cycle steps in more detail. Discussion of details follows in the following sections.
Some points to keep in mind while viewing the diagrams:
- diagrams describe code behavior rather than implementation. So for example:
- pseudocode does not directly correspond 1:1 to real code. However, it behaves exactly like the real code.
- seven life-cycle functions shown are actual functions (
Loader's methods, to be more precise)
- load_all(), init_all(), start_all(), main_loop(), stop_all(), deinit_all() are implemented functions (first 6 steps of life-cycle)
- unload_all() is the 7th step of life-cycle, but it's currently unimplemented
- when plugin functions exit with error, they do so by calling set_error() before exiting
- some things are not shown to keep the diagram simple:
- first error returned by any of the 7 life-cycle functions is saved and passed at the end of life-cycle flow to the calling code
Overview
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
\ \
\ START \
\ | \
\ | \
\ | \
\ V \
\ [load_all()] \
\ | \
\ V \
\ <LOAD_OK?> \
\ | | \
\ +---N Y \
\ | | \
\ | v \
\ | [init_all()] \
\ | | \
\ | v \
\ | <INIT_OK?> ( each plugin runs ) \
\ | | | (in a separate thread) \
\ | N Y \
\ | | | [plugin[1]->start()] \
\ | | v start plugin threads [plugin[2]->start()] \
\ | | [start_all()] - - - - - - - - - - - - - - - - ->[ .. .. ] \
\ | | | [ .. .. ] \
\ | | | + - - - - - - - - - - - - - - - - - - - - -[plugin[n]->start()] \
\ | | | notification when each ^ \
\ | | | | thread terminates \
\ | | | | \
\ | | | | stop plugin \
\ | | | | threads \
\ | | | | \
\ | | | | \
\ | | v v \
\ | | [main_loop()]= call ==>[stop_all()] - - - - - - - - - - - + \
\ | | | \
\ | | | \ \
\ | *<--+ \ \
\ | | \__ waits for all plugin \
\ | v threads to terminate \
\ | [deinit_all()] \
\ | | \
\ | v \
\ +-->* \
\ | \
\ v \
\ [unload_all()] \
\ | \
\ | \ \
\ | \ \
\ v \__ currently not implemented \
\ END \
\ \
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
Detailed View
START
|
|
v
\\\\ load_all() \\\\\\\\\\\\\\\\\\\\\\
\ \
\ LOAD_OK = true \
\ foreach plugin: \
\ load plugin \
\ if (exit_status != ok): \
\ LOAD_OK = false \
\ break loop \
\ \
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
|
|
v
<LOAD_OK?>
| |
Y N----> unload_all() (see further down)
|
|
\\\\\\\\\\\\\\|\\\ init_all() \\\\\\\\\\\\\\\\\
\ | \
\ v \
\ [INIT_OK = true, i = 1] \
\ | \
\ v \
\ +----><plugin[i] exists?> \
\ | | | \
\ [i++] Y N---------------------+ \
\ ^ | | \
\ | | | \
\ | v | \
\ | <plugin[i] has init()?> | \
\ | | | | \
\ | N Y---+ | \
\ | | | | \
\ | | | | \
\ | | v | \
\ | | [plugin[i]->init()] | \
\ | | | | \
\ | | | | \
\ | | | | \
\ | | | | \
\ | | v | \
\ | | <exit ok?> | \
\ | v | | | \
\ +-------*<------Y N | \
\ | | \
\ | | \
\ v | \
\ [INIT_OK = false] | \
\ | | \
\ v | \
\ *<----------------+ \
\ | \
\ v \
\ [LAST_PLUGIN = i-1] \
\ | \
\\\\\\\\\\\\\\\\\\\\\\|\\\\\\\\\\\\\\\\\\\\\\\\
|
|
v
<INIT_OK?>
| |
Y N----> deinit_all() (see further down)
|
|
v
\\\\ start_all() \\\\\\\\\\\\\\\\\\\\\\\\\
\ \
\ for i = 1 to LAST_PLUGIN: \
\ if plugin[i] has start(): \ start start() in new thread
\ new thread(plugin[i]->start()) - - - - - - - - - - - - - - - - +
\ \
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ |
|
| |
+-----------------+
| |
\\\\|\\\ main_loop() \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ \\\\\\\\
| | \
v \
+-->* | \
| | \
| v | \
| <any plugin threads running?> \
| | | | \
| N Y---+ \
| | | | \
| | <shutdown signal received && stop_all() not called yet?> \
| | | | | \
| | N Y \
| | | == call ==>[stop_all()]- - - - - - - - - - - - - + | \
| | | | tell (each) start() to exit \
| | *<--+ | | \
| | | \
| | | v v \
| | | [plugin[1]->start()] \
| | v (one) plugin thread exits [plugin[2]->start()] \
| | [wait for (any)]<- - - - - - - - - - - - - - - -[ .. .. ] \
| | [ thread exit ] [ .. .. ] \
| | | [plugin[n]->start()] \
| | | ^ \
| | | \
| | v | \
| | <thread exit ok?> \
| | | | | \
| | Y N---+ \
| | | | | \
| | | v \
| | | <stop_all() called already?> | \
| | v | | \
| | *<------Y N tell (each) \
| | | = call ==+ start() to exit \
| | | | | | \
| | v | | \
+---|-------*<----------+ *==>[stop_all()]- - - - - - - - + \
| | \
| | | \
v | | \
<stop_all() called already?> | | \
| | | | \
Y N | | \
| == call =================+ | \
| | | \
*---+ | \
| | \
\\\\|\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\|\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
| |
| |
v |
*<---- init_all() (if !INIT_OK) |
| |
| |
v |
\\\\ deinit_all() \\\\\\\\\\\\\\\\\\\\ |
\ \ |
\ for i = LAST_PLUGIN to 1: \ |
\ if plugin[i] has deinit(): \ |
\ plugin[i]->deinit() \ |
\ if (exit_status != ok): \ |
\ # ignore error \ |
\ \ |
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ |
| |
| |
v |
*<---- load_all() (if !LOAD_OK) |
| |
v |
\\\\ unload_all() \\\\\\\\\\\\\\\\\\\\ |
\ \ |
\ no-op (currently unimplemented) \ |
\ \ |
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ |
| |
| |
v /
END /
/
/
/ ( each plugin runs )
/ (in a separate thread)
\\\\ stop_all() \\\\\\\\\\\\\\\\\\\\\\ run_flag == false
\ \ tells start() [plugin[1]->start()]
\ for i = 1 to LAST_PLUGIN: \ to exit [plugin[2]->start()]
\ run_flag[i] = false - - - - - - - - - - - - - - - - ->[ .. .. ]
\ if plugin[i] has stop(): \ [ .. .. ]
\ plugin[i]->stop() \ [ .. .. ]
\ if (exit_status != ok): \ [plugin[n]->start()]
\ # ignore error \
\ \
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Discussion
Persistence (definition)
Before continuing, we need to define the word "persist", used later on. When we say "persist", we'll mean the opposite of "exit". So when we say a function or a thread persists, it means that it's not returning/exiting, but instead is running some forever-loop or is blocked on something, etc. What it's doing exactly doesn't matter, what matters, is that it hasn't terminated but "lives on" (in case of a function, to "persist" means the same as to "block", but for threads that might sound confusing, this is why we need a new word). So when we call start() in a new thread, it will either run and keep running (thread will persist), or it will run briefly and return (thread will exit). In short, "to persist" means the opposite of "to finish running and exit".
Plugin API functions
Each plugin can define none, any or all of the following 4 callbacks. They're function pointers, that can be null if not implemented. They're typically called in the order as listed below (under certain circumstances, stop() may overlap execution with start(), or even be called before start()).
- init() – called inside of main thread
- start() – main thread creates a new thread, then calls this
- stop() – called inside of main thread
- deinit() – called inside of main thread
Starting and Stopping: Start()
It is typical to implement start function in such a way that it will "persist" (i.e. it will run some forever-loop processing requests rather than exit briefly after being called). In such case, Harness must have a way to terminate it during shutdown operation.
For this purpose, Harness exposes a boolean "running" flag to each plugin, which serves as means to communicate the need to shutdown; it is read by is_running() function. This function should be routinely polled by plugin's start() function to determine if it should shut down, and once it returns false, plugin start() should terminate as soon as possible. Failure to terminate will block Harness from progressing further in its shutdown procedure, resulting in application "hanging" during shutdown. Typically, start() would be implemented more-or-less like so:
void start()
{
run-once code
while (is_running())
{
forever-loop code }
clean-up code }
There is also an alternative blocking function available, wait_for_stop(), should that be better suited for the particular plugin design. Instead of quickly returning a boolean flag, it will block (with an optional timeout) until Harness flags to shut down this plugin. It is an efficient functional equivalent of:
while (is_running())
{
sleep a little or break on timeout }
When entering shutdown phase, Harness will notify all plugins to shut down via mechanisms described above. It is also permitted for plugins to exit on their own, whether due to error or intended behavior, without consulting this "running" flag. Polling the "running" flag is only needed when start() "persists" and does not normally exit until told to do so.
Also, in some designs, start() function might find it convenient to be able to set the "running" flag to false, in order to trigger its own shutdown in another piece of code. For such cases, clear_running() function is provided, which will do exactly that.
IMPORTANT! Please note that all 3 functions described above (is_running(), wait_for_stop() and clear_running()) can only be called from a thread running start() function. If start() spawns more theads, these functions CANNOT be called from them. These functions also cannot be called from the other three plugin functions (init(), stop() and deinit()).
Starting and Stopping: Stop()
During shutdown, or after plugin start() function exits (whichever comes first), plugin's stop() function will be called, if defined.
IMPORTANT: start() function runs in a different thread than stop() function. By the time stop() runs, depending on the circumstances, start() thread may or may not exist.
IMPORTANT: stop() will always be called during shutdown, regardless of whether start() exited with error, exited successfully or is still running. stop() must be able to deal with all 3 scenarios. The rationale for this design decision is given Error Handling section.
Persistence in Plugin Functions
While start() may persist, the other three functions (init(), stop() and deinit()) must obviously not persist, since they run in the main thread. Any blocking behavior exhibited in these functions (caused by a bug or otherwise) will cause the entire application to hang, as will start() that does not poll and/or honor is_running() flag.
Returning Success/Failure from Plugin Function
Harness expects all four plugin functions (init(),start(),stop()and deinit()`) to notify it in case of an error. This is done via function:
set_error(PluginFuncEnv* env, ErrorType error, const char* format, ...);
Calling this function flags that the function has failed, and passes the error type and string back to Harness. The converse is also true: not calling this function prior to exiting the function implies success. This distinction is important, because Harness may take certain actions based on the status returned by each function.
IMPORTANT! Throwing exceptions from these functions is not supported. If your plugin uses exceptions internally, that is fine, but please ensure they are handled before reaching the Harness-Plugin boundary.
Threading Concerns
For each plugin (independent of other plugins): Of the 4 plugin functions, init() runs first. It is guaranteed that it will exit before start() and stop() are called. start() and stop() can be called in parallel to each other, in any order, with their lifetimes possibly overlapping. They are guaranteed to both have exited before deinit() is called.
If any of the 4 plugin functions spawn any additional threads, Harness makes no provisions for interacting with them in any way: calling Harness functions from them is not supported in particular; also such threads should exit before their parent function finishes running.
Error Handling
NOTE: WL#9558 HLD version of this section additionally discusses design, rationale for the approach chosen, etc; look there if interested.
When plugin functions encounter an error, they are expected to signal it via set_error(). What happens next, depends on the case, but in all four cases the error will be logged automatically by the harness. Also, the first error passed from any plugin will be saved until the end of life-cycle processing, then passed down to the code calling the harness. This will allow the application code to deal with it accordingly (probably do some of its own cleaning and shut down, but that's up to the application). In general, the first error from init() or start() will cause the harness to initiate shut down procedure, while the errors from stop() and deinit() will be ignored completely (except of course for being logged and possibly saved for passing on at the end). Actions taken for each plugin function are as follows:
init() fails:
- skip init() for remaining plugins
- don't run any start() and stop() (proceed directly to deinitialisation)
- run deinit() only for plugins initialiased so far (excluding the failing one), in reverse order of initialisation, and exit
- when init() is not provided (is null), it counts as if it ran, if it would have run before the failing plugin (according to topological order)
start() fails:
- proceed to stop all plugins, then deinit() all in reverse order of initialisation and exit. Please note that ALL plugins will be flagged to stop and have their stop() function called (not just the ones that succeeded in starting - plugin's stop() must be able to deal with such a situation)
stop() or deinit() fails:
- log error and ignore, proceed as if it didn't happen
1.9.2