WL#3859: Plug-in Service API

Sometimes plugins need to invoke some functionality implemented in the server.
They cannot simply call a function by its name - it will not work on Windows (a
limitation of dynamic linker), and it bypasses versioning checks in the plugin
API (which could easily cause a crash).

We need to have an API that:

1. Allow plug-ins to call code inside the server.

2. portable, works on Windows

3. safe, a plugin built expecting a certain interface in the server should not
   misbehave if server cannot provide this interface (simple example - function
   prototype in the server changes)

4. easy to use, plugin authors should spend as little time as possible
   implementing its requirements

5. flexible, a plugin may support several interfaces and could decide to chose
   which one to use depending on what the server can provide

6. plugins should be able to expose their functionality to another  plugins 
(and
   the server?) in a similar fashion

This WL will provide all that.

REFERENCES
==========

[1] *Plug-in system code for Windows.*
    Available at http://www.codeproject.com/dll/PluginSystem.asp

[2] WL#3653 Loadable Engines on Windows

The API
^^^^^^^

We'll split all exportable server functionality into "services".

Services will be versioned, so that we could track API changes.

Every service will be represented as a C structure of function callbacks:

  struct File_IO_Service {
    int (*open)(...);     // a pointer to my_open
    int (*close)(...);    // a pointer to my_close
    ...
  } File_IO_Service;

It has only fields that plugins need to see. We'll also have an array
enumerating all services (a plugin loader needs it), something like

  Array_of_Services= {
    { STRING_WITH_LEN("File_IO_Service"), 0x0109, &File_IO_Service },
    ...
  };

which, besides the pointer to the service structure, also contains the name and
the current version of the service.

The versioning logic is described in WL#2761

A plugin will request a service as

   // request a File_IO_Service version 1.9 or compatible
   struct File_IO_Service *File_IO_Service=
     get_service("File_IO_Service", 0x0109);

  if (!File_IO_Service)
    return -1;

   int file=File_IO_Service->open("data.txt", O_RDONLY);

An alternative way to request a service, more user friedly but less flexible
would be (rewriting the above example):

   int file=my_open("data.txt", O_RDONLY);

This will work as:

- in one of the headers we'll have
  #define my_open(A,B) (File_IO_Service->open(A,B))
  extern struct File_IO_Service *File_IO_Service;
- we'll have a static library libplugin.a that plugins will link with.
  I'll define a symbol File_IO_Service
- But! it'll define it as
  intptr File_IO_Service= 0x0109;
  that is, an integer, holding the version of the service.
- on load, sql_plugin.cc will check all symbols with well-defined names
  (like File_IO_Service) from the plugin, get their values - requested
  service versions - ensure they're compatible, and set their values to
  point to the real plugin structures
- if sql_plugin.cc will find the versions incompatible, it'll refuse to
  load the plugin, as usual (so this scheme is less flexible, plugin has no
  chance to continue to work with a crippled functionality even if it can
  technically do that).

in this scheme, simply using a function adds a dependency on a service version,
names are resolved automatically on load, versions are checked to guarantee
compatibility. Unused services don't add a dependency (make sure that every
intptr version variable in the libplugin.a is in a separate .c file).

get_service() from the first example comes from the Service_Registration_Service
and is resolved automatically using libplugin.a scheme.

Service_Registration_Service may also provide release_service() so that plugin
could explicitly release a service it was using (but all used services will be
automatically released anyway when a plugin is unloaded - or at least those that
were auto-requested), and register_service() and unregister_service() to allow
plugins export services that other plugins can use.

Just as plugins themselves, services are reference-counted. It could be skipped
for static built-in services, of course (but is vital for services provided by
plugins). Alternatively we can simply increment plugin's reference counter, and
don't count references to services at all.

We need to ensure services aren't changed inadvertently.
At least, abi-check should control all the plugin API, including service
declarations.

Creating Services: Examples and Guidelines
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Creating a Service is a development task, it should follow the usual procedure -
WL first, architecture approval, coding review. This WL task is only about
creating a framework.

We will have both C services and C++ services. In either case we should strive
not to export more than necessary, and to hide implementation details as much as
possible (using wrapper functions/methods if needed). In C++ the best option is
pure virtual abstract classes.

To reduce the risk of name collisions, especially when plugins will provide
their own services, we adopt a naming scheme where a service name looks like
"server.mysys.mem_root" (or, for example, "core.utility.allocator.mem_root")

The rule is - if something is exported, is part of the API, we have to keep it
and support it. Hiding implementation details allows us to change the internals
of a feature (optimize, fix a bug, whatever) without affecting plugins that rely
on it.

TABLE, TABLE_SHARE, Field, Item, and some others are not ready to be exported
yet, they need to be refactored. JimS has a proposal how to do it, but it is not
in Worklog yet.

Simpler examples are:

. memory allocation              (WL#3961)
. IO_CACHE                       (WL#3961)
. logging and error reporting    (WL#2940)
. hash
. tree
. character sets and collations,
. file I/O)
... etc

see WL's above for examples of C services. Below is an example of a C++ service:

 class Tree {
   public:
   typedef enum { free_init, free_free, free_end } FREE;
   typedef enum { left_root_right, right_root_left } WALK;

   typedef int (*qsort_cmp2)(void*, const void *,const void *);
   typedef void (*element_free)(void*, FREE, void *);
   typedef int (*walk_action)(void *,uint,void *);

   virtual void  reset() = 0;
   virtual void *insert(void *key, int key_size, void* custom_arg) = 0;
   virtual int   delete(void *key, uint key_size, void *custom_arg) = 0;
   virtual void *search(void *key, void *custom_arg) = 0;
   virtual int   walk(walk_action action, void *argument, WALK visit) = 0;
   virtual void ~Tree() { };
 };

 struct Tree_Service {
   Tree *(*new)(long default_alloc_size, long memory_limit,
                int size, Tree::qsort_cmp2 compare, int with_delete,
	        Tree::element_free free_element, void *custom_arg);
   void (*delete)(Tree *);
 } Tree_Service;