WL#4739: Physical Structure of Server

In order to simplify working with the server code, both as a developer and when
packaging distributions, we want to have a physical structure of the code that
makes the code easy to work with.

We are also aiming at a long-term solution that will allow the server code base
to grow significantly, without making the code unmaintainable.

In order to support this, we need a structure that supports:

- Clear and simple conventions for creating structure for the code

- Easy adding and removing of features to allow:

  - Features to be added late with a minimal risk of ripple effects into
    unrelated parts of the code. This can be introduced due to merges causing
    unintended code changes, as well as logical dependencies that are not clear.

  - It shall be easy to remove features, should it be necessary for some reason
   (that might not strictly be technical).

- Having a structure that allows the creation of various distribution packages
  from the same source, such as:

  - Client development distributions for application programmers

  - Storage engine development distributions for storage engine writers

  - Plug-in development distributions for plug-in writers (whom may or may not
    be storage engine developers)

- Having a structure that support working with the code using scripts to
  perform common tasks, like building special distributions, release testing,
  and packaging.


Working practice
================

There are some working practice that we need to support in this structure. These
practices are central to how we work with the code and not supporting them will
introduce severe problems for developers.

- Bug fixes and features is introduced as a sequence of patches, where each
  patch is a change to one or more files.

- A single patch should not cause a build failure and the server should
  still pass all tests.  If a bug fix or feature requires several
  patches, each patch should still leave the server in a stable state
  in the sense that it should still build and still pass all tests.

- A patch should not require unwarranted changes in other package. We
  should discourage practice that may require a developer to make
  changes in other packages than the one that he/she is working on.

  Forcing a developer to make changes in code that he/she is not familiar
  with, however small the changes are, increases the risk of introducing
  bugs and may go against design principles originally intended for a
  component or package.

- A patch is normally targeted for a single package only: features
  affecting several packages should be split into separate patches,
  committed in the right order, and preferably pushed together (bu
  this is not a requirement).


Notes
=====

- This worklog needs to be split up into several worklogs, at least:

  - one for the actual design (this one),

  - one for implementing the build frame (WL#4875)

  - one for fixing the current include file header mess (WL#4877)


Continuing work
===============

- In order to not stall the change of the structure for too long, it is
  necessary to set a bar for when the code should be changed. If that is not
  done, we will have to maintain two structures in parallel, which not offer
  any improvements to the development practice and instead solidify the current
  situation.

Open Issues
===========

- What names shall we use for the packages? We already have storage/ and
  server/ and client/ (which already exists) have been suggested.

Resolved issues
===============

- Shall each package have a unique prefix for the files? Also consider the
  exported header files.

  The reasons for having different prefixes for header files is to be able to
  separate header files with same names in different packages when including
  them. However, by using the package directory name as prefix, a header file
  prefix is not needed. It would be either:

     #include "pkg_table.h"

  or

     #include "pkg/table.h"

  The reason for using prefixes for source files would be that linkers have
  problems distinguishing between files with the same name, but some tests
  indicate that is not the case on some common platforms (Linux and Solaris).

  In short, there seems to be no good reason to use file prefixes together
  with a package structure.

- Shall a dynamically loadable module be a separate package or not?

  There might be reasons to why a loadable component may consist of several
  packages, so we should not require that each loadable component is a package.

Decisions
=========

2009-02-26: We agreed on going for approach 2 when handling header files.
            The basis was later questioned and clarification of the document
            was asked for.

2009-05-27: It was agreed that we should not  impose a structure on the
            packages from the build system and represent meta-data for a
            package separately(typically as a manifest or configuration file).
            Structure might still be mandated by coding styles and/or practical
            issues.


High-level structure
====================

We envision that the system consists of a number of *packages* that together
make up the code of the system. In order to build the server, and associated
components, we have a *build frame* (or just *frame*) that is used to manage
and, especially, build the system.

In order to support the easy addition and removal of features, we assume that
each feature is contained in a separate package (see below) and a minimum of
changes shall be required (preferably none) to code outside this package to
introduce the feature. To support this convention, the build frame has to be
independent on the number and type of packages that are available, and use
generic methods for deciding what packages are to be included in the build. 
This in turn requires the packages to provide the necessary information so that
the build frame can do its job.


Components
==========

A component consist of a set of header files and a set of associated
C/C++ files. The component is the smallest unit of the physical
design.

Typically, each component consists of a header file and a C/C++ file
with a common base name, for example "parser.h" and
"parser.cc". However, there are some cases where it makes sense to
have multiple header files for a component and cases when it makes
sense to have multiple source files.

- Using several header files can be used to present multiple
  interfaces into a single component.

- Using several source files could be mandated when the linker is
  file-based, and will just map symbols on file-level (loading/linking
  entire files, not individual functions).

In these cases, the files of each component shall have a common
prefix distinguishable from other components.


    =========== =================================================
    Component   Files
    =========== =================================================
    rpl_filter  rpl_filter.h rpl_filter.cc
    reg_main    reg_main_internal.h reg_main_public.h reg_main.cc
    =========== =================================================


Packages
========

Packages are collections of components that server a common purpose.  This
formulation is deliberately not exact since what actually makes sense to turn
into a package wary from case to case. However, the following issues should be
considered when deciding whether a candidate package makes sense as a package:

- Can the candidate package be released independently of the rest of the
  server? If not, i.e., changes to this package is likely to require changes
  to other packages, then maybe it should not be a package.

  Releasing here does *not* mean distributing the code in isolation, it means
  releasing, e.g., a new version of the package for use with the rest of the
  server.

- Is the candidate package very small, e.g., a single component? In this case
  it might make sense to group several such candidate packages with similar
  purpose into a single package.

  A typical example would be support for individual character sets, that does
  not make sense to place in a single package each, but is sensible as a
  package of "character set information". 


Package naming and structure
----------------------------

Each package is represented as a directory. The basic assumption is
that everything related to a package should be placed in the
directory. This includes, but is not limited to: header files, source
files, and unit tests.

Basic goals and assumptions are:

- Changes in the package internals should not inadvertently affect
  other packages that use the package

- It shall be possible to support third-party solutions as package in
  the package structure and shall not require re-organization to fit
  the package structure

The package directories will be placed in a *subsystem directory* alongside the
``sql/`` directory. Apart from that, all package directories are placed at the
same level. We are placing the packages in a new subsystem directory instead of
re-using ``sql/`` to be able to easily distinguish between "unorganized" and
"organized" code.

The following subsystem directories are proposed (some directories
already exists and almost have the basic structure proposed):

    ========== ==============================================
    Package    Purpose
    ========== ==============================================
    storage/   Storage engines
    server/    Server modules
    common/    Common utilities
    ========== ==============================================

There are some other directories that are being considered, such as
``mysys/``, and the above list will be extended as needed.

Package names shall be small letters only, with underscore to separate
individual words in the package name. Note that the package name may
not start with an underscore.  This choice of name is used to allow
the package name to be used both as a file name, a C/C++ symbol, and
as identifier in other tools (such as Doxygen).

Examples: registry, query_model


File names
~~~~~~~~~~

The choice and restrictions on file names is governed by the current coding
style.

The coding style takes into account operating system restrictions and
restrictions imposed by tools such as the compiler, linker, and other processing
tools. However, the physical structure itself does not impose any special
requirements on the file names.


Package namespace
~~~~~~~~~~~~~~~~~

All symbols of a package shall be placed in a single namespace, and the
namespace name shall be the same as the name of the package.  Since package
names as specified above are legal C/C++ symbol names, this will always be possible.


Package interfaces
------------------

For each package, there is a set of interfaces into the package. Each interface
is represented physically as a header file, meaning that each package have one
or more interfaces, but potentially have header files that are not package
interfaces.

The package owner shall be able to decide what header files are available for
users of the package. Initially we will not be able to do this for practical
reasons since it requires the build frame to support that. Instead, we will
assume that every header file in a package is available as a package interface.


Interface usage
~~~~~~~~~~~~~~~

In order to use an interface of a package, the header file is included using the
form:

    #include "package_name/interface.h"

The include path is set up by the build system so that this is possible. Note
that it is an error to include a file that is not a package interface or not a
header file of the same package.  Ideally, the build frame will not allow this,
but before that feature is implemented in the build frame, it will be possible
to do by mistake.

Header files of the same package are included using the form:

    #include "header.h"


Coding requirements
===================

This section outlines some basic rules that are meant to avoid common problems
associated with developing for a package structure as well as allowing
tool-support for checking and manipulating components and packages.  The need of
tool support is necessary to allow the system to grow, since manually resolving
issues will unnecessarily waste effort on maintaining inconsistencies.

The aim is to keep the rules to a bare minimum and specifically only consider
issues that (potentially can) traverse package boundaries or that cause problems
when maintaining or operating the build frame.  Issues on what is "good coding
style" is maintained separately and not part of this worklog.  This is done to
restrict the scope of the worklog and be able to close it.


Every header file should be self-sufficient
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For every header file "header.h", the following program shall compile without
errors:

    #include "header.h"

The reason is that when using a header file "header.h", it should be sufficient
to include "header.h" holding the functionality sought after. If it is necessary
to include any other files before "header.h" because there are definitions
required by "header.h", we have two problems:

1. It is hard to find out what dependencies are needed, and it will eventually
   lead to a trial and error approach that we are now seeing.

2. If the dependencies change, the file might include more files than
   necessary.


Every header file should have an include guard
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For a header file "header.h" in package "package", the include guard should have
the name PACKAGE_HEADER_INCLUDED. We choose to standardize the include guard so
that we can use external include guards if the need should arise. We omit the
extension from the name, since header files may have a number of different
extensions and we do not want to standardize on any one of them.

Existing include guards that are not violating the standard will not be changed
initially, but developers are encouraged to make the change if they are changing
the header file.


Source and header files should only include definitions it needs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For header files, it is critical to use forward declarations when that suffices.
The problems with including definitions that are not needed are twofold:

1. It introduces additional dependencies that are not necessary since
   definitions contain references to stuff that *it* needs. Note that
   dependencies may not only be on header files, but that unintended symbols
   may be pulled into the system.

2. It unnecessarily increase the compile time since it requires opening *at
   least* one more file (but usually several).  This problem is, however,
   secondary.


There shall be no convenience include files inside the server
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Convenience include files are include files whose only purpose is to bundle
other include files.

The reason to why we want to avoid this *inside* the server is that it
introduces unnecessary dependencies between packages (recall that dependencies
between components are represented as an #include directive). Should some
include file be added to the convenience include file because *one* component
needs it, *all* components that include this convenience include will be
affected. To avoid introducing unnecessary dependencies in this way we could:

1. Have a rule stating that convenience include may only hold includes that
   are used by *all* components including this convenience include. This adds
   an additional burden on developers wanting to add an include to the
   convenience include to locate each user of the convenience include and
   decide if they need it. Since the includers of the convenience include is
   not easily visible in the file, it means searching all packages.

   Furthermore: with this approach it can be expected that over time, the set
   of includes in the file will shrink and the purpose of having a convenience
   include will diminish.

2. Have a rule stating that convenience includes shall not be used, which
   requires all necessary include files to be mentioned. This is a minor
   problem from a development perspective and make dependencies between
   components explicit, hence clear.

We chose the latter.

However, convenience include files serve a purpose for maintaining interfaces
*into* the server is accepted (for example, to make it easier to work with the
client interface). For these files it is, however, critical that they are
convenience includes and not contain separate definitions.


No ``using`` directives (``using namespace``)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Placing using directives at namespace level in header files will force any file
that includes the header file to resolve symbols in a namespace they have no
control over. This can lead to subtle and hard to find bugs, and should
therefore not be used.

Placing ``using`` directives at namespace level in source files will inject all
symbols of that namespace (as ``pkga``) into another namespace (say ``pkgb``).
If changes are made to ``pkga``, they may conflict with definitions in ``pkgb``
and since a developer have to ensure the system builds for each patch, he would
be forced to make changes in ``pkgb`` despite the fact that the change itself is
localized to ``pkga``.


No ``using`` declarations before #include directives
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Placing a using declaration before including another file will place all the
symbols of the included file in a namespace and should not be used.


Entities declared in a component shall be defined in the component
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

An object or function declared in the header file of a component shall be
defined in the same component (usually in an implementation file). The reason
for this rule is that it shall be easy to know what components that need to be
linked in order to use the component. If some definition is in another file, it
will be hard to find and manage the right dependencies between components in the
system.


No gratuitous link-time dependencies between components
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Such dependencies can occur if a component, for example, declares an ``extern``
variable and do not include the proper header file. All dependencies shall be
explicit in the sense that they shall be visible in the file as an ``#include``
directive. This will allow dependencies to be detected and tracked automatically.


#######################################
Appendix A. Definitions and discussions
#######################################


Packages
========

Packages are collections of components organized as a cohesive unit (that is,
serve a common purpose).

Each package has one or more (exported) interfaces, which are represented by one
or more header files.

Defining files of a package
---------------------------

In order to define what files are part of a package, there are basically two
options: either supply a file for each package that lists the files of that
package, or put all the files of a package into a subdirectory.

The advantages of using the file system to define packages by putting all the
files of a package in a separate directory suggests that this approach should be
used.


Interfaces into a package
-------------------------

Each package have one or more interfaces represented physically as one or more
header files. The header files contains objects and definitions necessary to
interface with the package, so we have no restrictions on the structure (but do
have some recommendations for how to structure the interfaces for maintainability).

Each interface is normally defined with some strategic objective, i.e., it is
created for an intended set of users. We use *export target* to denote such a
set of users of an interface. Library functions usually have only one export
target, but many of our packages have several export targets such as "client
developers" who write application clients to the server, "storage engine
developers" who are creating a storage engine for the server, "plugin
developers" that are writing a plug-in for the server.

For each export target, we should ensure that the header files holding the
interfaces is defined in such a manner that only the parts needed by that export
target is included when that header file is included. Gratuitous definitions is
a problem since they might clash with the names defined by the user, and also
introduces an unnecessary dependency on parts of the server that the user does
not in reality depend on.

In short, interfaces into packages are represented as one or more header files,
and we have two basic methods to identify the interface files: by naming
convention (for example, placing the interface file in a separate directory) or
by using one or more configuration files that explicitly the interface interface
files.


Using naming conventions
~~~~~~~~~~~~~~~~~~~~~~~~

For this discussion, we assume that the exported interface header files are put
into the export/ directory.  However, the same arguments apply to other schemes
for using naming conventions.

Note that each header file might correspond to a source file placed in the main
package directory, like this:

      goobar/
        export/
          goo_interface.h
        goo_impl.cc
           .
           .
           .

The advantage of this approach is:

- Simplicity: normal file commands can be used to work with files. For
  example, to copy all files needed by a plugin-sdk could be as simple as:

   cp package/export/*.h /distro/include

The disadvantages are:

- Changing the status of a file from, e.g., internal to public requires
  moving the file and not all VCS systems support that well.

- Having multiple "export targets" (users of the interface) require separate
  directories. For example, a package could export an interface for
  third-party users and one towards the rest of the server packages.


Configuration file
~~~~~~~~~~~~~~~~~~

We somehow add extra configuration file(s) in the package to denote if the
header file is exported. For this approach, we have two alternatives:

a) Add a file parallel to the header file, e.g., the fact that "foo.h.export"
   exists could mean that the header file "foo.h" is an exported file.

b) We introduce a "manifest" file for each package, containing information
   about the files in the package.

The advantages of this approach is [incomplete list]:

- Changing the properties of a file (e.g., from "internal" to "exported") does
  not require any changes to the file itself.

- It allows header files to be marked with other properties, such as header
  files that are supposed to be exported to third-party developers.

The disadvantages are:

- Working with files is not trivial, e.g., copying all header files that goes
  into the plugin SDK could be:

     cp `grep plugin-sdk package/manifest | cut -f1` /distro/include


Include file and path management
--------------------------------

In order to manage the include path and the include files, it is necessary to
ensure that all the header files that are exported are available for every
package in the system, and *only* those files.

To handle this, we basically have two approaches:

1. We have an include path containing the directory where the exported header
   files for each package is stored.  This require the header files to be
   placed in a dedicated "export/" directory inside the package: otherwise,
   all header files of a package will be exportable, which is not the intention.

   So, for example, the include path could be set to

       pkb_a/export;pkg_b/export;pkg_c/export

   Whenever a package is added or removed, this would mean that the include
   path would have to be updated to match the actual packages available.

   The advantages of this approach are:

   - Simple model

   - No need to generate or copy files

   The disadvantages of this approach are:

   - If a package is added or removed, the include path have to be updated.
     Since every package depends on the include path, it might trigger a
     re-build.

   - If a header file with the same name is in multiple places, it will not be
     detected.

   - Is most cases, the source control systems will generate a conflict for the
     addition and removal of a directory to the include path in the build file
     (e.g., configure.ac or Makefile.am).

2. We have a dedicated include directory for, e.g., the server where exported
   header are available, and let the manifest file contain information on what
   files are to be made available in the central include directory.

   This would mean that the path stays the same regardless of what packages are
   available.

   For this approach, we have two "sub-approaches" on how to make the header
   files available from the include directory:

   a) Copy the files to the dedicated include directory

   b) Generate a header file holding only an #include directive referencing to
      the correct header file.

   The advantages of this approach is:

   - That there is no need to maintain an extensive include path to be able to
     compile a package (which might have dependencies on other packages).

   - Package maintenance is very easy. For example, adding a package does not
     require changing any include paths or anything at all in the build frame.

   - Conflicting header files will be detected during the build process (e.g.,
     when copying header files to the include directory).

   The disadvantages are:

   - Requires more work in the build frame.

   - It requires a "staging" phase, where header files are made available in
     the dedicated include directory, either by copying or generating files.

   - In the copy approach (2a), it is necessary to build a dependencies
     Makefile for the include directory, to trigger a copy whenever the
     original header file changes.

   - In the copy approach (2a), it is possible that a developer starts editing
     the wrong file, which will then be overwritten at some later point, which
     will be hard to discover.

Implementation
==============

In order to implement the structure described in the high-level specification,
we should approach it in well-contained steps that lead us to the goal. For
example, since we need to develop a build frame for supporting this, we need an
intermediate solution that does not cause problems for the final deployment of
the build structure and allow developers to work on creating packages without
introducing problems for the build frame.


Stage 1: Create the directory structure
---------------------------------------

Introduce the package directories and move the existing packages we have into
that directory. At this stage we will keep the existing autotools-based build
system and just do the minimal changes necessary to have a fully functional system.

We assume that the original "unstructured" sql/ code is dependent on the
packages, but that we have control over the dependencies between packages in the
"structured" directories.

In order to add a package, it will be necessary to:

- Create a Makefile.am for the package

- Add a reference under "SUBDIRS" in the parent directory

Note that in this stage, all header files in a package will be available as
package interface files, so care should be used when including header files from
other packages.

After this stage, developers will be able to create packages properly without
affecting the following stages.


Stage 2: Evaluate and optionally change to use CMake
----------------------------------------------------

It has been discussed if we should use CMake to build the server on all
platforms and not just Windows, since it seems to be a portable alternative.
However, concerns have been raised about the portability of CMake to the
platforms that we need to support, so this alternative need to be evaluated
before implementation starts.

The goal is to have an equivalently simple build system compared to existing
one, which also include being able to handle the system for defining pluggable
storage engines.

If the evaluation does not show problems with doing the switch, the replacement
should be done in two steps: just switching build system but otherwise maintain
the structure and build order of the old system.

We do this step separately since it will require merging the build process on
windows with the existing autotools-based build frame and still maintain the
same functionality.

After this stage, we will have a single build frame for all platforms, but there
will still be problems such as that package interfaces are not distinguished
from other header files.


Stage 3: Streamline and consolidate build frame
-----------------------------------------------

At this stage, the build frame will be consolidated by ensuring that there is
support for easily working with the code.