WL#3285: Test coverage tool for new changesets

Affects: Server-7.1   —   Status: Complete

Create a tool to check new server code for test coverage using gcov.

The gcov program is a very nice tool to check the code coverage of our test
suite (which paths in the code are covered by the test suite). Gcov can flag
lines that have never been executed.

However, as there is much code not covered by the existing test suite, it can
be difficult to spot manually with gcov whether a particular new changeset (or
changesets) are covered, or if there is missing a test case. This tool will
help in that task.

The idea is to determine the set of new changesets, and only check code
coverage of those lines that are modified by those changesets. Thus we can
check how the new code is or is not covered by the test suite.

Initially there will be a stand-alone tool, possibly integrated with
mysql-test-run.pl, that a developer can use to check coverage before pushing
new code.

Later, pushbuild will incorporate the tool to check each new push for test
coverage and report the results on the HTML pages.


Implementation
--------------

The tool will work from a list of changeset revisions/keys.

For the stand-alone tool, the default list will be the output of
'bk changes -L'. It will also be possible to specify an alternative list of
changesets.

For pushbuild, the list will consist of all changesets included in the push.

The tool will first run 'bk changes -v -r "-nd:REV: :GFILE:"' on
each changeset to obtain a list of source files, and for each file a list of
file revisions included in the changesets. If will then filter away files that
are not source files.

Next, the tool will run 'bk annotate' on each source file found, and find all
lines marked with one of the revisions collected in the previous step.

Next, the tool will run 'gcov ' on each source file found.

Finally, all lines which are marked by gcov with '####', and which matched a
revision number in 'bk annotate', are new lines not covered by tests. These
lines are reported.

There might also be some integration into mysql-test-run.pl being needed
(though that may be difficult, as full coverage may require running
mysql-test-run.pl multiple times, eg. with and without --ps-protocol).

The BUILD/compile-pentium-gcov scripts needs updating. Currently it seems to
fail on BDB, and it lacks a number of features (no MAX flags in there). And it
needs to set -O0 for best reports by gcov. And it seems that ccache needs to
be disabled, or gcov will not work.

The tool also needs to be aware of annotations like /* purecov: inspected */
on error branches that are impossible to cover in the test suite (out of
memory for example).
Here is a link describing the purecov: style annotations. We might as well use
the same:

http://computing.ee.ethz.ch/sepp/developmentstudio-2003.06.00-bo/purecov/html/ht_adj_manually.htm