WL#3285: Test coverage tool for new changesets
Affects: Server-7.1 — Status: Complete — Priority: Medium
Create a tool to check new server code for test coverage using gcov. The gcov program is a very nice tool to check the code coverage of our test suite (which paths in the code are covered by the test suite). Gcov can flag lines that have never been executed. However, as there is much code not covered by the existing test suite, it can be difficult to spot manually with gcov whether a particular new changeset (or changesets) are covered, or if there is missing a test case. This tool will help in that task. The idea is to determine the set of new changesets, and only check code coverage of those lines that are modified by those changesets. Thus we can check how the new code is or is not covered by the test suite. Initially there will be a stand-alone tool, possibly integrated with mysql-test-run.pl, that a developer can use to check coverage before pushing new code. Later, pushbuild will incorporate the tool to check each new push for test coverage and report the results on the HTML pages. Implementation -------------- The tool will work from a list of changeset revisions/keys. For the stand-alone tool, the default list will be the output of 'bk changes -L'. It will also be possible to specify an alternative list of changesets. For pushbuild, the list will consist of all changesets included in the push. The tool will first run 'bk changes -v -r<changeset> "-nd:REV: :GFILE:"' on each changeset to obtain a list of source files, and for each file a list of file revisions included in the changesets. If will then filter away files that are not source files. Next, the tool will run 'bk annotate' on each source file found, and find all lines marked with one of the revisions collected in the previous step. Next, the tool will run 'gcov <file>' on each source file found. Finally, all lines which are marked by gcov with '####', and which matched a revision number in 'bk annotate', are new lines not covered by tests. These lines are reported. There might also be some integration into mysql-test-run.pl being needed (though that may be difficult, as full coverage may require running mysql-test-run.pl multiple times, eg. with and without --ps-protocol). The BUILD/compile-pentium-gcov scripts needs updating. Currently it seems to fail on BDB, and it lacks a number of features (no MAX flags in there). And it needs to set -O0 for best reports by gcov. And it seems that ccache needs to be disabled, or gcov will not work. The tool also needs to be aware of annotations like /* purecov: inspected */ on error branches that are impossible to cover in the test suite (out of memory for example).
Here is a link describing the purecov: style annotations. We might as well use the same: http://computing.ee.ethz.ch/sepp/developmentstudio-2003.06.00-bo/purecov/html/ht_adj_manually.htm
Copyright (c) 2000, 2016, Oracle Corporation and/or its affiliates. All rights reserved.