WL#7730: There should be an option to mark test as failed when check-testcase fails
Affects: Server-8.0 — Status: Complete
MTR runs a validation check after each test is run, the validation process checks for any residue left by the server after the test is completed. In order to validate this MTR takes a copy of the state of the server before starting the test and a copy of the state of the server after the test. These two states are then matched against each other to check if any residue was left by the test. A typical example would be to see if the test had created any database in the server while running but did not remove it. On such occasions , MTR would find out that the test had although passed but left a residue. MTR would although pass the test but would complain(warning message) , that the test had some leftovers. Passing tests are not noticed in regression runs. Hence it will be good to have an option that will make the test fail if check-testcase fails
R1 : There should be a mechanism to detect test cases that have failed in checktestcase() Note : checktestcase() is the mechanism that checks for any leftovers by the test. R2: Every test that fails in checktestcase() should be treated as a failure. R3: The test case that failed should be reported properly in the summary report. R4: Test case that failed in this instance should follow --force mechanism as well.
A new option --fail-check-testcases to be introduced in MTR. --fail-check- testcases fails a test if the test fails in checktestcase() --fail-check-testcases is switched OFF be default. --fail-check-testcases can be switched ON in default, provided test cases that fail in checktestcase() has been rectified.
Introduce a new option fail-check-testcases . The option fail-check-test fails a test when the test fails in checktest. The option is switched off by default. MTR_RES_FAILURE is assigned to a test instead of MTR_RES_PASS to test that fail in checktest. Test that fail in checktest is to be treated as a normal failure , i.e. , the test should follow the execution for --force option. The test failure count should be reflected as before.
Copyright (c) 2000, 2020, Oracle Corporation and/or its affiliates. All rights reserved.