Define how PinDown handles environemental problems such as intermediate failures, no results etc

This command manages how PinDown responds to unreliable results, such as intermediate failures, partial results or no results at all. This may occur when there are IT-issues, the computer farm is overloaded, the revision control system server does not respond or for example when there is no available disk space. It also may occur when PinDown tries to run tests on code that does not even compile, e.g. on an old revision or on code modified locally by PinDown as part of its debug exploration.

All these scenarios are referred to errors and PinDown has a default response to each of them. This response is a sequence of PinDown commands which in most cases is the adequate way of handling the error. The list of errors and the default response is listed below.

Error Phase Default Response Description

test

test

repeat 1;

A normal test failure in the test phase. Response applies if the test is selected in its bug group for debug

build

test

repeat 1;

A normal build failure in the test phase. Response applies if the build is selected in its bug group for debug

noresultfile

test

repeat 3;
halt;

There is no results XML file in the test phase

test

debug

none

A requested build failed in the debug phase. Normal test failure.

build

debug

none

A requested build failed in the debug phase. Normal build failure.

otherbuild

debug

repeat 1;
set_result -type "test" -value "ignore";

A build failed when a test was requested on a different revision than in the test phase

otherbuild-onrepeat

debug

repeat 3;
halt;

A build failed when a test was requested on the same revision as in the test phase

validation

debug

repeat 1;
set_result -type "test" -value "fail";

PinDown tried to make the test pass but did not succeed. Got test failure, build failure or no result.

pushout

debug

set_result -type "test" -value "ignore";

Requesting a test that failed in a previous run, but has no result in this run (see set_test_noresult). Got no result.

noresult-testrequested

debug

repeat 1;
set_result -type "test" -value "ignore";

Requesting a test, got no results. Different revision than in the test phase.

noresult-testrequested-onrepeat

debug

repeat 3;
halt;

Requesting a test, got no results. Same revision as in the test phase.

noresult-buildrequested

debug

repeat 1;
set_result -type "build" -value "fail";

Requesting a build, got no results. Different revision as in the test phase.

noresult-buildrequested-onrepeat

debug

repeat 3;
halt;

Requesting a build, got no results. Same revision as in the test phase.

noresult-buildrequested-closedbug

debug

repeat 1;
set_result -type "build" -value "fail";

Requesting a build related to a closed build bug, on an old revision when the bug was open. Got no results.

noresultfile-testrequested

debug

repeat 1;
set_result -type "test" -value "ignore";

Requesting a test, got no results, not even an empty XML results file.

noresultfile-buildrequested

debug

repeat 1;
set_result -type "build" -value "fail";

Requesting a build, got no results, not even an empty XML results file.

setupcheck-testrequested

debug

repeat 3;
halt;

PinDown setup checked by re-running a passing test from the test phase (see check_setup) but did not get a passing test

setupcheck-buildrequested

debug

repeat 3;
halt;

PinDown setup checked by re-compiling a passing build from the test phase (see check_setup) but did not get a passing build

checkoutfailed-testrequested

both

repeat 3;
halt;

Checkout failed 3 attempts when requesting a test

checkoutfailed-buildrequested

both

repeat 3;
halt;

Checkout failed 3 attempts when requesting a build

system

both

repeat 3;
halt;

A user-defined error has occurred

Intermittent Issues - Avoid Debugging Test Failures That Are Not Real

In some cases the user may want to change the default behaviour and this is when this command, set_error_response, is used. The most common reason for the user to modify the default error responses is that there may be too many intermediate failures in the user’s system. This may cause PinDown to debug failures which are no real failures and sometimes even accuse somone for having made a bad commit, even though there is actually no issue. The reason is that PinDown by default assume, not perfect, but reasonable good results.

Before any test failure or build failure is debugged PinDown by default reproduces the test on the same revision where the tests or builds just failed. This filters out any intermittent issue. If the failure is a pass when it has been reproduced then the failure is reported as an intermittent failure and no further debug analysis is performed.

However, some times there are too many intermittent issues. The user may then decide to reproduce the failure not just once, but more times, say three times, in order to have a better filter for intermittent issues. Only if all three instances of the test run also fail will PinDown start debugging the issue. If it was a 20% chance that an intermittent issue would still be reported as a failure (even though the underlying design was fine) when it was reproduced just once then the probability would now be reduced to 0.8% (20%*20%*20% = 0.8%). Note that all these three tests would be run in parallel in the same debug iteration. The purpose with this is to stop PinDown from debugging failures which are just intermittent and blame a commit that is actually fine.

In order to do this the following set_error_response commands should be written to the PinDown configuration file. They must be placed in the configuration file before the test phase finishses.

set_error_response -type "build" -phase "test" -actions {repeat 3;};
set_error_response -type "test" -phase "test" -actions {repeat 3;};

Sometimes it is only the build failures that are not reliable or just the test failures. In that case only modify the error response to the type of failure that has intermittent issues.

Intermittent Issues - When Re-Running Tests During Debug is Unreliable

It is more common to experience intermittent issues in the test phase, because much more tests are run during the test phase, fighting for the same IT-resources. During debug PinDown re-runs very few tests in parallel, typically 10 if that is what the debug bandwidth is set to (see set_debug_bandwidth). However if the user’s system suffers a lot from intermediate issues you may have to also address how PinDown handles results during debug.

During debug, the default error response is to do nothing for normal test and build failures. However if we cannot rely on them then it is a good idea to introduce say one additional re-run of the tests or builds with these commands:

set_error_response -type "build" -phase "debug" -actions {repeat 1;};
set_error_response -type "test" -phase "debug" -actions {repeat 1;};

However, it is very rare that you cannot rely on the few debug test results that are run. Normally there is just one specific error message that is by mistake interpreted as the test failed or that actually indicates a real but intermittent failure. If there is an error message that needs to be filtered out then that is handled by the extract -filter option. If on the other hand, the test needs to be re-run only when this specific error message appears then it is more efficient to define this action only for this scenario then to re-run every single debug test. The next section deals with user-defined errors and responses.

User-Defined Reponses to User-Defined Errors ("System Errors")

The user can define their own errors which are not listed in the table above and which do not have any default responses. Using the extract -type "systemerror" option the user can identify any failure message as a system error.

The default response for a system error (see system in the table above) is to reproduce the test or build failure 3 times and then halt the complete PinDown execution as it is deemed no point in continuing. However to handle intermediate issues it may be more appropriate to just reproduce the failure 3 times and then continue. See example below:

extract -type "systemerror" -path "%testname%" -keywords "Cannot open file";
set_error_response -type "systemerror" -phase "debug" -actions {
        repeat 3;
};

It possible to identify several errors with different responses. Below we halt the PinDown execution immediately if there is no disk space and we are in the test phase. However if we are in the debug phase we tell Pindown to remove its debug files and retry 3 times before giving up. Also with a completely different error response we tell PinDown to try 3 times to get a license before giving up, only in the debug phase.

extract -type "systemerror" -label "no_licenses" -source "log" -path "%testname%" -keywords "Fail: Cannot get a license. All licenses taken at the moment.";
extract -type "systemerror" -label "showstopper" -source "log" -path "%testname%" -keywords "No disk space";
set_error_response -type "showstopper" -phase "test" -actions {
        halt;
};
set_error_response -type "showstopper" -phase "debug" -actions {
        shell "removeDebugFiles.sh";
        repeat 3;
        halt;
};
set_error_response -type "no_licenses" -phase "debug" -actions {
        repeat 3;
        halt;
};

Performance

If your system is very reliable and you never have any intermittent issues then you can improve the performance by changing the default responses by removing the repeat-commands from some errors-types.

For example, if PinDown tries to run a test on an old revision which does not even compile (error-type "otherbuild") and in the user’s system this is never due to an intermittent issue then you can remove the repeat-command and double the performance for this particular scenario:

set_error_response -type "otherbuild" -phase "debug" -actions {
        set_result -type "test" -value "ignore";
};

Syntax

set_error_response -type "error" -phase "phase" -actions { command1; … ; commandN;};

‑type

Set to an error defined above in the table or an error type defined by extract -type "systemerror"

‑phase

Set to "test" or "debug"

‑actions

Any list of PinDown commands or unix commands (by using the PinDown shell-command)

Example

Example 1

Suffering from a lot of intermediate build failures due to IT issues on some computer farm machines By re-running the failures we are probably allocated a different computer next time we submit a job to the farm

set_error_response -type "build" -phase "test" -actions {repeat 3;};
set_error_response -type "otherbuild" -phase "debug" -actions {
        repeat 2;
        set_result -type "test" -value "ignore";
};

Example 2

We have a stable system with hardly ever any intermediate issues. Sometimes we do have build failures checked in. The following is done to improve the debug performance when re-running tests or builds on older or modified revisions which do not compile. All repeat 1-commands have been removed from the default error responses.

set_error_response -type "otherbuild" -phase "debug" -actions {
        set_result -type "test" -value "ignore";
};
set_error_response -type "validation" -phase "debug" -actions {
        set_result -type "test" -value "fail";
};
set_error_response -type "noresult-testrequested" -phase "debug" -actions {
        set_result -type "test" -value "ignore";
};
set_error_response -type "noresult-buildrequested" -phase "debug" -actions {
        set_result -type "build" -value "fail";
};
set_error_response -type "noresult-buildrequested-closedbug" -phase "debug" -actions {
        set_result -type "build" -value "fail";
};

Example 3

If repeating a failing test on the same revision as in the test phase does not produce any results then either 1) PinDown is not setup correctly or 2) the computer farm was too busy or 3) there is a problem only with this specific test. Default PinDown stops because we assume there is a setup issue, but if it is always due to one of the other issues then you can tell PinDown to ignore this specific test and move on to debug other failures by doing the following:

set_error_response -type "otherbuild-onrepeat" -phase "debug" -actions {
        set_result -type "test" -value "ignore";
};
set_error_response -type "noresult-testrequested-onrepeat" -phase "debug" -actions {
        set_result -type "test" -value "ignore";
};
set_error_response -type "noresult-buildrequested-onrepeat" -phase "debug" -actions {
        set_result -type "build" -value "ignore";
};

Example 4

If build failures found during testing is treated as a non-issue they can be ignored by doing the following:

set_error_response -type "build" -phase "test" -actions {
        set_result -type "build" -value "ignore";
};