PinDown was not able to identify exactly which commit that caused the failure, because it did not get any test results between the last revision where the test passed and the first revision where the test failed. There is a range of revisions in between for which no test results were produced, only build failures and sometimes not even that (often due to a busy computer farm). Any commit in this range may have caused the test to fail. However, any such issue within this range may also have been fixed at a later stage and another bug may have been introduced after this range of revisions. The only thing we know for certain is that this test passed before this range of revisions so the bug was introduced within this range, or slightly less likely, at a later stage. Not earlier though.

Note The confidence level for this type of bug report is low.



How Did PinDown Reach This Conclusion?

PinDown tested the following during debug to reach this conclusion (using the same test and seed as in the test phase):

  • The test fails when retested a second time on the same revision as during the test phase

  • The test fails when tested on the specified commit itself (including all older commits, but no newer commits)

  • The test passes on the revision listed as "Test Pass" in the bug report

  • No test results are produced between the test failure and the test pass mentioned above. This is due to either compilation failures or no results at all (potentially due to busy computer farm).

Potential Stability Issues

Most often this type of bug reports is due to lack of computer resources, i.e. the computer farm is busy. However, sometimes it may also be due to some kind of stability problem making the test results untrustworthy.

  • Random Instability: With constrained random testing, reproducing the same test scenario using the same seed number is not possible if the testbench is too different than the revision of the testbench that was used during the test phase. You can judge the level of risk for random instability by simply reading the bug report. There is only a risk for random instability if the bug report describes substantial changes to the testbench (design changes does not matter).

  • Uncaptured Randomness: With constrained random testing, it is important to capture all randomly generated inputs to the test. If there is a random generator which is not controlled by a seed then this random generator needs to be turned off or captured in some other way (maybe with a different seed). Otherwise we are not retesting the same test. A variant of this scenario is when running in a very flaky environment. Maybe the hardware on which the tests are running are very different from each other. This is not the case with computer farms, but when you are testing ASIC samples and PCB boards in an early development phase it may be the case. In this scenario it is important to re-run on exactly the same piece of hardware, again to minimise randomn behaviour. Some instability can be handled with the set_error_response command.

  • PinDown Setup Problem: If PinDown has been setup incorrectly, e.g. not having the correct criterias for pass and fail then PinDown will interpret the results incorrectly (see the Extraction User Guide).

  • Severe IT issues: If all failures are due to IT issues, both in the test phase and when re-running the tests during the debug phase then there may not be any real failures at all, just intermittent IT-issues. In this scenario you should increase the number of times a test is repeated in the debug phase to make sure it is a real deterministic failure before the debugging starts (see the set_error_response command).