PinDown concluded that this failure is intermittent. During the test phase PinDown registered a failure, but when re-running the test and seed on the same revision the test now passes. PinDown will not investigate it further.

In a slave-mode setup where PinDown is reading results generated by another tool and then debugging the problems on the latest revision, an intermittent result may be because the issue has already been fixed. It becomes intermittent because PinDown has not been told exactly which revision that the test results relate to other than that they are ("the latest"). That is fine, as all we want to do is to debug open issues.

Note The confidence level for this type of bug report (intermittent error) is high. If the test passes on the same revision on the second attempt then this failure is definitely not due to a bad commit. In slave-mode it may signal that the issue was a stable failure, but which since has been fixed.

Example

images/bugreport_intermittent_v3.png

How Did PinDown Reach This Conclusion?

PinDown tested the following during debug to reach this conclusion (using the same test and seed as in the test phase):

  • The test now passes when retested a second time on the same revision as during the test phase

Potential Stability Issues

An intermittent issue may be a sign of instability:

  • Random Instability: With constrained random testing, reproducing the same test scenario using the same seed number is not possible if the testbench is too different than the revision of the testbench that was used during the test phase. However in this case, as intermittent issues means different results on exactly the same revision of the code base, including the testbench, this cannot be the issue here.

  • Uncaptured Randomness: With constrained random testing, it is important to capture all randomly generated inputs to the test. If there is a random generator which is not controlled by a seed then this random generator needs to be turned off or captured in some other way (maybe with a different seed). Otherwise we are not retesting the same test. A variant of this scenario is when running in a very flaky environment. Maybe the hardware on which the tests are running are very different from each other. This is not the case with computer farms, but when you are testing ASIC samples and PCB boards in an early development phase it may be the case. In this scenario it is important to re-run on exactly the same piece of hardware, again to minimise randomn behaviour. Some instability can be handled with the set_error_response command.

  • PinDown Setup Problem: If PinDown has been setup incorrectly, e.g. not having the correct criterias for pass and fail then PinDown will interpret the results incorrectly (see the Extraction User Guide). For intermittent issues it could be a mistake where a failure keyword has only been added to the definition of failures in the test phase and not in the debug phase.

  • Severe IT issues: If all failures are due to IT issues, both in the test phase and when re-running the tests during the debug phase then there may not be any real failures at all, just intermittent IT-issues. In this scenario you should increase the number of times a test is repeated in the debug phase to make sure it is a real deterministic failure before the debugging starts (see the set_error_response command).