PinDown managed to make the test pass again by only undoing one of the conflicting commits, while keeping all other commits both younger and older as they are. The test passes on each conflicting commit in isolation, but fails after the merger. If each conflicting commit is undone, one by one, then the test passes on the merger, proving that there is no other problem than a conflict between two otherwise good commits.

Note The confidence level for this type of bug report (validated true) is high. However, please note that PinDown only claims that there is a conflict between two otherwise good commits in isolation.

Example

images/bugreport_mergerofpasses.png

How Did PinDown Reach This Conclusion?

The graph below illustrates the bug report above. Note that the test does not start to fail until the changes from the conflicting commits have been merged.

images/graph_mergerofpasses.png

PinDown did the following to come to this conclusion:

  • The test fails when retested a second time on the same revision as during the test phase

  • The test fails on the merger

  • The test passes on both the parents to the merger

  • The test passes on the merger, when either of the conflicting commits have been unrolled

  • Most important: The test passes on the same revision as during the test phase with only one of the conflicting commits unrolled

Potential Stability Issues

In rare cases this type of bug report may be incorrect. If so, then this is due to some kind of stability problem making the test results untrustworthy.

  • Random Instability: With constrained random testing, reproducing the same test scenario using the same seed number is not possible if the testbench is too different than the revision of the testbench that was used during the test phase. However, in this case PinDown managed to make the failing test pass again on the same revision of both the testbench and the design, except for the specified list of commits whose changes were unrolled. Consequently the only possibility for random instability are the changes made in these commits. You can judge the level of risk for random instability by simply reading the bug reports. There is only a risk for random instability if any of the bug reports describe substantial changes to the testbench (design changes don’t affect random stability). Please note that this is the same level of risk as if you fix a testbench problem and re-run the test just to make sure it passes before you commit the change. There is a risk the test may pass only due to random instability.

  • Uncaptured Randomness: With constrained random testing, it is important to capture all randomly generated inputs to the test. If there is a random generator which is not controlled by a seed then this random generator needs to be turned off or captured in some other way (maybe with a different seed). Otherwise we are not retesting the same test. A variant of this scenario is when running in a very flaky environment. Maybe the hardware on which the tests are running are very different from each other. This is not the case with computer farms, but when you are testing ASIC samples and PCB boards in an early development phase it may be the case. In this scenario it is important to re-run on exactly the same piece of hardware, again to minimise randomn behaviour. Some instability can be handled with the set_error_response command.

  • PinDown Setup Problem: If PinDown has been setup incorrectly, e.g. not having the correct criterias for pass and fail then PinDown will interpret the results incorrectly (see the Extraction User Guide).

  • Severe IT issues: If all failures are due to IT issues, both in the test phase and when re-running the tests during the debug phase then there may not be any real failures at all, just intermittent IT-issues. In this scenario you should increase the number of times a test is repeated in the debug phase to make sure it is a real deterministic failure before the debugging starts (see the set_error_response command).