This bug report is a progress report and contains what PinDown knows about the issue so far. It is sent before the conclusive bug report is issued.

Note This debug progress report is enabled by the user (see "progressreports" in the man page for set_mail). PinDown can be set to send a progress report once to capture the groups of failure signatures (or "buckets"). Alternatively, PinDown can be set to send progress reports continuously during the debug session, once per iteration, all the way until the conclusion. The progress reports are by default off.

Example

images/bugreport_progress_report_v3.png

How Did PinDown Reach This Conclusion?

PinDown has not yet reached a conclusion when these progress reports are sent out.

Potential Stability Issues

You have to wait for a conclusion before you can draw conclusions about stability issues. However for completeness, the types of stability problem that may affect the outcome are these:

  • Random Instability: With constrained random testing, reproducing the same test scenario using the same seed number is not possible if the testbench is too different than the revision of the testbench that was used during the test phase. This may cause the test to fail on every single revision older than such a major testbench update.

  • Uncaptured Randomness: With constrained random testing, it is important to capture all randomly generated inputs to the test. If there is a random generator which is not controlled by a seed then this random generator needs to be turned off or captured in some other way (maybe with a different seed). Otherwise we are not retesting the same test. A variant of this scenario is when running in a very flaky environment. Maybe the hardware on which the tests are running are very different from each other. This is not the case with computer farms, but when you are testing ASIC samples and PCB boards in an early development phase it may be the case. In this scenario it is important to re-run on exactly the same piece of hardware, again to minimise randomn behaviour. Some instability can be handled with the set_error_response command.

  • PinDown Setup Problem: If PinDown has been setup incorrectly, e.g. not having the correct criterias for pass and fail then PinDown will interpret the results incorrectly (see the Extraction User Guide).

  • Severe IT issues: If all failures are due to IT issues, both in the test phase and when re-running the tests during the debug phase then there may not be any real failures at all, just intermittent IT-issues. In this scenario you should increase the number of times a test is repeated in the debug phase to make sure it is a real deterministic failure before the debugging starts (see the set_error_response command).